Keyword-based data collection method and device

Abstract

The present application provides a keyword-based data collection method and device, which relates to the field of data processing technology and solves the technical problem of low accuracy of data collection results. The method includes: extracting multiple text keywords based on the first collected content corresponding to the data collection request; determining a target text keyword whose appearance frequency is greater than a first preset frequency value among the multiple text keywords, and determining a second collected content whose appearance frequency of the target text keyword is greater than a second preset frequency value in the second data source; identifying the image data in the first collected content and the second collected content, determining the target image data for the same thing at multiple different shooting angles according to the recognition results, and dividing all the target image data for the same thing into the same type of images; determining multiple event keywords according to multiple events corresponding to the multiple types of images, and collecting data through the event keywords to obtain the third collected content.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular to a keyword-based data collection method and device. Background Art

[0002] At present, there are situations where it is necessary to collect various data on the Internet based on keywords. For example, data related to a certain event is collected from various news through keywords. For another example, the website administrator needs to save some keyword-related content from the department websites corresponding to multiple different departments in the company to the server of the company headquarters, or download multiple website data of multiple subsidiaries under the company to the website of the head office, and then extract relevant fields from the content and publish them to the website system of the headquarters. Sometimes it is necessary to save files related to web pages locally, such as pictures, attachments, etc. However, in the process of collecting data through keywords, the accuracy of the data collection results is often low due to the accuracy of the keyword description. Summary of the invention

[0003] The purpose of the present invention is to provide a keyword-based data collection method and device to solve the technical problem of low accuracy of data collection results.

[0004] In a first aspect, an embodiment of the present application provides a keyword-based data collection method, the method comprising:

[0005] Obtaining a data collection request, and extracting a plurality of text keywords based on first collected content corresponding to the data collection request, wherein the first collected content is derived from a first data source;

[0006] Determine a target text keyword whose appearance frequency is greater than a first preset frequency value among the multiple text keywords, and determine at least one second collected content whose appearance frequency of the target text keyword is greater than a second preset frequency value in at least one second data source; wherein the second preset frequency value is greater than the first preset frequency value;

[0007] Recognize the image data in the first collected content and the second collected content to obtain a recognition result, determine target image data of the same object at multiple different shooting angles according to the recognition result, and classify all the target image data of the same object into the same type of image;

[0008] Determine multiple release times of the target image for each category, and classify the target images whose release times have a time interval greater than a preset time threshold into different categories of images, to obtain multiple events corresponding to the multiple categories of images;

[0009] Determine a plurality of event keywords according to the plurality of events corresponding to the plurality of types of images, and collect data using the event keywords to obtain third collected content;

[0010] A final collection result corresponding to the data collection request is obtained based on the first collection content, the second collection content, and the third collection content.

[0011] In a possible implementation, determining a plurality of event keywords according to the plurality of events corresponding to the plurality of images of the categories, and performing data collection using the event keywords to obtain the third collected content includes:

[0012] According to the multiple events corresponding to the multiple types of images, data collection and data summarization are performed through the AI ​​model to obtain data summary results, and based on the data summary results, multiple event keywords are obtained through AI model analysis;

[0013] According to each of the multiple event keywords, a description text of the event keyword under the corresponding multiple expression dimensions is generated; wherein the multiple expression dimensions include any one or more of the following: a processing method for a target event corresponding to the event keyword, a processing result for the target event, an event summary content for the target event, and an event analysis content for the target event;

[0014] The AI ​​model is used to filter and exclude everyday terms in the description text to obtain a target description text, and data collection is performed based on the target description text to obtain third collected content.

[0015] In a possible implementation, the data summary result includes a location keyword, a time keyword, and a collection target keyword; the multiple event keywords obtained by analyzing the data summary result through an AI model include:

[0016] Based on the location keyword and the time keyword, a range search is performed through an AI model to obtain first data within a target range; wherein the data format of the first data includes at least one of an image, text, and audio;

[0017] Analyze the data content of the first data, and determine and summarize at least one occurred event described in the first data according to the data content analysis result;

[0018] A corresponding first event keyword is generated according to each of the occurred events, and event keywords irrelevant to the collection target keyword are excluded from the first event keywords to obtain second event keywords.

[0019] In a possible implementation, after the step of generating a corresponding first event keyword according to each of the occurred events and excluding event keywords that are not related to the collection target keyword from the first event keywords to obtain the second event keywords, the step further includes:

[0020] Perform a specific search based on the second event keyword through the AI ​​model to obtain second data;

[0021] The AI ​​model is trained and optimized based on the second data to obtain a trained AI model, so as to optimize the search process using the trained AI model.

[0022] In a possible implementation, after the step of obtaining a plurality of event keywords through AI model analysis based on the data summary result, the method further includes:

[0023] In response to an exclusion selection operation on the multiple event keywords, determining an event keyword to be excluded corresponding to the exclusion selection operation among the multiple event keywords, and performing exclusion processing on the event keyword to be excluded among the multiple event keywords to obtain an event keyword after the exclusion processing;

[0024] In response to a key selection operation on multiple event keywords after the exclusion process, the key event keyword corresponding to the key selection operation is determined among the multiple event keywords after the exclusion process, and the event associated with the key event keyword is used as the key data collection object in the data collection process.

[0025] In a possible implementation, determining target image data of the same object at multiple different shooting angles according to the recognition result includes:

[0026] Determine, in the first collected content and the second collected content, a plurality of target object images whose object image similarity is greater than a preset similarity threshold according to the recognition result, and perform noise elimination processing on the plurality of target object images to obtain target object images after noise elimination;

[0027] The objects in the plurality of noise-processed target object images are determined to be the same object, and target image data for the same object at a plurality of different shooting angles are determined in the first acquisition content and the second acquisition content.

[0028] In a possible implementation, the step of extracting a plurality of text keywords based on the first collected content corresponding to the data collection request includes:

[0029] Based on the first collected content corresponding to the data collection request, a word segmentation algorithm in machine learning is used to perform natural language recognition to obtain a language recognition result, and a plurality of text keywords are extracted from the first collected content according to the language recognition result;

[0030] Setting a label for the first collected content according to the text keyword, and associating the labeled first collected content with related content based on the text keyword to obtain labeled and mutually associated historical collected data;

[0031] The labels of the target historical collection data whose collection frequency is higher than the preset frequency threshold in the historical collection data are marked, and the probability of the target historical collection data with the marked labels being collected again is reduced.

[0032] In a second aspect, a keyword-based data collection device is provided, comprising:

[0033] An acquisition module, used to acquire a data acquisition request, and extract a plurality of text keywords based on first acquired content acquired corresponding to the data acquisition request; the first acquired content is derived from a first data source;

[0034] A determination module, configured to determine a target text keyword whose occurrence frequency is greater than a first preset frequency value among the plurality of text keywords, and to determine at least one second collected content whose occurrence frequency of the target text keyword is greater than a second preset frequency value in at least one second data source; wherein the second preset frequency value is greater than the first preset frequency value;

[0035] an identification module, configured to identify the image data in the first collected content and the second collected content to obtain an identification result, determine target image data of the same object at multiple different shooting angles according to the identification result, and classify all the target image data of the same object into images of the same type;

[0036] A classification module is used to determine multiple release times of the target image in each category, and classify the target images whose time intervals among the multiple release times are greater than a preset time threshold into different categories of images, so as to obtain multiple events corresponding to the multiple categories of images;

[0037] The acquisition module is used to determine multiple event keywords based on the multiple events corresponding to the multiple types of images, perform data acquisition through the event keywords, obtain third acquisition content, and obtain a final acquisition result corresponding to the data acquisition request based on the first acquisition content, the second acquisition content and the third acquisition content.

[0038] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0039] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the method described in the first aspect above.

[0040] The embodiments of the present application bring the following beneficial effects:

[0041] A keyword-based data collection method and device provided in an embodiment of the present application can obtain a data collection request, and extract multiple text keywords based on a first collected content corresponding to the data collection request; the first collected content comes from a first data source; a target text keyword whose appearance frequency is greater than a first preset frequency value is determined among the multiple text keywords, and at least one second collected content whose appearance frequency of the target text keyword is greater than a second preset frequency value is determined in at least one second data source; wherein the second preset frequency value is greater than the first preset frequency value; the image data in the first collected content and the second collected content are recognized to obtain a recognition result, and a target text keyword for the same is determined according to the recognition result. Target image data of an object at multiple different shooting angles, and classifying all the target image data for the same object into the same category of images; determining multiple release times of the target image for each category of the target image, and classifying the target images with a time interval greater than a preset time threshold among the multiple release times into different categories of images, to obtain multiple events corresponding to the multiple categories of images; determining multiple event keywords based on the multiple events corresponding to the multiple categories of images, and performing data collection through the event keywords to obtain a third collection content; and obtaining a final collection result corresponding to the data collection request based on the first collection content, the second collection content, and the third collection content. In this solution, by determining target text keywords with higher occurrence frequencies of the same keywords from other data sources except one data source, the contents related to the target text keywords in these other data sources are also collected. In this way, the analysis and expansion process of hot keywords is realized, making the collected content more comprehensive, not only limited to the first data source, but also by dividing all target image data of the same object under multiple different shooting angles into the same type of images, the precise division of events in the spatial dimension of the object is realized, and then the events released at at least two times with a large time difference are determined as different events, thereby realizing a fine division of events in the time dimension. Furthermore, the multiple event keywords determined by the multiple events corresponding to the multiple types of images can make the data collection process more targeted for each event, so as to improve the accuracy of data collection, avoid the situation in which the accuracy of data collection results is low due to the description accuracy of keywords in the process of data collection through keywords, and solve the technical problem of low accuracy of data collection results.

[0042] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 A flowchart of a keyword-based data collection method provided in an embodiment of the present application;

[0045] Figure 2 Another schematic diagram of a flow chart of a keyword-based data collection method provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of the structure of a keyword-based data acquisition device provided in an embodiment of the present application;

[0047] Figure 4 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0049] The terms "including" and "having" and any variations thereof mentioned in the embodiments of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0050] At present, in the process of collecting data through keywords, the accuracy of the description of keywords often leads to low accuracy of the data collection results. Based on this, the embodiment of the present application provides a keyword-based data collection method and device, which can solve the technical problem of low accuracy of data collection results.

[0051] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0052] Figure 1A flowchart of a keyword-based data collection method provided in an embodiment of the present application. Figure 1 As shown, the method includes:

[0053] Step S110: obtaining a data collection request, and extracting a plurality of text keywords based on the first collected content corresponding to the data collection request.

[0054] The first collected content originates from the first data source. In a possible implementation manner, text keywords may be extracted from the text content (the first collected content) collected from the first data source.

[0055] As an optional implementation, this step may include the following specific steps: based on the first collected content corresponding to the data collection request, using the word segmentation algorithm in machine learning to perform natural language recognition to obtain a language recognition result, and extracting multiple text keywords from the first collected content according to the language recognition result; setting labels for the first collected content according to the text keywords, and associating the labeled first collected content with related content based on the text keywords to obtain labeled and mutually related historical collected data; labeling the target historical collected data in the historical collected data whose collection frequency is higher than a preset frequency threshold, and reducing the probability of the labeled target historical collected data being collected again.

[0056] For example, corresponding tags are set at the first collection, related content associations or collected content associations are established, and then historical collection data is determined, and then target historical collection data with a high historical collection frequency is determined based on the historical collection data.

[0057] As an optional implementation method for extracting multiple text keywords, a word segmentation algorithm in machine learning can be used in combination with natural language recognition to extract multiple text keywords.

[0058] Step S120, determining a target text keyword whose occurrence frequency is greater than a first preset frequency value among multiple text keywords, and determining at least one second collected content whose occurrence frequency of the target text keyword is greater than a second preset frequency value in at least one second data source.

[0059] Among them, the second preset frequency value is greater than the first preset frequency value. Exemplarily, for words with a higher frequency of occurrence as keywords, according to the keywords with a higher frequency of occurrence in the news, the target objects whose frequency of occurrence of the same keywords is greater than the preset value are determined from other data sources (second data sources) other than the first data source, and the content related to the keyword in the second data source is also collected. In this way, the analysis process of hot keywords is realized, making the collected content more comprehensive and not limited to the first data source.

[0060] As an optional implementation, first, collect and process text data: collect text data from a first data source and extract keywords. Then, count the keyword frequency: calculate the frequency of occurrence of each keyword in the first data source, and filter out target keywords whose frequency of occurrence is greater than a first preset frequency value. Then, search for target keywords in a second data source: search for these target keywords in at least one second data source and count their frequency of occurrence. Finally, filter out collected content that meets the second preset frequency value: filter out target keywords and their corresponding collected content whose frequency of occurrence in the second data source is greater than the second preset frequency value.

[0061] Step S130, identifying the image data in the first collected content and the second collected content to obtain a recognition result, determining target image data for the same object at multiple different shooting angles based on the recognition result, and classifying all target image data for the same object into the same type of image.

[0062] As an example, this step may include the following specific steps: determining, in the first acquisition content and the second acquisition content, multiple target object images whose degree of object image similarity is greater than a preset similarity threshold according to the recognition result, and performing noise elimination processing on the multiple target object images to obtain noise-processed target object images; determining the objects in the multiple noise-processed target object images as the same object, and determining target image data for the same object at multiple different shooting angles in the first acquisition content and the second acquisition content.

[0063] For example, AI summarizes the content of an article and the time it was published, and excludes noise. For the process of AI summarizing the content of an article, the following two methods can be used in combination: the first method is to use AI to compare whether the content of each summary is similar; the second method is for AI to compare videos or pictures, identify the images of the same person (or object) under different shooting angles, and classify the pictures describing the same thing according to the recognition results.

[0064] In an optional implementation, image preprocessing is first performed to preprocess the collected images, including scaling, cropping, denoising, etc. Then, image feature extraction: a deep learning model (such as a convolutional neural network CNN) is used to extract the feature vector of the image. Image clustering is then performed: based on the extracted feature vector, similar images are clustered together using a clustering algorithm (such as K-means or DBSCAN). Finally, classification and labeling are performed: each cluster is classified and labeled to ensure that the images in each cluster are taken from different angles of the same object.

[0065] Step S140, determining multiple release times of target images in each category, and classifying target images whose release times have a time interval greater than a preset time threshold into different categories of images, to obtain multiple events corresponding to the multiple categories of images.

[0066] In a possible implementation, after the pictures describing the same thing are classified according to the recognition results, the time period in which the thing was published is determined, and the event analysis is performed based on this time period. For example, two events published at a large time interval are determined to be different events. By summarizing the content of the article and the time when the article was published, two events with a long time interval between publications are classified as different events, thereby finely dividing the events by time dimension.

[0067] As an example, first, get the release time of the image: extract the release time of each image from the image metadata (such as EXIF ​​data) or related databases. If the image does not have direct release time information, it can be obtained from the file system or other records. Then, perform time interval calculation: calculate the release time interval between adjacent images, and use Python's datetime module to process time. Finally, perform event division: according to the preset time interval threshold, divide the images with a time interval greater than the threshold into different events.

[0068] For obtaining the release time of an image, exemplarily, the get_image_creation_time function reads the EXIF ​​data of the image and extracts the Image DateTime or EXIF ​​DateTimeOriginal field as the release time of the image. If the image has no EXIF ​​data, the modification time of the file is used as the release time. For the main function, exemplarily, the main function first obtains the release time of all images and sorts them by time; then, initializes an event list events and a current event list current_event; traverses the sorted image list and calculates the time interval between adjacent images. If the time interval is greater than the preset time interval threshold, a new event is created, otherwise, the image is added to the current event; finally, each event and the image path it contains are output.

[0069] Step S150, determining a plurality of event keywords according to a plurality of events corresponding to a plurality of types of images, and performing data collection through the event keywords to obtain third collected content.

[0070] In practical applications, multiple event keywords determined by multiple events corresponding to multiple types of images can make the data collection process more targeted to each event, thereby improving the accuracy of data collection.

[0071] In an optional implementation, first, extract event keywords and extract keywords from the image of each event. Exemplarily, image content analysis: use image recognition technology (such as deep learning models) to identify objects and scenes in images. For example, use a pre-trained MobileNetV2 model to classify images and extract the most likely object labels. Then, perform text analysis: if the image has relevant text descriptions or tags, natural language processing technology can be used to extract keywords. For example, use the NLTK library to segment text and remove stop words to extract keywords. Then perform data collection: use the extracted keywords to perform data collection and obtain relevant content. Exemplarily, web crawler: use web crawler technology to search for relevant information on the Internet through event keywords. For example, use Google Custom Search API or other search engine APIs to search for relevant content through event keywords. Finally, perform API calls: obtain search results and extract title, link, and summary information. For example, use third-party APIs (such as news APIs, social media APIs, etc.) to obtain content related to keywords.

[0072] Step S160, obtaining a final collection result corresponding to the data collection request based on the first collection content, the second collection content and the third collection content.

[0073] As an example, the first collected content is content obtained from a news website or social media platform; the second collected content is content obtained from another data source (such as a professional database, forum, etc.); and the third collected content is content obtained based on image event keywords. In this step, first, all content is merged: the first collected content, the second collected content, and the third collected content are merged into a list all_content. Then deduplication is performed: the set seen is used to record the links that have been processed to ensure that each link appears only once. The all_content list is traversed. If the link is not in the seen set, it is added to the unique_content list, and the link is added to the seen set. Then sorting is performed: the unique_content list is sorted using the sorted function. In the example, sorting is performed by title. The request loading method can select other sorting methods (such as time, relevance, etc.) according to actual needs. Finally, the final result is generated: the final result is stored in the final_result list, which can be further processed or output. Through the above steps, the first collected content, the second collected content, and the third collected content can be integrated into a comprehensive final collection result. The result has duplicates removed and is sorted as needed to facilitate subsequent analysis and use.

[0074] In an embodiment of the present application, by determining target text keywords with higher occurrence frequencies of the same keywords from other data sources except one data source, the contents related to the target text keywords in these other data sources are also collected. In this way, the analysis and expansion process of hot keywords is realized, making the collected content more comprehensive, not only limited to the first data source, but also by dividing all target image data of the same object under multiple different shooting angles into the same type of images, accurate division of events in the spatial dimension of the object is achieved, and then by determining that events released at at least two times with a large time difference are different events, a fine division of events in the time dimension is achieved. Furthermore, multiple event keywords determined by multiple events corresponding to multiple types of images can make the data collection process more targeted for each event, so as to improve the accuracy of data collection, avoid the situation in which the accuracy of data collection results is low due to the description accuracy of keywords in the process of data collection through keywords, and solve the technical problem of low accuracy of data collection results.

[0075] The above steps are described in detail below.

[0076] In some embodiments, Figure 2 As shown, the above step S150 may include the following steps:

[0077] Step S152, collecting and summarizing data according to multiple events corresponding to multiple types of images through the AI ​​model to obtain data summary results, and obtaining multiple event keywords through AI model analysis based on the data summary results;

[0078] Step S154, generating a description text of the event keyword under the corresponding multiple expression dimensions according to each event keyword in the multiple event keywords; wherein the multiple expression dimensions include any one or more of the following: a processing method for a target event corresponding to the event keyword, a processing result for the target event, an event summary content for the target event, and an event analysis content for the target event;

[0079] Step S156, using the AI ​​model to filter and exclude everyday terms in the description text to obtain the target description text, and perform data collection based on the target description text to obtain the third collected content.

[0080] As an example, according to the processing results of the two methods specifically described in the above step S130, AI can be used to collect and summarize images, videos, or audios, etc., analyze the keywords appearing therein, and generate expressions of different description dimensions corresponding to multiple keywords based on the analysis results, for example, how to handle event A (keyword), what happened in event A (keyword), and other expressions of multiple description dimensions. By using multiple expressions of different description dimensions for keywords, the use, search, and data collection process of keywords are made more comprehensive and flexible.

[0081] In some embodiments, the data summary result includes location keywords, time keywords, and collection target keywords; the above process of obtaining multiple event keywords through AI model analysis based on the data summary result may specifically include the following steps:

[0082] Performing a range search through an AI model based on the location keyword and the time keyword to obtain first data within a target range; wherein the data format of the first data includes at least one of an image, text, and audio;

[0083] Analyze the data content of the first data, and determine and summarize at least one occurred event described in the first data according to the data content analysis result;

[0084] A corresponding first event keyword is generated according to each event that has occurred, and event keywords that are irrelevant to the collection target keyword are excluded from the first event keywords to obtain second event keywords.

[0085] For example, the user sends a query request, such as what events occurred in area C today. The system conducts a rough, large-scale search based on this, summarizes the events that have occurred based on the searched related images, articles, audio, etc., and then conducts a detailed search based on the keywords generated for each summarized event.

[0086] In the embodiment of the present application, it is realized that more refined keywords (such as A event, B event, C1 sub-region in C region, C2 sub-region in C region, etc.) are generated according to the keywords issued by the user terminal (such as today, C region, event), and then AI learning (model training) is performed based on the generated more refined keywords, and irrelevant keywords are excluded. By learning and training while searching and continuing to search according to the training results, the keywords used for search are gradually made more accurate.

[0087] In some embodiments, after the above process of generating a corresponding first event keyword according to each occurred event and excluding event keywords that are not related to the collection target keyword from the first event keywords to obtain the second event keyword, the method may further include the following steps:

[0088] Based on the second event keyword, a specific search is performed using the AI ​​model to obtain second data; based on the second data, the AI ​​model is trained and optimized to obtain a trained AI model, so as to optimize the search process using the trained AI model.

[0089] The process of learning and collecting based on keywords through AI is to summarize all the conceivable keywords, and then search based on all the keywords, while searching, learning and collecting. If AI does not summarize the keywords initially, but finds them during the search process, it will continue to learn based on the target keywords found during the search process, thus realizing the process of further supplementing, training, optimizing and improving the keyword summarization logic of AI.

[0090] In some embodiments, after the above process of obtaining multiple event keywords through AI model analysis based on the data summary results, the method may further include the following steps:

[0091] In response to an exclusion selection operation on multiple event keywords, determining an event keyword to be excluded corresponding to the exclusion selection operation among the multiple event keywords, and performing an exclusion process on the event keyword to be excluded among the multiple event keywords to obtain an event keyword after the exclusion process;

[0092] In response to a key selection operation on multiple event keywords after exclusion processing, a key event keyword corresponding to the key selection operation is determined among the multiple event keywords after exclusion processing, and events associated with the key event keyword are used as key data collection objects in the data collection process.

[0093] In a possible implementation, in the middle of the search process, the user can also select from these keywords key words for focused search (e.g., if the user is particularly interested in event B, then select the keyword of event B as the key word for focused search), general search keywords, and keywords that can be excluded (e.g., if the user is not interested in event A, then choose to exclude the keyword of event A), thereby enabling more flexible adjustment of keywords through user operations.

[0094] Figure 3 A structural diagram of a keyword-based data acquisition device is provided. Figure 3 As shown, the keyword-based data collection device 300 includes:

[0095] The acquisition module 301 is used to acquire a data acquisition request, and extract a plurality of text keywords based on the first acquisition content acquired corresponding to the data acquisition request; the first acquisition content comes from a first data source;

[0096] A determination module 302 is used to determine a target text keyword whose appearance frequency is greater than a first preset frequency value among the multiple text keywords, and to determine at least one second collected content whose appearance frequency of the target text keyword is greater than a second preset frequency value in at least one second data source; wherein the second preset frequency value is greater than the first preset frequency value;

[0097] The recognition module 303 is used to recognize the image data in the first collected content and the second collected content to obtain a recognition result, determine the target image data of the same object at multiple different shooting angles according to the recognition result, and classify all the target image data of the same object into the same type of image;

[0098] The classification module 304 is used to determine multiple release times of the target image in each category, and classify the target images whose release times have a time interval greater than a preset time threshold into different categories of images, so as to obtain multiple events corresponding to the multiple categories of images;

[0099] The acquisition module 305 is used to determine multiple event keywords based on the multiple events corresponding to the multiple types of images, perform data acquisition through the event keywords to obtain third acquisition content, and obtain a final acquisition result corresponding to the data acquisition request based on the first acquisition content, the second acquisition content and the third acquisition content.

[0100] In some embodiments, the acquisition module is specifically used to:

[0101] According to the multiple events corresponding to the multiple types of images, data collection and data summarization are performed through the AI ​​model to obtain data summary results, and based on the data summary results, multiple event keywords are obtained through AI model analysis;

[0102] According to each of the multiple event keywords, a description text of the event keyword under the corresponding multiple expression dimensions is generated; wherein the multiple expression dimensions include any one or more of the following: a processing method for a target event corresponding to the event keyword, a processing result for the target event, an event summary content for the target event, and an event analysis content for the target event;

[0103] The AI ​​model is used to filter and exclude everyday terms in the description text to obtain a target description text, and data collection is performed based on the target description text to obtain third collected content.

[0104] In some embodiments, the data summary result includes location keywords, time keywords, and collection target keywords; the collection module is further used to:

[0105] Based on the location keyword and the time keyword, a range search is performed through an AI model to obtain first data within a target range; wherein the data format of the first data includes at least one of an image, text, and audio;

[0106] Analyze the data content of the first data, and determine and summarize at least one occurred event described in the first data according to the data content analysis result;

[0107] A corresponding first event keyword is generated according to each of the occurred events, and event keywords irrelevant to the collection target keyword are excluded from the first event keywords to obtain second event keywords.

[0108] In some embodiments, the apparatus further comprises:

[0109] An optimization module is used to generate a corresponding first event keyword according to each of the occurred events, exclude event keywords in the first event keywords that are not related to the collection target keywords, and obtain a second event keyword. Then, based on the second event keyword, a specific search is performed through the AI ​​model to obtain second data; based on the second data, the AI ​​model is trained and optimized to obtain a trained AI model, so as to optimize the search process using the trained AI model.

[0110] In some embodiments, the apparatus further comprises:

[0111] An exclusion module is used to, after obtaining multiple event keywords through AI model analysis based on the data summary result, respond to an exclusion selection operation on the multiple event keywords, determine the event keywords to be excluded corresponding to the exclusion selection operation among the multiple event keywords, and exclude the event keywords to be excluded from the multiple event keywords to obtain the event keywords after the exclusion processing; respond to a key selection operation on the multiple event keywords after the exclusion processing, determine the key event keywords corresponding to the key selection operation among the multiple event keywords after the exclusion processing, and use the events associated with the key event keywords as the key data collection objects in the data collection process.

[0112] In some embodiments, the identification module is specifically used to:

[0113] Determine, in the first collected content and the second collected content, a plurality of target object images whose object image similarity is greater than a preset similarity threshold according to the recognition result, and perform noise elimination processing on the plurality of target object images to obtain target object images after noise elimination;

[0114] The objects in the plurality of noise-processed target object images are determined to be the same object, and target image data for the same object at a plurality of different shooting angles are determined in the first acquisition content and the second acquisition content.

[0115] In some embodiments, the acquisition module is specifically used to:

[0116] Based on the first collected content corresponding to the data collection request, a word segmentation algorithm in machine learning is used to perform natural language recognition to obtain a language recognition result, and a plurality of text keywords are extracted from the first collected content according to the language recognition result;

[0117] Setting a label for the first collected content according to the text keyword, and associating the labeled first collected content with related content based on the text keyword to obtain labeled and mutually associated historical collected data;

[0118] The labels of the target historical collection data whose collection frequency is higher than the preset frequency threshold in the historical collection data are marked, and the probability of the target historical collection data with the marked labels being collected again is reduced.

[0119] The keyword-based data collection device provided in the embodiment of the present application has the same technical features as the keyword-based data collection method provided in the above-mentioned embodiment, so it can also solve the same technical problems and achieve the same technical effects.

[0120] An electronic device provided in an embodiment of the present application is Figure 4 As shown, the electronic device 400 includes a processor 402 and a memory 401, wherein the memory stores a computer program that can be run on the processor, and the processor implements the steps of the method provided in the above embodiment when executing the computer program.

[0121] See also Figure 4 The electronic device further includes: a bus 403 and a communication interface 404, a processor 402, a communication interface 404 and a memory 401 are connected via the bus 403; the processor 402 is used to execute an executable module stored in the memory 401, such as a computer program.

[0122] The memory 401 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 404 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0123] The bus 403 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0124] Among them, the memory 401 is used to store programs, and the processor 402 executes the program after receiving the execution instruction. The method executed by the device defined by the process disclosed in any embodiment of the present application can be applied to the processor 402 or implemented by the processor 402.

[0125] The processor 402 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 402. The above processor 402 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined to execute. The software module may be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 401, and the processor 402 reads the information in the memory 401 and completes the steps of the above method in combination with its hardware.

[0126] Corresponding to the above-mentioned keyword-based data collection method, an embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the steps of the above-mentioned keyword-based data collection method.

[0127] The keyword-based data acquisition device provided in the embodiment of the present application can be specific hardware on the device or software or firmware installed on the device. The device provided in the embodiment of the present application, its implementation principle and the technical effect produced are the same as those in the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the device embodiment can refer to the corresponding contents in the aforementioned method embodiment. It can be clearly understood by technicians in the relevant field that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0128] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0129] For another example, the flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the device, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0130] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0131] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0132] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the keyword-based data collection method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0133] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0134] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solution of the present application, rather than to limit it. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the aforementioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solution recorded in the aforementioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solution deviate from the scope of the technical solution of the embodiment of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A keyword-based data collection method, characterized in that: The method comprises: Obtaining a data collection request, and extracting a plurality of text keywords based on first collected content corresponding to the data collection request, wherein the first collected content is derived from a first data source; Determine a target text keyword whose appearance frequency is greater than a first preset frequency value among the multiple text keywords, and determine at least one second collected content whose appearance frequency of the target text keyword is greater than a second preset frequency value in at least one second data source; wherein the second preset frequency value is greater than the first preset frequency value; Recognize the image data in the first collected content and the second collected content to obtain a recognition result, determine target image data of the same object at multiple different shooting angles according to the recognition result, and classify all the target image data of the same object into the same type of image; Determine multiple release times of the target image for each category, and classify the target images whose release times have a time interval greater than a preset time threshold into different categories of images, to obtain multiple events corresponding to the multiple categories of images; Determine a plurality of event keywords according to the plurality of events corresponding to the plurality of types of images, and collect data using the event keywords to obtain third collected content; A final collection result corresponding to the data collection request is obtained based on the first collection content, the second collection content, and the third collection content.

2. The keyword-based data collection method according to claim 1, characterized in that: The determining of a plurality of event keywords according to the plurality of events corresponding to the plurality of types of images, and performing data collection through the event keywords to obtain third collection content includes: According to the multiple events corresponding to the multiple types of images, data collection and data summarization are performed through the AI ​​model to obtain data summary results, and based on the data summary results, multiple event keywords are obtained through AI model analysis; According to each of the multiple event keywords, a description text of the event keyword under the corresponding multiple expression dimensions is generated; wherein the multiple expression dimensions include any one or more of the following: a processing method for a target event corresponding to the event keyword, a processing result for the target event, an event summary content for the target event, and an event analysis content for the target event; The AI ​​model is used to filter and exclude everyday terms in the description text to obtain a target description text, and data collection is performed based on the target description text to obtain third collected content.

3. The keyword-based data collection method according to claim 2, characterized in that: The data summary result includes location keywords, time keywords, and collection target keywords; the multiple event keywords obtained by analyzing the data summary result through the AI ​​model include: Based on the location keyword and the time keyword, a range search is performed through an AI model to obtain first data within a target range; wherein the data format of the first data includes at least one of an image, text, and audio; Analyze the data content of the first data, and determine and summarize at least one occurred event described in the first data according to the data content analysis result; A corresponding first event keyword is generated according to each of the occurred events, and event keywords irrelevant to the collection target keyword are excluded from the first event keywords to obtain second event keywords.

4. The keyword-based data collection method according to claim 3, characterized in that: After the step of generating a corresponding first event keyword according to each of the occurred events and excluding event keywords that are irrelevant to the acquisition target keyword from the first event keywords to obtain the second event keywords, the method further includes: Perform a specific search based on the second event keyword through the AI ​​model to obtain second data; The AI ​​model is trained and optimized based on the second data to obtain a trained AI model, so as to optimize the search process using the trained AI model.

5. The keyword-based data collection method according to claim 2, characterized in that: After the step of obtaining a plurality of event keywords through AI model analysis based on the data summary result, the method further includes: In response to an exclusion selection operation on the multiple event keywords, determining an event keyword to be excluded corresponding to the exclusion selection operation among the multiple event keywords, and performing exclusion processing on the event keyword to be excluded among the multiple event keywords to obtain an event keyword after the exclusion processing; In response to a key selection operation on multiple event keywords after the exclusion process, the key event keyword corresponding to the key selection operation is determined among the multiple event keywords after the exclusion process, and the event associated with the key event keyword is used as the key data collection object in the data collection process.

6. The keyword-based data collection method according to claim 1, characterized in that: The determining, according to the recognition result, target image data of the same object at a plurality of different shooting angles comprises: Determine, in the first collected content and the second collected content, a plurality of target object images whose object image similarity is greater than a preset similarity threshold according to the recognition result, and perform noise elimination processing on the plurality of target object images to obtain target object images after noise elimination; The objects in the plurality of noise-processed target object images are determined to be the same object, and target image data for the same object at a plurality of different shooting angles are determined in the first acquisition content and the second acquisition content.

7. The keyword-based data collection method according to claim 1, characterized in that: The step of extracting a plurality of text keywords based on the first collected content corresponding to the data collection request comprises: Based on the first collected content corresponding to the data collection request, a word segmentation algorithm in machine learning is used to perform natural language recognition to obtain a language recognition result, and a plurality of text keywords are extracted from the first collected content according to the language recognition result; Setting a label for the first collected content according to the text keyword, and associating the labeled first collected content with related content based on the text keyword to obtain labeled and mutually associated historical collected data; The labels of the target historical collection data whose collection frequency is higher than the preset frequency threshold in the historical collection data are marked, and the probability of the target historical collection data with the marked labels being collected again is reduced.

8. A data collection device based on keywords, characterized in that: include: An acquisition module, used to acquire a data acquisition request, and extract a plurality of text keywords based on the first acquisition content acquired corresponding to the data acquisition request; The first collected content comes from a first data source; A determination module, configured to determine a target text keyword whose occurrence frequency is greater than a first preset frequency value among the plurality of text keywords, and to determine at least one second collected content whose occurrence frequency of the target text keyword is greater than a second preset frequency value in at least one second data source; wherein the second preset frequency value is greater than the first preset frequency value; an identification module, configured to identify the image data in the first collected content and the second collected content to obtain an identification result, determine target image data of the same object at multiple different shooting angles according to the identification result, and classify all the target image data of the same object into images of the same type; A classification module is used to determine multiple release times of the target image in each category, and classify the target images whose time intervals among the multiple release times are greater than a preset time threshold into different categories of images, so as to obtain multiple events corresponding to the multiple categories of images; The acquisition module is used to determine multiple event keywords based on the multiple events corresponding to the multiple types of images, perform data acquisition through the event keywords, obtain third acquisition content, and obtain a final acquisition result corresponding to the data acquisition request based on the first acquisition content, the second acquisition content and the third acquisition content.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the method according to any one of claims 1 to 7.

Images