Robot-based data grabbing method, device and computer equipment

Abstract

The application relates to a robot-based data grabbing method and device, computer equipment, a storage medium and a computer program product, and relates to the field of artificial intelligence. The method comprises the following steps: obtaining a data topic of to-be-grabbed data, wherein the data topic comprises a to-be-grabbed data area, to-be-grabbed data timeliness and to-be-grabbed data types; positioning a data grabbing platform corresponding to the to-be-grabbed data area; grabbing collected data within the to-be-grabbed data timeliness on the data grabbing platform according to the to-be-grabbed data types; performing redundancy processing and thumbnail processing on the collected data to obtain corresponding target data; and generating a data grabbing report of the to-be-grabbed data according to the target data. The method can improve the accuracy of data grabbing.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a robot-based data grasping method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] Against the backdrop of the big data era, all kinds of information and data are increasing at an unimaginable speed. Data explosion and data overload have plunged people into a new dilemma in the information age.

[0003] Due to the explosive growth of internet data resources and the increasing power of data collection technologies, coupled with the urgent need for targeted data collection from various users, how to quickly and effectively obtain the data needed by users from massive data resources has become a critical issue that urgently needs to be addressed. Currently, users often obtain the data they need by manually searching and filtering. However, this method is not only time-consuming but also prone to finding incorrect data, resulting in low accuracy in data collection. Summary of the Invention

[0004] Therefore, it is necessary to provide a robot-based data grasping method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of data grasping in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a robot-based data acquisition method. The method includes:

[0006] Obtain the data topic of the data to be crawled, wherein the data topic includes the data area to be crawled, the timeliness of the data to be crawled, and the data type to be crawled;

[0007] Locate the data capture platform corresponding to the data area to be captured;

[0008] Based on the data type to be crawled, collect the data within the time frame of the data to be crawled on the data crawling platform;

[0009] The collected data is subjected to redundancy and abbreviation processing to obtain the corresponding target data, and a data capture report of the data to be captured is generated based on the target data.

[0010] In one embodiment, the step of retrieving collected data within the timeframe of the data to be retrieved on the data retrieval platform according to the data type to be retrieved includes:

[0011] The risk level of the data to be crawled is determined based on the type of data to be crawled.

[0012] Based on the aforementioned level of risk, select an appropriate platform verification method;

[0013] The platform verification method is used to perform platform verification in order to enter the data capture platform and capture the collected data within the time frame of the data to be captured.

[0014] In one embodiment, the step of capturing the collected data within the time frame of the data to be captured includes:

[0015] The corresponding data crawling model is determined based on the data type to be crawled, wherein the data crawling model includes a table data type crawling model, a text data type crawling model, and an image data type crawling model;

[0016] The table data type crawling model, the text data type crawling model, and the image data type crawling model are used to crawl the collected data within the time period of the data to be crawled.

[0017] In one embodiment, the process of performing redundancy processing and abbreviation on the collected data to obtain the corresponding target data includes:

[0018] The feature vector set of the collected data is generated using a vector space model;

[0019] The feature vectors in the feature vector set are compared and judged using a preset similarity algorithm to obtain the comparison result of the feature vectors;

[0020] Based on the comparison results, redundant data is removed from the collected data to obtain standard data of the collected data.

[0021] The standard data is abbreviated to obtain the target data.

[0022] In one embodiment, the step of abbreviating the standard data to obtain the target data includes:

[0023] A data abbreviation model is constructed based on a conditional random field, and the standard data is abbreviated using the data abbreviation model to obtain simplified data of the standard data.

[0024] The credibility of the simplified data is calculated using a preset forward-backward algorithm, and the simplified data corresponding to the maximum credibility is selected as the target data.

[0025] In one embodiment, generating a data scraping report of the data to be scraped based on the target data includes:

[0026] The target data is segmented using natural language processing techniques to obtain a candidate word sequence;

[0027] The candidate word sequence is classified using a classifier to obtain a keyword sequence, wherein the classifier is trained based on a preset standard word group;

[0028] The keyword sequence is aggregated to generate the data scraping report.

[0029] Secondly, this application also provides a robot-based data grasping device. Applied to a data grasping robot, the device includes:

[0030] The data topic acquisition module is used to acquire the data topic of the data to be crawled, wherein the data topic includes the data area to be crawled, the timeliness of the data to be crawled, and the data type to be crawled;

[0031] The data capture platform positioning module is used to locate the data capture platform corresponding to the data area to be captured;

[0032] The data capture module is used to capture collected data within the time frame of the data to be captured on the data capture platform according to the data type to be captured.

[0033] The data processing module is used to perform redundancy processing and abbreviation processing on the collected data to obtain the corresponding target data, and generate a data capture report of the data to be captured based on the target data.

[0034] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0035] Obtain the data topic of the data to be crawled, wherein the data topic includes the data area to be crawled, the timeliness of the data to be crawled, and the data type to be crawled;

[0036] Locate the data capture platform corresponding to the data area to be captured;

[0037] Based on the data type to be crawled, collect the data within the time frame of the data to be crawled on the data crawling platform;

[0038] The collected data is subjected to redundancy and abbreviation processing to obtain the corresponding target data, and a data capture report of the data to be captured is generated based on the target data.

[0039] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0040] Obtain the data topic of the data to be crawled, wherein the data topic includes the data area to be crawled, the timeliness of the data to be crawled, and the data type to be crawled;

[0041] Locate the data capture platform corresponding to the data area to be captured;

[0042] Based on the data type to be crawled, collect the data within the time frame of the data to be crawled on the data crawling platform;

[0043] The collected data is subjected to redundancy and abbreviation processing to obtain the corresponding target data, and a data capture report of the data to be captured is generated based on the target data.

[0044] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0045] Obtain the data topic of the data to be crawled, wherein the data topic includes the data area to be crawled, the timeliness of the data to be crawled, and the data type to be crawled;

[0046] Locate the data capture platform corresponding to the data area to be captured;

[0047] Based on the data type to be crawled, collect the data within the time frame of the data to be crawled on the data crawling platform;

[0048] The collected data is subjected to redundancy and abbreviation processing to obtain the corresponding target data, and a data capture report of the data to be captured is generated based on the target data.

[0049] The aforementioned robot-based data acquisition method, apparatus, computer equipment, storage medium, and computer program products, compared to the current method of users manually searching and filtering to obtain the data they need, are applied to data acquisition robots. This eliminates the need for manual searching and filtering, achieving automated data acquisition and improving efficiency. The entire solution first obtains the data topic of the data to be acquired, which includes the data area, the timeframe, and the data type. Then, it locates the data acquisition platform corresponding to the data area, and based on the data type, acquires the collected data within the timeframe of the data area on the data acquisition platform. Finally, it performs redundancy and abbreviated processing on the collected data to remove duplicate data and simplify the data, making the target data more accurate and clear. A data acquisition report is then generated based on the target data. The entire solution performs data scraping based on the type of data to be scraped, the timeliness of the data to be scraped, and the area of ​​the data to be scraped, achieving the goal of accurate data scraping. It can extract the data needed by users from massive amounts of data, better meet users' data needs, and thus improve the accuracy of data scraping. Attached Figure Description

[0050] Figure 1 This is an application scenario diagram of a robot-based data grasping method in one embodiment;

[0051] Figure 2 This is a flowchart illustrating a robot-based data acquisition method in one embodiment;

[0052] Figure 3 This is a schematic diagram of the platform verification process in one embodiment;

[0053] Figure 4 This is a flowchart illustrating the process of determining a data scraping model in one embodiment;

[0054] Figure 5 This is a flowchart illustrating the redundancy handling process in one embodiment;

[0055] Figure 6 This is a schematic diagram of the abbreviation process in one embodiment;

[0056] Figure 7 This is a schematic diagram of the data scraping report generation process in one embodiment;

[0057] Figure 8 This is a structural block diagram of a robot-based data grasping device in one embodiment;

[0058] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0060] The robot-based data grasping method provided in this disclosure can be applied to, for example... Figure 1 In the application environment shown, the execution entity can be a data-grabbing robot 102, which can be connected to a terminal device 104. Specifically, the user sends a data-grabbing signal to the data-grabbing robot 102 through the terminal device 104. After parsing the data-grabbing signal, the data-grabbing robot 102 obtains the data topic of the data to be grabbed. The data topic includes the area of ​​the data to be grabbed, the time frame of the data to be grabbed, and the data type of the data to be grabbed. The data-grabbing robot 102 then locates the corresponding data-grabbing platform 106 based on the area of ​​the data to be grabbed. Based on the data type of the data to be grabbed, the robot grabs the collected data within the time frame of the data to be grabbed on the data-grabbing platform 106. Finally, the data-grabbing robot 102 performs redundancy processing and abbreviated processing on the collected data to obtain the corresponding target data and generates a data-grabbing report of the data to be grabbed based on the target data. The terminal device 104 can be any device capable of intelligent input and output and signal transmission to the data-grabbing robot 102, such as a mobile terminal, which can be a smartphone, tablet computer, in-vehicle computer, wearable smart device, etc.

[0061] In one embodiment, such as Figure 2 As shown, a robot-based data grasping method is provided, applied to a data grasping robot. In this embodiment, the method includes the following steps:

[0062] Step S202: Obtain the data topic of the data to be scraped.

[0063] The data topic includes the data area to be crawled, the time frame of the data to be crawled, and the data type to be crawled; the data to be crawled is the information that needs to be obtained for targeted crawling.

[0064] Specifically, users can send data acquisition signals to the data-grabbing robot via their mobile terminals. These signals carry the data subject of the data to be captured. After receiving the data acquisition signals, the data-grabbing robot parses them to obtain the data area to be captured, the time frame of the data to be captured, and the data type to be captured, in order to perform subsequent data capture.

[0065] As an example, the mobile terminal can send data acquisition signals to the data-grabbing robot in any of the following ways: 5G network, 4G network, 3G network, WIFI, or Bluetooth.

[0066] Step S204: Locate the data capture platform corresponding to the data area to be captured.

[0067] The data region to be crawled can refer to the area where the data is to be crawled, such as the provincial, municipal, county, or town level. The data crawling platform can refer to the management platform on which the data crawling robot can crawl data. The data crawling platform corresponds to the data region to be crawled, that is, different data regions to be crawled have corresponding data crawling platforms, such as provincial, municipal, county, or town management platforms.

[0068] Specifically, the correspondence between the data crawling platform and the data area to be crawled can be pre-written into the data crawling robot. When the data crawling robot obtains the data area to be crawled, the data crawling platform to be crawled can be determined based on the correspondence.

[0069] As an example, the correspondence between the data crawling platform and the data region to be crawled can be established based on the region name. For example, if Changsha City and the Changsha City Management Platform have the same region name, then a correspondence can be established between the two.

[0070] Step S206: Based on the data type to be crawled, crawl the collected data within the time frame of the data to be crawled on the data crawling platform.

[0071] The timeframe of the data to be crawled can refer to the start and end time range of the data to be crawled. The timeframe of the data to be crawled can be used to prompt the data crawling robot to crawl data within a specific time range. The data type of the data to be crawled can include tabular data, text data, and image data.

[0072] Specifically, after the data crawling robot locates the data crawling platform, it can perform platform verification. Once the platform verification is successful, the data crawling robot can enter the data crawling platform, filter out data that matches the type to be crawled, and filter out data whose generation time is within the time limit of the data to be crawled from these data. Then, it can crawl the finally filtered data to obtain the collected data.

[0073] Step S208: Perform redundancy processing and abbreviation processing on the collected data to obtain the corresponding target data, and generate a data capture report of the data to be captured based on the target data.

[0074] The redundancy processing can refer to removing redundancy from the collected data, that is, removing duplicate data; the abbreviation processing can refer to simplifying the collected data, that is, reducing the data volume of the collected data while ensuring the integrity of the collected data; the data capture report refers to the report obtained after segmenting the target data and extracting keywords.

[0075] Specifically, the feature vector set of the collected data can be obtained, and the similarity comparison results among all feature vectors in the feature vector set can be calculated. Based on the comparison results, redundancy processing can be performed on the collected data to remove duplicate data and obtain standard data. The standard data can then be abbreviated to obtain simplified data. Finally, the data with the highest credibility in the simplified data can be selected as the target data. The target data can then be segmented to obtain a candidate word sequence. Based on the candidate word sequence, keywords can be extracted to obtain the keyword sequence of the target data. Finally, the keyword sequence is aggregated to form a data crawling report.

[0076] This embodiment is applied to a data crawling robot, eliminating the need for manual searching and filtering, and achieving automated data crawling, thus improving the efficiency of data crawling. The entire embodiment first obtains the data topic of the data to be crawled, wherein the data topic includes the data region to be crawled, the data timeframe to be crawled, and the data type to be crawled. Then, it locates the data crawling platform corresponding to the data region to be crawled, and then, according to the data type to be crawled, it crawls the collected data within the data timeframe on the data crawling platform. Finally, it performs redundancy processing and abbreviated processing on the collected data to remove duplicate data and simplify the collected data, making the target data more accurate and clear, and generates a data crawling report based on the target data. The entire solution performs data crawling based on the data type, data timeframe, and data region to be crawled, achieving the goal of accurate data crawling. It can extract the data needed by users from massive amounts of data, better meeting users' data needs, thereby improving the accuracy of data crawling.

[0077] In one embodiment, such as Figure 3 As shown, the step of capturing collected data within the time frame of the data to be captured on the data capture platform according to the data type to be captured includes:

[0078] Step S302: Determine the risk level of the data to be crawled based on the data type to be crawled.

[0079] The risk level can be used to characterize the importance of the data to be crawled, that is, the higher the risk level, the more important the data. The risk level is related to the platform's verification method. That is, for data to be crawled with different risk levels, the data crawling platform uses different verification methods when the data crawling robot enters.

[0080] Specifically, the risk level of the data to be crawled is determined based on the data type to be crawled. The risk level corresponding to the data type to be crawled can be set by relevant personnel based on experience or actual application scenarios. Taking the data types to be crawled as tabular data, text data, and image data as examples, the risk level can be set as follows: tabular data > image data > text data. The risk level can be represented by high, medium, and low.

[0081] As an example, the risk level of the data to be crawled can also be determined based on the keywords of the data to be crawled. That is, keywords are extracted from the data to be crawled, and it is detected whether there are sensitive word groups in the extracted keywords. If the number of sensitive word groups exceeds a preset maximum threshold, the risk level of the data to be crawled is determined to be high. If the number of sensitive word groups is lower than a preset minimum threshold, the risk level of the data to be crawled is determined to be low. If the number of sensitive word groups is between the preset maximum threshold and the preset minimum threshold, the risk level of the data to be crawled is determined to be medium.

[0082] Step S304: Based on the risk level, select an appropriate platform verification method.

[0083] The platform verification method can refer to the verification method used by the data crawling robot before it crawls the data crawling platform, and may include any one or more of sliding verification, formula verification, and image verification.

[0084] Specifically, depending on the level of risk, the corresponding platform verification method is selected. For data to be crawled with a high level of risk, the verification method for the data crawling robot to enter the platform can be sliding verification, formula verification, and image verification. For data to be crawled with a medium level of risk, the verification method for the data crawling robot to enter the platform can be any combination of two verification methods. For data to be crawled with a low level of risk, the verification method for the data crawling robot to enter the platform can be any one of the above platform verification methods.

[0085] Step S306: Perform platform verification using the platform verification method to enter the data capture platform and capture the collected data within the time frame of the data to be captured.

[0086] Specifically, if the platform verification method is sliding verification, the data crawling robot will perform a sliding operation to verify; if the platform verification method is formula verification, the data crawling robot will calculate the formula result to verify; if the platform verification method is image verification, the data crawling robot will select the correct image to verify. Only when all platform verifications are passed can the data crawling robot enter the data crawling platform to perform data crawling.

[0087] In this embodiment, the risk level of the data to be crawled is first determined based on the type of data to be crawled. Then, based on the risk level, an appropriate platform verification method is selected, and platform verification is performed to enter the data crawling platform for data crawling. By assigning different platform verification methods to data with different risk levels, data leakage is avoided, thereby improving the security of data crawling.

[0088] In one embodiment, such as Figure 4 As shown, the process of capturing the collected data within the time frame of the data to be captured includes:

[0089] Step S402: Determine the corresponding data crawling model according to the data type to be crawled, wherein the data crawling model includes a table data type crawling model, a text data type crawling model, and an image data type crawling model.

[0090] Step S404: Use the table data type crawling model, the text data type crawling model, and the image data type crawling model to crawl the collected data within the time frame of the data to be crawled.

[0091] The data crawling model can refer to a model for crawling data, which may include, but is not limited to, decision tree models, deep learning models, autoregressive models, etc. The tabular data crawling model can be a data crawling model for crawling tabular data, the text data crawling model can be a data crawling model for crawling text data, and the image data crawling model can be a data crawling model for crawling image data.

[0092] Specifically, if the data type to be crawled is tabular data, then the tabular data crawling model is used to crawl the tabular data; if the data type to be crawled is text data, then the text data crawling model is used to crawl the text data; and if the data type to be crawled is image data, then the image data crawling model is used to crawl the image data.

[0093] Compared to using a single data crawling model, this embodiment utilizes different data crawling models to crawl different types of data, making data crawling more targeted. This avoids the problem of low crawling accuracy caused by a large amount of data when using a single data crawling model, thereby improving the accuracy of data crawling.

[0094] In one embodiment, such as Figure 5 As shown, the process of performing redundancy processing and abbreviation on the collected data to obtain the corresponding target data includes:

[0095] Step S502: Generate the feature vector set of the collected data using the vector space model.

[0096] Step S504: Use a preset similarity algorithm to compare and judge the feature vectors in the feature vector set to obtain the comparison result of the feature vectors.

[0097] The vector space model refers to simplifying the collected data stored in text form into vector operations in a vector space, and expressing the similarity of feature vectors in terms of spatial similarity, which is more intuitive and easier to understand. Since the most commonly used similarity measure in text data is cosine distance, the preset similarity algorithm can be an algorithm that calculates the cosine distance of each feature vector.

[0098] Specifically, the collected data first exists in the form of text data. The collected data is input into a vector space model to convert between text data and vector data, thereby obtaining a feature vector set of the converted collected data. Each feature vector in the feature vector set is obtained, and the cosine distance between each feature vector is calculated using a preset similarity algorithm to represent the similarity result between each feature vector. Based on the similarity result, each feature vector is compared and judged to obtain the comparison result.

[0099] As an example, after calculating the similarity results between each feature vector, a comparison can be made between the similarity results and a preset similarity threshold. The closer the similarity result is to the preset similarity threshold, the more similar the two feature vectors are, and the smaller the individual differences. The larger the gap between the similarity result and the preset similarity threshold, the more dissimilar the two feature vectors are, indicating that they represent opposite directions. The preset similarity threshold can be set manually.

[0100] Step S506: Redundancy is removed from the collected data based on the comparison results to obtain standard data of the collected data.

[0101] The comparison result can refer to the positive or negative result of the difference between the similarity result and the preset similarity threshold.

[0102] Specifically, if the comparison result is positive, that is, the difference between the similarity result and the preset similarity threshold is positive, it means that the two feature vectors are in the same direction, that is, they are similar. If the comparison result is negative, that is, the difference between the similarity result and the preset similarity threshold is negative, it means that the two feature vectors are in opposite directions, that is, they are not similar. Similar feature vectors are selected and the collected data corresponding to these feature vectors are removed to obtain standard data.

[0103] Step S508: The standard data is abbreviated to obtain the target data.

[0104] Specifically, the standard data is abbreviated using a data abbreviation model to obtain the corresponding simplified data. The credibility of the simplified data is then calculated, and the simplified data with the highest credibility is used as the target data.

[0105] In this embodiment, by eliminating redundancy in the collected data, the number of storage media required is greatly reduced, saving storage space and thus reducing costs. It can even make the cost of a hard disk-based storage system lower than that of a tape library, while providing better performance. At the same time, it can improve write performance and save network bandwidth.

[0106] In one embodiment, such as Figure 6 As shown, the process of abbreviating the standard data to obtain the target data includes:

[0107] Step S602: Construct a data abbreviation model based on the conditional random field, and use the data abbreviation model to abbreviate the standard data to obtain simplified data of the standard data.

[0108] The conditional random field can be an undirected graphical model or a Markov random field, that is, an undirected graphical model that calculates the conditional probability of the output node given the input node, mainly used as a statistical framework model for labeling and segmenting serialized data; the data abbreviation model or information abbreviation model is for simplifying standard data, and it can be generated based on the conditional random field and preset empirical rules.

[0109] Specifically, a data abbreviated model is constructed based on conditional random fields and preset empirical rules. Standard data is input into the data abbreviated model, and key data in the standard data is extracted and used as simplified data of the standard data.

[0110] Step S604: Calculate the credibility of the simplified data using a preset forward-backward algorithm, and select the simplified data corresponding to the maximum credibility as the target data.

[0111] The preset forward and backward algorithms are collectively referred to as the forward algorithm and the backward algorithm, both of which can calculate the probability of the observation sequence of the Hidden Markov Model.

[0112] Specifically, the probability of the observation sequence of simplified data is calculated using a preset forward-backward algorithm. The credibility of the simplified data is determined based on the probability of the observation sequence. That is, when the probability of the observation sequence is greater than the preset probability threshold, it means that the probability of the observation sequence of the simplified data is credible and the credibility is high. When the probability of the observation sequence is less than the preset probability threshold, it means that the probability of the observation sequence of the simplified data is unreliable and the credibility is low. The simplified data with the highest credibility is taken as the target data.

[0113] In this embodiment, the standard data is abbreviated using a constructed data abbreviation model, which can extract key information from the standard data, reduce data redundancy, and calculate the credibility of the simplified data generated by the preset forward-backward algorithm. The simplified data with the highest credibility is used as the target data, making the target data more reliable and more accurate.

[0114] In one embodiment, such as Figure 7 As shown, the step of generating a data scraping report of the data to be scraped based on the target data includes:

[0115] Step S702: Using natural language processing technology, the target data is segmented to obtain a candidate word sequence.

[0116] Step S704: Use a classifier to classify the candidate word sequence to obtain a keyword sequence.

[0117] Step S706: Aggregate the keyword sequence to generate the data capture report.

[0118] The Natural Language Processing (NLP) technology refers to the analysis and processing of language information; the classifier is trained based on a preset standard phrase, which can be a phrase that needs to be written into the data capture report.

[0119] Specifically, natural language processing technology is used to segment the data to be crawled. This segmentation can include character segmentation and semantic segmentation, resulting in character segmentation sequences and semantic segmentation sequences. The character segmentation sequences are filtered out, and the semantic segmentation sequences are used as candidate word sequences. A classifier is used to classify the candidate word sequences, and each candidate word belonging to a preset standard word group is used as a keyword sequence, while each candidate word not belonging to a preset standard word group is used as a non-keyword sequence. The extracted keyword sequences are aggregated and summarized to generate a data crawling report.

[0120] In this embodiment, natural language processing technology is used to segment the target data to ensure the accuracy of the obtained candidate word sequence. Then, a classifier trained based on preset standard word groups is used to extract the keyword sequence, thereby aggregating it into a data crawling report that meets the user's requirements.

[0121] In one embodiment, the data crawling robot can be an investment project data crawling robot, the data to be crawled can be investment project data, and the data crawling platform can be an investment project data crawling platform. Specifically, the investment project data crawling robot obtains the data region to be crawled, the time frame of the data to be crawled, and the data type of the data to be crawled from the investment project data. It locates the investment project data crawling platform corresponding to the data region to be crawled, and according to the data type of the data to be crawled, it crawls the investment project data collected within the time frame of the data to be crawled from the investment project data crawling platform. It performs redundancy processing and abbreviated processing on the investment project data collected to obtain the corresponding investment project target data. Based on the investment project target data, it generates an investment project data crawling report, which may include data such as project name, project content, total investment amount, and financing entity.

[0122] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0123] Based on the same inventive concept, this application also provides a robot-based data grasping device for implementing the robot-based data grasping method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more robot-based data grasping device embodiments provided below can be found in the limitations of the robot-based data grasping method described above, and will not be repeated here.

[0124] In one embodiment, such as Figure 8 As shown, a robot-based data grasping device is provided, applied to a data grasping robot, including: a data topic acquisition module 802, a grasping platform positioning module 804, a data grasping module 806, and a data processing module 808, wherein:

[0125] The data topic acquisition module 802 is used to acquire the data topic of the data to be crawled, wherein the data topic includes the data area to be crawled, the timeliness of the data to be crawled, and the data type to be crawled;

[0126] The data capture platform positioning module 804 is used to locate the data capture platform corresponding to the data area to be captured;

[0127] The data capture module 806 is used to capture collected data within the time frame of the data to be captured on the data capture platform according to the data type to be captured.

[0128] The data processing module 808 is used to perform redundancy processing and abbreviation processing on the collected data to obtain the corresponding target data, and generate a data capture report of the data to be captured based on the target data.

[0129] In one embodiment, the data capture module 806 further includes:

[0130] A risk determination unit is used to determine the risk level of the data to be crawled based on the type of data to be crawled.

[0131] The verification method selection unit is used to select an appropriate platform verification method based on the risk level.

[0132] The platform verification unit is used to perform platform verification using the platform verification method, so as to enter the data capture platform to capture the collected data within the time limit of the data to be captured.

[0133] In one embodiment, the platform verification unit is further configured to:

[0134] The corresponding data crawling model is determined based on the data type to be crawled, wherein the data crawling model includes a table data type crawling model, a text data type crawling model, and an image data type crawling model;

[0135] The table data type crawling model, the text data type crawling model, and the image data type crawling model are used to crawl the collected data within the time period of the data to be crawled.

[0136] In one embodiment, the data processing module 808 further includes:

[0137] A feature vector set generation unit is used to generate a feature vector set of the collected data using a vector space model.

[0138] The comparison unit is used to compare and judge the feature vectors in the feature vector set using a preset similarity algorithm, and obtain the comparison result of the feature vectors.

[0139] A redundancy processing unit is used to remove redundancy from the collected data based on the comparison results to obtain standard data of the collected data.

[0140] A reduction processing unit is used to reduce the standard data to obtain the target data.

[0141] In one embodiment, the abbreviation processing unit is further configured to:

[0142] A data abbreviation model is constructed based on a conditional random field, and the standard data is abbreviated using the data abbreviation model to obtain simplified data of the standard data.

[0143] The credibility of the simplified data is calculated using a preset forward-backward algorithm, and the simplified data corresponding to the maximum credibility is selected as the target data.

[0144] In one embodiment, the data processing module 808 is further configured to:

[0145] The target data is segmented using natural language processing techniques to obtain a candidate word sequence;

[0146] The candidate word sequence is classified using a classifier to obtain a keyword sequence, wherein the classifier is trained based on a preset standard word group;

[0147] The keyword sequence is aggregated to generate the data scraping report.

[0148] The modules in the aforementioned robot-based data grasping device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0149] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores item recommendation data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a robot-based data grasping method.

[0150] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0151] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0152] The process involves: acquiring the data topic of the data to be crawled, wherein the data topic includes the data region to be crawled, the time frame of the data to be crawled, and the data type to be crawled; locating the data crawling platform corresponding to the data region to be crawled; crawling the collected data within the time frame of the data to be crawled on the data crawling platform according to the data type to be crawled; performing redundancy processing and abbreviation processing on the collected data to obtain the corresponding target data; and generating a data crawling report of the data to be crawled based on the target data.

[0153] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0154] The risk level of the data to be crawled is determined based on the data type to be crawled; based on the risk level, an appropriate platform verification method is selected; platform verification is performed using the platform verification method to enter the data crawling platform to crawl the collected data within the time frame of the data to be crawled.

[0155] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0156] The corresponding data crawling model is determined based on the data type to be crawled, wherein the data crawling model includes a table data type crawling model, a text data type crawling model, and an image data type crawling model; the table data type crawling model, the text data type crawling model, and the image data type crawling model are used to crawl the collected data within the time period of the data to be crawled.

[0157] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0158] A feature vector set of the collected data is generated using a vector space model; a preset similarity algorithm is used to compare and judge the feature vectors in the feature vector set to obtain the comparison result of the feature vectors; redundancy is removed from the collected data according to the comparison result to obtain the standard data of the collected data; the standard data is abbreviated to obtain the target data.

[0159] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0160] A data abbreviation model is constructed based on a conditional random field. The standard data is abbreviated using the data abbreviation model to obtain simplified data of the standard data. The credibility of the simplified data is calculated using a preset forward-backward algorithm, and the simplified data corresponding to the maximum credibility is selected as the target data.

[0161] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0162] Using natural language processing technology, the target data is segmented to obtain a candidate word sequence; the candidate word sequence is classified using a classifier to obtain a keyword sequence, wherein the classifier is trained based on a preset standard word group; the keyword sequence is aggregated to generate the data capture report.

[0163] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0164] The process involves: acquiring the data topic of the data to be crawled, wherein the data topic includes the data region to be crawled, the time frame of the data to be crawled, and the data type to be crawled; locating the data crawling platform corresponding to the data region to be crawled; crawling the collected data within the time frame of the data to be crawled on the data crawling platform according to the data type to be crawled; performing redundancy processing and abbreviation processing on the collected data to obtain the corresponding target data; and generating a data crawling report of the data to be crawled based on the target data.

[0165] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0166] The risk level of the data to be crawled is determined based on the data type to be crawled; based on the risk level, an appropriate platform verification method is selected; platform verification is performed using the platform verification method to enter the data crawling platform to crawl the collected data within the time frame of the data to be crawled.

[0167] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0168] The corresponding data crawling model is determined based on the data type to be crawled, wherein the data crawling model includes a table data type crawling model, a text data type crawling model, and an image data type crawling model; the table data type crawling model, the text data type crawling model, and the image data type crawling model are used to crawl the collected data within the time period of the data to be crawled.

[0169] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0170] A feature vector set of the collected data is generated using a vector space model; a preset similarity algorithm is used to compare and judge the feature vectors in the feature vector set to obtain the comparison result of the feature vectors; redundancy is removed from the collected data according to the comparison result to obtain the standard data of the collected data; the standard data is abbreviated to obtain the target data.

[0171] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0172] A data abbreviation model is constructed based on a conditional random field. The standard data is abbreviated using the data abbreviation model to obtain simplified data of the standard data. The credibility of the simplified data is calculated using a preset forward-backward algorithm, and the simplified data corresponding to the maximum credibility is selected as the target data.

[0173] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0174] Using natural language processing technology, the target data is segmented to obtain a candidate word sequence; the candidate word sequence is classified using a classifier to obtain a keyword sequence, wherein the classifier is trained based on a preset standard word group; the keyword sequence is aggregated to generate the data capture report.

[0175] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0176] The process involves: acquiring the data topic of the data to be crawled, wherein the data topic includes the data region to be crawled, the time frame of the data to be crawled, and the data type to be crawled; locating the data crawling platform corresponding to the data region to be crawled; crawling the collected data within the time frame of the data to be crawled on the data crawling platform according to the data type to be crawled; performing redundancy processing and abbreviation processing on the collected data to obtain the corresponding target data; and generating a data crawling report of the data to be crawled based on the target data.

[0177] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0178] The risk level of the data to be crawled is determined based on the data type to be crawled; based on the risk level, an appropriate platform verification method is selected; platform verification is performed using the platform verification method to enter the data crawling platform to crawl the collected data within the time frame of the data to be crawled.

[0179] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0180] The corresponding data crawling model is determined based on the data type to be crawled, wherein the data crawling model includes a table data type crawling model, a text data type crawling model, and an image data type crawling model; the table data type crawling model, the text data type crawling model, and the image data type crawling model are used to crawl the collected data within the time period of the data to be crawled.

[0181] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0182] A feature vector set of the collected data is generated using a vector space model; a preset similarity algorithm is used to compare and judge the feature vectors in the feature vector set to obtain the comparison result of the feature vectors; redundancy is removed from the collected data according to the comparison result to obtain the standard data of the collected data; the standard data is abbreviated to obtain the target data.

[0183] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0184] A data abbreviation model is constructed based on a conditional random field. The standard data is abbreviated using the data abbreviation model to obtain simplified data of the standard data. The credibility of the simplified data is calculated using a preset forward-backward algorithm, and the simplified data corresponding to the maximum credibility is selected as the target data.

[0185] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0186] Using natural language processing technology, the target data is segmented to obtain a candidate word sequence; the candidate word sequence is classified using a classifier to obtain a keyword sequence, wherein the classifier is trained based on a preset standard word group; the keyword sequence is aggregated to generate the data capture report.

[0187] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0188] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0189] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0190] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A robot-based data grasping method, applied to a data grasping robot, characterized in that, The method includes: Obtain the data topic of the data to be crawled, wherein the data topic includes the data area to be crawled, the timeliness of the data to be crawled, and the data type to be crawled; Locate the data capture platform corresponding to the data area to be captured; The risk level of the data to be crawled is determined based on the type of data to be crawled. Based on the aforementioned level of risk, select an appropriate platform verification method; Platform verification is performed using the aforementioned platform verification method to access the data scraping platform. The corresponding data scraping model is determined based on the data type to be scraped, wherein the data scraping model includes a table data scraping model, a text data scraping model, and an image data scraping model. The table data type crawling model, the text data type crawling model, and the image data type crawling model are used to crawl the collected data within the time period of the data to be crawled; The collected data is subjected to redundancy and abbreviation processing to obtain the corresponding target data, and a data capture report of the data to be captured is generated based on the target data.

2. The method as described in claim 1, characterized in that, The process of performing redundancy processing and abbreviation on the collected data to obtain the corresponding target data includes: The feature vector set of the collected data is generated using a vector space model; The feature vectors in the feature vector set are compared and judged using a preset similarity algorithm to obtain the comparison result of the feature vectors; Based on the comparison results, redundant data is removed from the collected data to obtain standard data of the collected data. The standard data is abbreviated to obtain the target data.

3. The method as described in claim 2, characterized in that, The process of abbreviating the standard data to obtain the target data includes: A data abbreviation model is constructed based on a conditional random field, and the standard data is abbreviated using the data abbreviation model to obtain simplified data of the standard data. The credibility of the simplified data is calculated using a preset forward-backward algorithm, and the simplified data corresponding to the maximum credibility is selected as the target data.

4. The method as described in claim 1, characterized in that, The step of generating a data scraping report of the data to be scraped based on the target data includes: The target data is segmented using natural language processing techniques to obtain a candidate word sequence; The candidate word sequence is classified using a classifier to obtain a keyword sequence, wherein the classifier is trained based on a preset standard word group; The keyword sequence is aggregated to generate the data scraping report.

5. A robot-based data grasping device, applied to a data grasping robot, characterized in that, The device includes: The data topic acquisition module is used to acquire the data topic of the data to be crawled, wherein the data topic includes the data area to be crawled, the timeliness of the data to be crawled, and the data type to be crawled; The data capture platform positioning module is used to locate the data capture platform corresponding to the data area to be captured; The data crawling module is used to determine the risk level of the data to be crawled based on the data type to be crawled; select an appropriate platform verification method based on the risk level; perform platform verification using the platform verification method to enter the data crawling platform; determine the corresponding data crawling model based on the data type to be crawled, wherein the data crawling model includes a table data crawling model, a text data crawling model, and an image data crawling model; and use the table data crawling model, the text data crawling model, and the image data crawling model to crawl the collected data within the time frame of the data to be crawled. The data processing module is used to perform redundancy processing and abbreviation processing on the collected data to obtain the corresponding target data, and generate a data capture report of the data to be captured based on the target data.

6. The apparatus according to claim 5, characterized in that, The data processing module further includes: A feature vector set generation unit is used to generate a feature vector set of the collected data using a vector space model. The comparison unit is used to compare and judge the feature vectors in the feature vector set using a preset similarity algorithm, and obtain the comparison result of the feature vectors. A redundancy processing unit is used to remove redundancy from the collected data based on the comparison results to obtain standard data of the collected data. A reduction processing unit is used to reduce the standard data to obtain the target data.

7. The apparatus according to claim 6, characterized in that, The abbreviation processing unit is also used for: A data abbreviation model is constructed based on a conditional random field, and the standard data is abbreviated using the data abbreviation model to obtain simplified data of the standard data. The credibility of the simplified data is calculated using a preset forward-backward algorithm, and the simplified data corresponding to the maximum credibility is selected as the target data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images