A data collection processing method, device and medium

Abstract

The application relates to the technical field of electric digital data processing, in particular to a data processing method, a device and a medium. The method comprises the following steps: acquiring collection data of a target website; if the collection data of the target website comprises a picture, acquiring a blur degree of the picture comprised in the collection data of the target website; if the blur degree is less than or equal to a preset blur degree threshold, using an OCR recognition technology to acquire a first keyword set corresponding to the picture, using a multi-modal large model to acquire a second keyword set corresponding to the picture, and determining a union set of the first keyword set and the second keyword set as a target keyword set corresponding to the picture; otherwise, using the multi-modal large model to acquire an initial keyword set corresponding to the picture, and determining a keyword set obtained by processing the initial keyword set as the target keyword set corresponding to the picture. The application can improve the accuracy of the keywords corresponding to the acquired picture.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric digital data processing, in particular to a data collection processing method, device and medium. BACKGROUND

[0002] Network data collection technology is widely used in various fields, which provides rich materials for information analysis and knowledge mining by collecting data containing pictures, texts and tables and other forms from target websites. Among them, pictures as an important data type contain a large amount of information, and accurately extracting the keywords corresponding to the pictures is crucial for data processing and utilization. At present, the commonly used technical means for extracting the keywords corresponding to the pictures includes OCR recognition technology. OCR recognition technology can convert the text content in the pictures into editable text data, so as to extract the keywords, but the technology has high requirements for the clarity of the pictures, and when the pictures have quality problems such as blur, the recognition accuracy will be greatly reduced. How to improve the accuracy of the keywords corresponding to the pictures obtained is a problem to be solved. SUMMARY

[0003] The present application aims to provide a data collection processing method, device and medium to improve the accuracy of the keywords corresponding to the pictures obtained.

[0004] According to a first aspect of the present application, a data collection processing method is provided, which comprises the following steps:

[0005] Obtaining the collection data of the target website; the collection data of the target website includes at least one of pictures, texts and tables.

[0006] If the collection data of the target website includes pictures, obtaining the blur degree of the pictures included in the collection data of the target website.

[0007] If the blur degree of the pictures is less than or equal to a preset blur degree threshold, using OCR recognition technology to obtain a first keyword set corresponding to the pictures, using a multi-modal large model to obtain a second keyword set corresponding to the pictures, and determining the union of the first keyword set and the second keyword set corresponding to the pictures as a target keyword set corresponding to the pictures.

[0008] If the blur degree of the pictures is greater than the preset blur degree threshold, using a multi-modal large model to obtain an initial keyword set corresponding to the pictures, and determining the keyword set obtained by processing the initial keyword set as a target keyword set corresponding to the pictures.

[0009] Further, the processing of the initial keyword set comprises:

[0010] obtaining a confidence degree of each keyword in the initial keyword set corresponding to the picture; the confidence degree of any keyword in the initial keyword set corresponding to the picture is obtained by the multi-modal large model.

[0011] If the collected data of the target website includes text and tables, extracting keywords from the text and tables included in the collected data of the target website.

[0012] Determining a context correlation degree of each keyword in the initial keyword set corresponding to the picture according to the keywords extracted from the text and tables included in the collected data of the target website.

[0013] Determining a comprehensive value of each keyword in the initial keyword set corresponding to the picture according to the confidence degree and the context correlation degree of each keyword in the initial keyword set corresponding to the picture.

[0014] Deleting the keyword in the initial keyword set corresponding to the picture whose comprehensive value is less than a preset comprehensive value threshold.

[0015] Further, determining the context correlation degree of each keyword in the initial keyword set corresponding to the picture according to the keywords extracted from the text and tables included in the collected data of the target website comprises:

[0016] Obtaining the co-occurrence times between the keywords extracted from the text and tables included in the collected data of the target website.

[0017] Constructing a keyword association graph according to the co-occurrence times; the nodes of the keyword association graph are keywords, and the weight of the edge of the keyword association graph is the co-occurrence times of the two keywords connected by the edge.

[0018] Obtaining the co-occurrence times sum of each keyword in the keyword association graph; the co-occurrence times sum of any keyword in the keyword association graph is the sum of the weights of all edges connected by the keyword.

[0019] Obtaining the correlation degree between any keyword in the initial keyword set corresponding to the picture and each candidate keyword in the keyword association graph, and taking the maximum correlation degree as the context correlation degree of the keyword in the initial keyword set corresponding to the picture; the candidate keyword is a keyword whose co-occurrence times sum is greater than or equal to a preset sum threshold.

[0020] Further, determining the comprehensive value of each keyword in the initial keyword set corresponding to the picture according to the confidence degree and the context correlation degree of each keyword in the initial keyword set corresponding to the picture comprises:

[0021] Obtaining a first value of any keyword in the initial keyword set corresponding to the picture according to the confidence degree of the keyword and a preset confidence degree weight.

[0022] The second value of any keyword in the initial keyword set corresponding to the picture is obtained according to the context correlation degree of the keyword and a preset context correlation degree weight.

[0023] The sum of the first value and the second value of any keyword in the initial keyword set corresponding to the picture is determined as the comprehensive value of the keyword.

[0024] Further, the blur degree of the picture is determined according to edges and textures of the picture.

[0025] Further, the use of OCR recognition technology to obtain the first keyword set corresponding to the picture comprises:

[0026] The text corresponding to the picture is obtained using OCR recognition technology.

[0027] The keyword set extracted from the text corresponding to the picture is determined as the first keyword set corresponding to the picture.

[0028] Further, the method further comprises:

[0029] If the collection data of the target website includes text, the target keyword set of the text included in the collection data of the target website is obtained.

[0030] Further, the method further comprises: if the collection data of the target website includes a table, the target keyword set of the table included in the collection data of the target website is obtained.

[0031] According to a second aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned collection data processing method when executing the computer program.

[0032] According to a third aspect of the present application, a computer readable storage medium is provided, which stores a computer program, wherein the computer program is executed by a processor to implement the above-mentioned collection data processing method.

[0033] Compared with the prior art, the present application has at least the following beneficial effects:

[0034] The application obtains the blur degree of the picture in the target website collected data, and classifies according to the blur degree threshold value, when the blur degree of the picture is less than or equal to the preset threshold value, the OCR recognition technology and the multi-modal large model are combined, the OCR recognition technology has the accurate extraction ability of the text information of the clear picture, and the multi-modal large model has the understanding ability of the overall content of the picture, the advantages of the two are complementary, the union of the keyword sets obtained by the two is taken as the target keyword set, which can more comprehensively and accurately cover the key information in the picture, and avoid the limitation of single technology; when the blur degree of the picture is greater than the preset threshold value, the OCR recognition technology is not used for keyword extraction, but only relies on the understanding ability of the multi-modal large model to obtain the initial keyword set, and processes the initial keyword set to obtain the target keyword set, which can avoid the problem that the accuracy of the keywords extracted based on the OCR recognition technology is low, and is beneficial to improve the accuracy and reliability of keyword extraction under the condition that the picture quality is poor. Through this differentiated processing mode, the application can provide a better keyword extraction scheme for pictures with different blur degrees, improve the effect of picture keyword extraction, and thus provide a more reliable basis for subsequent analysis and mining of collected data, and improve the efficiency and quality of the entire data processing process. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0036] Figure 1 The flowchart of the processing method of the collected data provided by the first embodiment of the application is shown in the figure.

[0037] Figure 2 The flowchart of the process of processing the initial keyword set provided by the first embodiment of the application is shown in the figure.

[0038] Figure 3 The flowchart of the process of determining the context correlation degree of each keyword in the initial keyword set corresponding to the picture provided by the first embodiment of the application is shown in the figure.

[0039] Figure 4 The flowchart of the process of determining the comprehensive value of each keyword in the initial keyword set corresponding to the picture provided by the first embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0040] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of the present application.

[0041] Embodiment one:

[0042] According to the present embodiment, as shown in Figure 1 a data collection processing method is provided, the method comprising the following steps:

[0043] S100, acquiring the collection data of the target website; the collection data of the target website comprises at least one of pictures, texts and tables.

[0044] In the present embodiment, the target website is a certain website.

[0045] As a preferred specific embodiment, the acquisition process of the collection data of the target website comprises:

[0046] S110, determining the target crawling time period of the target website and the crawling time step corresponding to the target crawling time period according to the historical update time set of the target website.

[0047] As a preferred specific embodiment, S110 comprises:

[0048] S111, acquiring the historical update time set of the target website.

[0049] In the present embodiment, the update time does not have year, month and day, only time in 24-hour system. For example, the historical update time of the website corresponding to a certain website comprises 8:59, 9:01 and 9:00, etc.

[0050] S112, clustering the historical update time set of the target website to obtain several update time clusters.

[0051] In the present embodiment, the update times belonging to the same cluster have small differences, and the update times belonging to different clusters have large differences.

[0052] Preferably, the density clustering method is selected for clustering, thereby the large time period in which the target website is updated can be acquired. It should be understood that if the historical update time of the target website is usually around 9:00 am and around 2:00 pm, then using the density clustering method can obtain two clusters, one of which comprises the update time around 9:00 am, and the other comprises the update time around 2:00 pm. Those skilled in the art know that the process of density clustering is prior art, which will not be described here.

[0053] S113: Obtain a crawling time step corresponding to any update time cluster according to the update time included in the update time cluster.

[0054] As a preferred embodiment, S113 includes:

[0055] S1131, obtain a distribution histogram corresponding to a specified update time cluster; the horizontal axis of the distribution histogram is the update time, the vertical axis of the distribution histogram is the number of update times, the distribution histogram includes a plurality of rectangular bars, different rectangular bars correspond to different ranges of update times, and different rectangular bars correspond to ranges of update times of equal length; the specified update time cluster is any update time cluster.

[0056] Those skilled in the art know that the process of constructing a histogram is an existing technology and will not be described in detail here.

[0057] Optionally, the length of the range of update time corresponding to any rectangular bar is preset and may be an empirical value, such as 0.5 seconds or 1 second.

[0058] S1132, segmenting the rectangular bars in the distribution histogram according to the number of update moments corresponding to adjacent rectangular bars in the distribution histogram; the difference value of the number of update moments corresponding to adjacent rectangular bars included in any segment is less than or equal to a preset number difference threshold.

[0059] Optionally, the preset quantity difference threshold is an empirical value; if the difference value of the quantity of update moments corresponding to adjacent rectangular bars is less than or equal to the preset quantity difference threshold, it is determined that the difference in the quantity of update moments corresponding to the adjacent rectangular bars is small.

[0060] S1133, determine the range of update time corresponding to any segment as a target crawling time period of the target website, and determine the crawling time step corresponding to the target crawling time period corresponding to the segment according to the mean value of the number of update time corresponding to the rectangular bars included in the segment; the crawling time step corresponding to the target crawling time period corresponding to any segment is negatively correlated with the mean value of the number of update time corresponding to the rectangular bars included in the segment.

[0061] In this embodiment, the start time of any target crawling time period is the earliest update time corresponding to the target crawling time period, and the start time of any target crawling time period is the latest update time corresponding to the target crawling time period.

[0062] As a preferred embodiment, the crawling time step corresponding to the target crawling time period of the i-th segment is T i , T i =T min +(T max -T min )×k / (ui +k), T max and T min are preset maximum and minimum crawling time steps respectively, u i is the number of update time points corresponding to the rectangular bar included in the ith segment, and k is a preset adjustment coefficient, k > 0. Optionally, k is an empirical value, which is used to adjust the sensitivity of the crawling time step to the number of update time points. Based on the preferred specific embodiment, the crawling time step corresponding to the target crawling time period of any segment can be obtained.

[0063] Based on S1131-S1133, a more suitable target crawling time period and corresponding crawling time step can be determined for any segment, which can improve the timeliness of data collection for different update time periods of the target website, and can reduce the resources consumed by collection.

[0064] Based on S111-S113, different more suitable segmented collection strategies can be formulated for the target website, which can improve the timeliness of data collection for the target website, and can reduce the resources consumed by collection.

[0065] S120, in the target crawling time period of the target website, judging whether the data of the target website is updated or not with the crawling time step corresponding to the target crawling time period of the target website as the judgment period, if updated, crawling data from the target website.

[0066] Those skilled in the art know that the process of judging whether the data of the target website is updated or not is prior art, which will not be described here.

[0067] Those skilled in the art know that the process of crawling data from the website is prior art, which will not be described here.

[0068] It should be understood that the data crawled from the target website is the collection data of the target website.

[0069] S200, if the collection data of the target website includes pictures, obtaining the blur degree of the pictures included in the collection data of the target website.

[0070] In this embodiment, if the collection data of the target website includes text, a target keyword set of the text included in the collection data of the target website is obtained.

[0071] Those skilled in the art know that the method of obtaining the target keyword set of the text is prior art, which will not be described here.

[0072] In this embodiment, if the collection data of the target website includes a table, a target keyword set of the table included in the collection data of the target website is obtained.

[0073] Those skilled in the art know that the method of obtaining the target keyword set of the table is prior art, which will not be described here. Alternatively, the BERT-Table model is used to obtain the target keyword set of the table.

[0074] Those skilled in the art know that any method for determining the blur degree of the picture in the prior art falls within the protection scope of the present application. As an optional embodiment, the blur degree of the picture is determined using the method for determining the blur degree of the picture in the prior art. For example, the blur degree of the picture is determined based on the edges and textures of the picture. Specifically, the edges are extracted and quantified by calculating the image gradient or second derivative, for example, the edges are detected using the Canny operator, and the proportion of edge pixels to total pixels is calculated (the lower the proportion, the more blurred the image), based on which the edge index can be obtained; the texture features (such as contrast and entropy, where the higher the contrast, the clearer the texture; the lower the entropy, the simpler the texture) are calculated by counting the gray scale distribution of pixel pairs, based on which the texture index can be obtained; finally, the edge index and the texture index are linearly weighted to determine the blur degree.

[0075] As a preferred embodiment, the blur degree of the picture is determined according to the first preset type parameter in the picture crawling stage, the second preset type parameter in the processing and storage stage, the third preset type parameter in the transmission stage, and the fourth preset type parameter in the rendering stage. Optionally, the first preset type parameter includes the resolution of the original picture and the picture quality limited by the crawling tool, etc., the second preset type parameter includes the compression ratio and the scaling ratio, etc., the third preset type parameter includes the bandwidth, etc., and the fourth preset parameter type includes the picture rendering quality, etc. By matching these parameters with the preset rule base, the blur degree of the picture can be predicted; wherein the preset rule base includes a plurality of rules, and each rule includes a corresponding relationship between a plurality of parameters and the blur degree of the picture. Thus, without analyzing the blur degree of each picture, the efficiency of obtaining the blur degree of the picture can be improved, which is suitable for scenarios with a large number of pictures.

[0076] S300, if the blur degree of the picture is less than or equal to a preset blur degree threshold, the first keyword set corresponding to the picture is obtained using the OCR recognition technology, and the second keyword set corresponding to the picture is obtained using the multi-modal large model, and the union of the first keyword set and the second keyword set corresponding to the picture is determined as the target keyword set corresponding to the picture.

[0077] Optionally, the preset blur degree threshold is an empirical value.

[0078] In the embodiment, the first keyword set corresponding to the picture is obtained by using an OCR recognition technology, including: obtaining the text corresponding to the picture by using the OCR recognition technology; and determining the keyword set extracted from the text corresponding to the picture as the first keyword set corresponding to the picture.

[0079] In the embodiment, the multi-modal large model is a prior art, which is not described herein. The keyword corresponding to the picture can be directly obtained based on the multi-modal large model by inputting the picture into the multi-modal large model and a preset prompt word. The preset keyword is used to instruct the multi-modal large model to extract the keyword of the picture, which can be preset in advance.

[0080] In the embodiment, when the blur degree of the picture is less than or equal to the preset blur degree threshold, that is, when the picture is relatively clear, the OCR recognition technology and the multi-modal large model are used to obtain the keyword of the picture at the same time, which can improve the comprehensiveness of the obtained keyword while ensuring the accuracy of the keyword.

[0081] S400, if the blur degree of the picture is greater than the preset blur degree threshold, the initial keyword set corresponding to the picture is obtained by using the multi-modal large model, and the keyword set obtained by processing the initial keyword set is determined as the target keyword set corresponding to the picture.

[0082] As a preferred specific implementation, the processing of the initial keyword set includes, for example, Figure 2

[0083] S410, the confidence of each keyword in the initial keyword set corresponding to the picture is obtained; and the confidence of any keyword in the initial keyword set corresponding to the picture is obtained by using the multi-modal large model.

[0084] In the embodiment, the multi-modal large model can output the confidence of each keyword while outputting the keyword, the confidence is the probability that the corresponding keyword is the keyword of the picture, and the confidence is greater than or equal to 0 and less than or equal to 1; the greater the confidence, the higher the accuracy of the corresponding keyword. Optionally, the multi-modal large model is instructed to output the confidence of each keyword while outputting the keyword by a method of presetting the prompt word.

[0085] S420, if the collection data of the target website includes text and tables, the keyword is extracted from the text and tables included in the collection data of the target website.

[0086] ​In this embodiment, if the collection data of the target website only includes text and does not include tables, the keywords are extracted from the text included in the collection data of the target website; if the collection data of the target website only includes tables and does not include text, the keywords are extracted from the tables included in the collection data of the target website; the subsequent steps are similar and will not be described here.

[0087] S430, determining the context relevance of each keyword in the initial keyword set corresponding to the picture according to the keywords extracted from the text and the tables included in the collection data of the target website.

[0088] As a preferred specific implementation, as shown in Figure 3 S430 includes:

[0089] S431, obtaining the co-occurrence times between the keywords extracted from the text and the tables included in the collection data of the target website.

[0090] Optionally, if two keywords appear in a sentence in the text or appear in a cell in the table, it is determined that the two keywords co-occur.

[0091] S432, constructing a keyword association graph according to the co-occurrence times; the nodes of the keyword association graph are keywords, and the weights of the edges of the keyword association graph are the co-occurrence times of the two keywords connected by the edges.

[0092] S433, obtaining the co-occurrence times sum of each keyword in the keyword association graph; the co-occurrence times sum of any keyword in the keyword association graph is the sum of the weights of all edges connected to the keyword.

[0093] S434, obtaining the relevance of any keyword in the initial keyword set corresponding to the picture to each candidate keyword in the keyword association graph, and taking the maximum relevance as the context relevance of the keyword in the initial keyword set corresponding to the picture; the candidate keyword is a keyword whose co-occurrence times sum is greater than or equal to a preset sum threshold.

[0094] In this example, the relevance of two keywords is the semantic similarity of the two keywords, and the relevance is greater than or equal to 0 and less than or equal to 1.

[0095] In this embodiment, the candidate keywords are keywords of the target website with high accuracy, and if the relevance of a keyword in the initial keyword set corresponding to the picture to a candidate keyword is large, the relevance of the keyword in the initial keyword set corresponding to the picture to the text and the tables of the target website is large, and the accuracy of the keyword in the initial keyword set corresponding to the picture is also high.

[0096] Based on S431-S434, the context correlation degree of each keyword can be obtained.

[0097] S440, determining a comprehensive value of each keyword in the initial keyword set corresponding to the picture according to the confidence and the context correlation degree of each keyword in the initial keyword set corresponding to the picture.

[0098] As an optional implementation, as shown in Figure 4 S440 includes:

[0099] S441, obtaining a first value of any keyword in the initial keyword set corresponding to the picture according to the confidence of the keyword and a preset confidence weight.

[0100] Optionally, the preset confidence weight is an empirical value, and the preset confidence weight is greater than 0 and less than 1.

[0101] S442, obtaining a second value of any keyword in the initial keyword set corresponding to the picture according to the context correlation degree of the keyword and a preset context correlation degree weight.

[0102] Optionally, the preset context correlation degree weight is an empirical value, and the preset context correlation degree weight is greater than 0 and less than 1. Optionally, the sum of the preset confidence weight and the preset context correlation degree weight is 1, for example, the preset confidence weight and the preset context correlation degree weight are both 0.5.

[0103] S443, determining the sum of the first value and the second value of any keyword in the initial keyword set corresponding to the picture as the comprehensive value of the keyword.

[0104] Based on S441-S443, the comprehensive value of each keyword in the initial keyword set can be accurately obtained; the greater the comprehensive value of a keyword is, the higher the accuracy of the keyword as the keyword of the picture is.

[0105] S450, deleting the keyword in the initial keyword set corresponding to the picture whose comprehensive value is less than a preset comprehensive value threshold.

[0106] Based on S410-S450, the accuracy of the target keyword set corresponding to the picture obtained under the condition that the blur degree of the picture is greater than the preset blur degree threshold can be improved.

[0107] The embodiment obtains the blur degree of the picture in the collection data of the target website, and classifies according to the blur degree threshold. When the blur degree of the picture is less than or equal to the preset threshold, the OCR recognition technology and the multi-modal large model are combined. The OCR recognition technology has the ability to accurately extract the text information of the clear picture, and the multi-modal large model has the ability to understand the overall content of the picture. The advantages of the two are complementary. The union of the keyword sets obtained by the two is taken as the target keyword set. The target keyword set can more comprehensively and accurately cover the key information in the picture, avoiding the limitations of a single technology. When the blur degree of the picture is greater than the preset threshold, the OCR recognition technology is no longer used for keyword extraction, but only relies on the understanding ability of the multi-modal large model to obtain an initial keyword set, and processes the initial keyword set to obtain a target keyword set. The accuracy of the keywords extracted based on the OCR recognition technology is low. The accuracy and reliability of keyword extraction under the condition of poor picture quality are improved. Through this differentiated processing method, the embodiment can provide an optimal keyword extraction scheme for pictures of different blur degrees, improve the effect of picture keyword extraction, and thus provide a more reliable basis for subsequent analysis and mining of collection data, and improve the efficiency and quality of the entire data processing process.

[0108] Embodiment two

[0109] The embodiment provides an electronic device, including a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:

[0110] Obtain the collection data of the target website; the collection data of the target website includes at least one of pictures, texts and tables.

[0111] If the collection data of the target website includes pictures, obtain the blur degree of the pictures included in the collection data of the target website.

[0112] If the blur degree of the picture is less than or equal to the preset blur degree threshold, use the OCR recognition technology to obtain a first keyword set corresponding to the picture, and use the multi-modal large model to obtain a second keyword set corresponding to the picture, and determine the union of the first keyword set and the second keyword set corresponding to the picture as the target keyword set corresponding to the picture.

[0113] If the blur degree of the picture is greater than the preset blur degree threshold, use the multi-modal large model to obtain an initial keyword set corresponding to the picture, and determine the keyword set obtained by processing the initial keyword set as the target keyword set corresponding to the picture.

[0114] Embodiment three

[0115] The embodiment provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0116] Acquire collection data of the target website; the collection data of the target website includes at least one of pictures, texts and tables.

[0117] If the collection data of the target website includes pictures, acquire the blur degree of the pictures included in the collection data of the target website.

[0118] If the blur degree of the pictures is less than or equal to a preset blur degree threshold, use an OCR recognition technology to acquire a first keyword set corresponding to the pictures, use a multi-modal large model to acquire a second keyword set corresponding to the pictures, and determine the union of the first keyword set and the second keyword set corresponding to the pictures as a target keyword set corresponding to the pictures.

[0119] If the blur degree of the pictures is greater than the preset blur degree threshold, use the multi-modal large model to acquire an initial keyword set corresponding to the pictures, and determine a keyword set obtained by processing the initial keyword set as a target keyword set corresponding to the pictures.

[0120] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the computer program can include the processes of the above-mentioned embodiments. Any reference to a memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM).

[0121] While certain specific embodiments of the application have been described in detail herein for the purposes of exemplification, numerous other variations and modifications will be apparent to persons skilled in the art. Any and all such variations and modifications are within the scope of this application as defined in the appended claims.

Claims

1. A processing method of collecting data, characterized by, The method comprises the following steps: acquiring collection data of a target website; the collection data of the target website comprises at least one of a picture, text and a table; if the collection data of the target website comprises a picture, acquiring a blur degree of the picture comprised in the collection data of the target website; if the blur degree of the picture is less than or equal to a preset blur degree threshold, using an OCR recognition technology to acquire a first keyword set corresponding to the picture, using a multi-modal large model to acquire a second keyword set corresponding to the picture, and determining a union set of the first keyword set and the second keyword set corresponding to the picture as a target keyword set corresponding to the picture; if the blur degree of the picture is greater than the preset blur degree threshold, using the multi-modal large model to acquire an initial keyword set corresponding to the picture, and determining a keyword set obtained by processing the initial keyword set as the target keyword set corresponding to the picture; processing the initial keyword set comprises: acquiring a confidence degree of each keyword in the initial keyword set corresponding to the picture; the confidence degree of any keyword in the initial keyword set corresponding to the picture is acquired by the multi-modal large model; if the collection data of the target website comprises text and a table, extracting keywords from the text and the table comprised in the collection data of the target website; determining a context correlation degree of each keyword in the initial keyword set corresponding to the picture according to the keywords extracted from the text and the table comprised in the collection data of the target website; determining a comprehensive value of each keyword in the initial keyword set corresponding to the picture according to the confidence degree and the context correlation degree of each keyword in the initial keyword set corresponding to the picture; deleting a keyword in the initial keyword set corresponding to the picture whose comprehensive value is less than a preset comprehensive value threshold.

2. The processing method of collecting data according to claim 1, characterized in that, determining a context correlation degree of each keyword in the initial keyword set corresponding to the picture according to the keywords extracted from the text and the table comprised in the collection data of the target website comprises: acquiring a co-occurrence frequency between the keywords extracted from the text and the table comprised in the collection data of the target website; constructing a keyword association graph according to the co-occurrence frequency; a node of the keyword association graph is a keyword, and a weight of an edge of the keyword association graph is a co-occurrence frequency of two keywords connected by the edge; acquiring a co-occurrence frequency sum of each keyword in the keyword association graph; the co-occurrence frequency sum of any keyword in the keyword association graph is a sum of weights of all edges connected by the keyword; acquiring an association degree of any keyword in the initial keyword set corresponding to the picture and each candidate keyword in the keyword association graph, and taking a maximum association degree as the context correlation degree of the keyword in the initial keyword set corresponding to the picture; the candidate keyword is a keyword whose co-occurrence frequency sum is greater than or equal to a preset sum threshold.

3. The method of claim 1, wherein, determining a comprehensive value of each keyword in the initial keyword set corresponding to the picture according to the confidence degree and the context correlation degree of each keyword in the initial keyword set corresponding to the picture comprises: The first value of any keyword in the initial keyword set corresponding to the picture is obtained according to the confidence of the keyword and a preset confidence weight; The second value of any keyword in the initial keyword set corresponding to the picture is obtained according to the context correlation degree of the keyword and a preset context correlation degree weight; The sum of the first value and the second value of any keyword in the initial keyword set corresponding to the picture is determined as the comprehensive value of the keyword.

4. The method of claim 1, wherein, The blur degree of the picture is determined according to edges and textures of the picture.

5. The method of claim 1, wherein, The first keyword set corresponding to the picture is obtained using an OCR recognition technology, comprising: Text corresponding to the picture is obtained using an OCR recognition technology; A keyword set extracted from the text corresponding to the picture is determined as the first keyword set corresponding to the picture.

6. The method of claim 1, wherein, The method further comprises: If the collection data of the target website includes text, a target keyword set of the text included in the collection data of the target website is obtained.

7. The method of claim 1, wherein, If the collection data of the target website includes a table, a target keyword set of the table included in the collection data of the target website is obtained.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the collection data processing method of any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the collection data processing method of any one of claims 1 to 7.

Images