Acquired data processing method and device and medium

Abstract

The invention relates to the technical field of electric digital data processing, in particular to an acquired data processing method and device and a medium. The method comprises the following steps: acquiring collection data of a target website; if the acquired data of the target website comprises the picture, acquiring the fuzzy degree of the picture included in the acquired data of the target website; if the fuzzy degree is smaller than or equal to a preset fuzzy degree threshold value, obtaining a first keyword set corresponding to the picture by using an OCR identification technology, and obtaining a second keyword set corresponding to the picture by using a multi-modal large model, determining a union set of the first keyword set and the second keyword set as a target keyword set corresponding to the picture; and otherwise, acquiring an initial keyword set corresponding to the picture by using a multi-modal large model, and determining a keyword set obtained by processing the initial keyword set as a target keyword set corresponding to the picture. According to the invention, the accuracy of the keyword corresponding to the obtained picture can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, and in particular to a method, device and medium for processing collected data. Background Art

[0002] Network data collection technology is widely used in various fields. By collecting data in various forms such as images, text, and tables from target websites, it provides rich materials for information analysis and knowledge mining. Among them, images, as an important data type, contain a large amount of information. Accurately extracting keywords corresponding to images is crucial for data processing and utilization. Currently, commonly used technical means for extracting keywords corresponding to images include OCR recognition technology. OCR recognition technology can convert the text content in images into editable text data, thereby extracting keywords. However, this technology has high requirements for image clarity. When the image has quality problems such as blur, the recognition accuracy will drop significantly. How to improve the accuracy of keywords corresponding to the acquired images is an urgent problem to be solved. Summary of the Invention

[0003] The present invention aims to provide a method, device and medium for processing collected data to improve the accuracy of keywords corresponding to acquired images.

[0004] According to a first aspect of the present invention, a method for processing collected data is provided, the method comprising the following steps: Acquire collected data of a target website; the collected data of the target website includes at least one of images, texts, and tables.

[0005] If the collected data of the target website includes pictures, the blur degree of the pictures included in the collected data of the target website is obtained.

[0006] If the blurriness of the image is less than or equal to a preset blurriness threshold, OCR recognition technology is used to obtain the first keyword set corresponding to the image, and a multimodal large model is used to obtain the second keyword set corresponding to the image, and the union of the first keyword set and the second keyword set corresponding to the image is determined as the target keyword set corresponding to the image.

[0007] If the blurriness of the image is greater than a preset blurriness threshold, the multimodal large model is used to obtain the initial keyword set corresponding to the image, and the keyword set obtained by processing the initial keyword set is determined as the target keyword set corresponding to the image.

[0008] Furthermore, processing the initial keyword set includes: The confidence of each keyword in the initial keyword set corresponding to the image is obtained; the confidence of any keyword in the initial keyword set corresponding to the image is obtained through the multimodal large model.

[0009] If the collected data of the target website includes text and tables, keywords are extracted from the text and tables included in the collected data of the target website.

[0010] The context relevance of each keyword in the initial keyword set corresponding to the image is determined based on the keywords extracted from the text and table included in the collected data of the target website.

[0011] The comprehensive value of each keyword in the initial keyword set corresponding to the picture is determined according to the confidence and context relevance of each keyword in the initial keyword set corresponding to the picture.

[0012] Keywords with comprehensive values ​​less than a preset comprehensive value threshold in the initial keyword set corresponding to the image are deleted.

[0013] Furthermore, determining the context relevance of each keyword in the initial keyword set corresponding to the image based on keywords extracted from texts and tables included in the collected data of the target website includes: Get the co-occurrence counts between keywords extracted from text and tables in the collected data of the target website.

[0014] A keyword association graph is constructed according to the co-occurrence counts; the nodes of the keyword association graph are keywords, and the weights of the edges of the keyword association graph are the co-occurrence counts of the two keywords connected by the edge.

[0015] The total number of co-occurrences of each keyword in the keyword association graph is obtained; the total number of co-occurrences of any keyword in the keyword association graph is the sum of the weights of all edges connected to the keyword.

[0016] Obtain the correlation between any keyword in the initial keyword set corresponding to the image and each candidate keyword in the keyword association graph, and use the maximum correlation as the context correlation of the keyword in the initial keyword set corresponding to the image; the candidate keyword is a keyword whose total number of co-occurrences is greater than or equal to a preset total threshold.

[0017] Furthermore, determining the comprehensive value of each keyword in the initial keyword set corresponding to the image according to the confidence and context relevance of each keyword in the initial keyword set corresponding to the image includes: The first value of any keyword in the initial keyword set corresponding to the image is obtained according to the confidence of the keyword and a preset confidence weight.

[0018] The second value of any keyword in the initial keyword set corresponding to the image is obtained according to the context relevance of the keyword and a preset context relevance weight.

[0019] The sum of the first value and the second value of any keyword in the initial keyword set corresponding to the picture is determined as the comprehensive value of the keyword.

[0020] Furthermore, the blur degree of the image is determined according to the edge and texture of the image.

[0021] Furthermore, using OCR recognition technology to obtain a first keyword set corresponding to the image includes: Use OCR recognition technology to obtain the text corresponding to the image.

[0022] A keyword set extracted from the text corresponding to the picture is determined as a first keyword set corresponding to the picture.

[0023] Furthermore, the method further comprises: If the collected data of the target website includes text, a target keyword set of the text included in the collected data of the target website is obtained.

[0024] Furthermore, the method further includes: if the collected data of the target website includes a table, obtaining a target keyword set of the table included in the collected data of the target website.

[0025] According to a second aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for processing collected data when executing the computer program.

[0026] According to a third aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements the above-mentioned method for processing collected data.

[0027] Compared with the prior art, the present invention has at least the following beneficial effects: The present invention obtains the blur degree of images in the target website's collected data and classifies them according to the blur degree threshold. When the blur degree of the image is less than or equal to a preset threshold, the OCR recognition technology and the multimodal large model are combined to utilize the OCR recognition technology's ability to accurately extract text information from clear images and the multimodal large model's ability to understand the overall content of the image. The two complement each other's advantages and the keyword sets obtained by the two are taken as the target keyword set. This can more comprehensively and accurately cover the key information in the image and avoid the limitations of a single technology. When the blur degree of the image is greater than the preset threshold, the OCR recognition technology is no longer used for keyword extraction. Instead, the initial keyword set is obtained by relying solely on the multimodal large model's ability to understand the blurry image and then processed to obtain the target keyword set. This can avoid the problem of low accuracy of keywords extracted based on OCR recognition technology and is conducive to improving the accuracy and reliability of keyword extraction in the case of poor image quality. Through this differentiated processing method, the present invention can provide a better keyword extraction scheme for images with different blur degrees, improve the effect of image keyword extraction, thereby providing a more reliable foundation for subsequent analysis, mining, and other operations based on the collected data, and improving the efficiency and quality of the entire data processing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 Flowchart of the method for processing collected data provided in Example 1 of the present invention; Figure 2 A flowchart of a process for processing the initial keyword set provided in the first embodiment of the present invention; Figure 3 A flowchart of a process for determining the contextual relevance of each keyword in the initial keyword set corresponding to the image provided in the first embodiment of the present invention; Figure 4 This is a flowchart of the process of determining the comprehensive value of each keyword in the initial keyword set corresponding to the image provided by the first embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the scope of protection of the present invention.

[0031] Example 1: According to this embodiment, Figure 1 As shown, a method for processing collected data is provided, the method comprising the following steps: S100, acquiring collected data of a target website; the collected data of the target website includes at least one of images, texts, and tables.

[0032] In this embodiment, the target website is a certain website.

[0033] As a preferred embodiment, the process of acquiring the collected data of the target website includes: S110 , determining a target crawling time period of the target website and a crawling time step corresponding to the target crawling time period according to a historical update time set of the target website.

[0034] As a preferred embodiment, S110 includes: S111, obtaining a set of historical update times of the target website.

[0035] In this embodiment, the update time does not include the year, month and day, but only the 24-hour time. For example, the historical update times of a website corresponding to a certain URL include 8:59, 9:01 and 9:00, etc.

[0036] S112: Cluster the historical update time set of the target website to obtain several update time clusters.

[0037] In this embodiment, the difference in update time of the same cluster is small, and the difference in update time of different clusters is large.

[0038] Preferably, a density clustering method is selected for clustering, thereby obtaining a large time period during which the target website is updated. It should be understood that if the target website is typically updated at around 9 a.m. and around 2 p.m. in history, then using the density clustering method can yield two clusters, one of which includes the update time around 9 a.m. and the other includes the update time around 2 p.m. Those skilled in the art will appreciate that the density clustering process is prior art and will not be further described herein.

[0039] S113: Obtain a crawling time step corresponding to any update time cluster according to the update time included in the update time cluster.

[0040] As a preferred embodiment, S113 includes: S1131, obtain a distribution histogram corresponding to a specified update time cluster; the horizontal axis of the distribution histogram is the update time, the vertical axis of the distribution histogram is the number of update times, the distribution histogram includes a plurality of rectangular bars, different rectangular bars correspond to different ranges of update times, and different rectangular bars correspond to ranges of update times of equal length; the specified update time cluster is any update time cluster.

[0041] Those skilled in the art know that the process of constructing a histogram is an existing technology and will not be described in detail here.

[0042] Optionally, the length of the range of update time corresponding to any rectangular bar is preset and may be an empirical value, such as 0.5 seconds or 1 second.

[0043] S1132, segmenting the rectangular bars in the distribution histogram according to the number of update moments corresponding to adjacent rectangular bars in the distribution histogram; the difference value of the number of update moments corresponding to adjacent rectangular bars included in any segment is less than or equal to a preset number difference threshold.

[0044] Optionally, the preset quantity difference threshold is an empirical value; if the difference value of the quantity of update moments corresponding to adjacent rectangular bars is less than or equal to the preset quantity difference threshold, it is determined that the difference in the quantity of update moments corresponding to the adjacent rectangular bars is small.

[0045] S1133, determine the range of update time corresponding to any segment as a target crawling time period of the target website, and determine the crawling time step corresponding to the target crawling time period corresponding to the segment according to the mean value of the number of update time corresponding to the rectangular bars included in the segment; the crawling time step corresponding to the target crawling time period corresponding to any segment is negatively correlated with the mean value of the number of update time corresponding to the rectangular bars included in the segment.

[0046] In this embodiment, the start time of any target crawling time period is the earliest update time corresponding to the target crawling time period, and the start time of any target crawling time period is the latest update time corresponding to the target crawling time period.

[0047] As a preferred embodiment, the crawling time step corresponding to the target crawling time period of the i-th segment is T i , T i =T min +(T max -T min )×k / (u i +k), T max and T min are the preset maximum crawling time step and minimum crawling time step, ui is the number of update moments corresponding to the rectangular bars included in the i-th segment, k is a preset adjustment coefficient, and k>0. Optionally, k is an empirical value used to adjust the sensitivity of the crawling time step to the number of update moments. Based on this preferred embodiment, the crawling time step corresponding to any target crawling time period can be obtained.

[0048] Based on S1131-S1133, a more suitable target crawling time period and corresponding crawling time step can be determined for any period, which can improve the timeliness of data collection for different update time periods of the target website and reduce the resources consumed by collection.

[0049] Based on S111-S113, different and more suitable segmented collection strategies can be formulated for the target website, which can improve the timeliness of data collection for the target website and reduce the resources consumed by collection.

[0050] S120, within the target crawling time period of the target website, using the crawling time step corresponding to the target crawling time period of the target website as a judgment period to determine whether the data of the target website is updated, and if updated, crawling data from the target website.

[0051] Those skilled in the art know that the process of determining whether the data of the target website is updated is an existing technology and will not be described in detail here.

[0052] Those skilled in the art know that the process of how to crawl data from a website is an existing technology and will not be described in detail here.

[0053] It should be understood that the data crawled from the target website is the collected data of the target website.

[0054] S200: If the collected data of the target website includes pictures, obtain the blur level of the pictures included in the collected data of the target website.

[0055] In this embodiment, if the collected data of the target website includes text, a target keyword set of the text included in the collected data of the target website is obtained.

[0056] Those skilled in the art know that the method of obtaining a target keyword set from a text is an existing technology and will not be described in detail here.

[0057] In this embodiment, if the collected data of the target website includes a table, a target keyword set of the table included in the collected data of the target website is obtained.

[0058] Those skilled in the art know that the method for obtaining the target keyword set of the table is a prior art and will not be described in detail here. Optionally, the BERT-Table model is used to obtain the target keyword set of the table.

[0059] Those skilled in the art know that any method for determining the degree of blur of an image in the prior art falls within the scope of protection of the present invention. As an optional specific embodiment, the degree of blur of an image is determined using a method for determining the degree of blur of an image in the prior art. For example, the degree of blur of an image is determined based on the edge and texture of the image. Specifically, by calculating the image gradient or second-order derivative, the edge is extracted and its clarity is quantified. For example, the edge is detected using the Canny operator, and the ratio of edge pixels to total pixels is counted (the lower the ratio, the blurrier the image), based on which an edge index can be obtained; texture features (such as contrast and entropy, where the higher the contrast, the clearer the texture; the lower the entropy, the simpler the texture) are calculated by counting the grayscale distribution of pixel pairs, based on which a texture index can be obtained; finally, the obtained edge index and texture index are linearly weighted to determine the degree of blur.

[0060] As a preferred specific implementation method, the blurriness of the image is determined based on a first preset type parameter in the image crawling stage, a second preset type parameter in the processing and storage stage, a third preset type parameter in the transmission stage, and a fourth preset type parameter in the rendering stage. Optionally, the first preset type parameter includes the resolution of the original image and the image quality limited by the crawling tool, the second preset type parameter includes the compression ratio and the zoom ratio, the third preset type parameter includes the bandwidth, and the fourth preset parameter type includes the image rendering quality. By matching these parameters with a preset rule library, the blurriness of the image can be predicted; wherein the preset rule library includes a number of rules, and each rule includes a correspondence between a number of parameters and the blurriness of the image. Therefore, there is no need to analyze the blurriness of each image, which can improve the efficiency of obtaining the blurriness of the image, and is suitable for scenarios with a large number of images.

[0061] S300: If the blurriness of the image is less than or equal to a preset blurriness threshold, use OCR recognition technology to obtain a first keyword set corresponding to the image, and use a multimodal large model to obtain a second keyword set corresponding to the image, and determine the union of the first keyword set and the second keyword set corresponding to the image as the target keyword set corresponding to the image.

[0062] Optionally, the preset blur level threshold is an empirical value.

[0063] In this embodiment, using OCR recognition technology to obtain the first keyword set corresponding to the image includes: using OCR recognition technology to obtain text corresponding to the image; and determining a keyword set extracted from the text corresponding to the image as the first keyword set corresponding to the image.

[0064] In this embodiment, the multimodal large model is a prior art and will not be further described here. By inputting an image and a preset prompt word into the multimodal large model, keywords corresponding to the image can be directly obtained based on the multimodal large model; the preset keywords are used to indicate the keywords that the multimodal large model should extract from the image and can be pre-set.

[0065] In this embodiment, when the blurriness of an image is less than or equal to a preset blurriness threshold, that is, when the image is relatively clear, OCR recognition technology and a multimodal large model are used simultaneously to obtain keywords of the image, which can improve the comprehensiveness of the obtained keywords while ensuring the accuracy of the keywords.

[0066] S400: If the blurriness of the image is greater than a preset blurriness threshold, an initial keyword set corresponding to the image is obtained using a multimodal large model, and a keyword set obtained by processing the initial keyword set is determined as a target keyword set corresponding to the image.

[0067] As a preferred embodiment, processing the initial keyword set includes: Figure 2 As shown: S410, obtaining the confidence of each keyword in the initial keyword set corresponding to the image; the confidence of any keyword in the initial keyword set corresponding to the image is obtained through the multimodal large model.

[0068] In this embodiment, the multimodal large model outputs keywords and also outputs the confidence level of each keyword. The confidence level represents the probability that the corresponding keyword is the keyword of the image. The confidence level is greater than or equal to 0 and less than or equal to 1. The greater the confidence level, the higher the accuracy of the corresponding keyword. Optionally, a prompt word can be pre-set to instruct the multimodal large model to output the confidence level of each keyword at the same time as the keyword output.

[0069] S420: If the collected data of the target website includes text and tables, extract keywords from the text and tables included in the collected data of the target website.

[0070] In this embodiment, if the collected data of the target website includes only text but not tables, keywords are extracted from the text included in the collected data of the target website; if the collected data of the target website includes only tables but not text, keywords are extracted from the tables included in the collected data of the target website; the subsequent steps are similar and will not be repeated here.

[0071] S430 : Determine the context relevance of each keyword in the initial keyword set corresponding to the image based on the keywords extracted from the text and table included in the collected data of the target website.

[0072] As a preferred embodiment, Figure 3 As shown, S430 includes: S431 , obtaining the co-occurrence counts between keywords extracted from the text and table included in the collected data of the target website.

[0073] Optionally, if two keywords appear in a sentence in a text, or appear in a cell in a table, the two keywords are determined to co-occur.

[0074] S432 , constructing a keyword association graph according to the co-occurrence counts; the nodes of the keyword association graph are keywords, and the weights of the edges of the keyword association graph are the co-occurrence counts of the two keywords connected by the edge.

[0075] S433, obtaining the total number of co-occurrences of each keyword in the keyword association graph; the total number of co-occurrences of any keyword in the keyword association graph is the sum of the weights of all edges connected to the keyword.

[0076] S434, obtaining the correlation between any keyword in the initial keyword set corresponding to the image and each candidate keyword in the keyword association graph, and taking the maximum correlation as the context correlation of the keyword in the initial keyword set corresponding to the image; the candidate keyword is a keyword whose total number of co-occurrences is greater than or equal to a preset total threshold.

[0077] In this example, the correlation between two keywords is the semantic similarity between the two related words, and the correlation is greater than or equal to 0 and less than or equal to 1.

[0078] In this embodiment, the accuracy of the candidate keywords being keywords of the target website is relatively high. If a keyword in the initial keyword set corresponding to the picture has a high correlation with a candidate keyword, then the keyword in the initial keyword set corresponding to the picture has a high correlation with the text and table of the target website, and the accuracy of the keyword in the initial keyword set corresponding to the picture is also relatively high.

[0079] Based on S431 - S434 , the context relevance of each keyword can be obtained.

[0080] S440 : Determine a comprehensive value of each keyword in the initial keyword set corresponding to the image according to the confidence and context relevance of each keyword in the initial keyword set corresponding to the image.

[0081] As an optional specific implementation, Figure 4 As shown, S440 includes: S441 : Obtain a first value of any keyword in the initial keyword set corresponding to the image according to the confidence of the keyword and a preset confidence weight.

[0082] Optionally, the preset confidence weight is an empirical value, and the preset confidence weight is greater than 0 and less than 1.

[0083] S442 : Obtain a second value of any keyword in the initial keyword set corresponding to the image according to the context relevance of the keyword and a preset context relevance weight.

[0084] Optionally, the preset context relevance weight is an empirical value, and the preset context relevance weight is greater than 0 and less than 1. Optionally, the sum of the preset confidence weight and the preset context relevance weight is 1, for example, the preset confidence weight and the preset context relevance weight are both 0.5.

[0085] S443: Determine the sum of the first value and the second value of any keyword in the initial keyword set corresponding to the image as the comprehensive value of the keyword.

[0086] Based on S441-S443, the comprehensive value of each keyword in the initial keyword set can be accurately obtained; the greater the comprehensive value of a keyword, the higher the accuracy of the keyword being the keyword of the image.

[0087] S450: Delete keywords whose comprehensive values ​​are less than a preset comprehensive value threshold in the initial keyword set corresponding to the image.

[0088] Based on S410 - S450 , the accuracy of the target keyword set corresponding to the image obtained when the blurriness of the image is greater than a preset blurriness threshold can be improved.

[0089] This embodiment obtains the blur level of images in the target website's collected data and classifies them according to a blur level threshold. When the blur level of the image is less than or equal to a preset threshold, it combines OCR recognition technology and a multimodal large model, leveraging the OCR recognition technology's ability to accurately extract text information from clear images and the multimodal large model's ability to understand the overall content of the image. The two complement each other's strengths, and the resulting keyword sets are combined as the target keyword set. This can more comprehensively and accurately cover the key information in the image, avoiding the limitations of a single technology. When the blur level of the image is greater than the preset threshold, OCR recognition technology is no longer used for keyword extraction. Instead, the initial keyword set is obtained by relying solely on the multimodal large model's ability to understand the blurry image, and then processed to obtain the target keyword set. This can avoid the problem of low accuracy of keywords extracted based on OCR recognition technology, and is conducive to improving the accuracy and reliability of keyword extraction in the case of poor image quality. Through this differentiated processing method, this embodiment can provide a better keyword extraction solution for images with different blur levels, improve the effect of image keyword extraction, thereby providing a more reliable foundation for subsequent analysis and mining operations based on the collected data, and improving the efficiency and quality of the entire data processing process.

[0090] Example 2: This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed: Acquire collected data of a target website; the collected data of the target website includes at least one of images, texts, and tables.

[0091] If the collected data of the target website includes pictures, the blur degree of the pictures included in the collected data of the target website is obtained.

[0092] If the blurriness of the image is less than or equal to a preset blurriness threshold, OCR recognition technology is used to obtain the first keyword set corresponding to the image, and a multimodal large model is used to obtain the second keyword set corresponding to the image, and the union of the first keyword set and the second keyword set corresponding to the image is determined as the target keyword set corresponding to the image.

[0093] If the blurriness of the image is greater than a preset blurriness threshold, the multimodal large model is used to obtain the initial keyword set corresponding to the image, and the keyword set obtained by processing the initial keyword set is determined as the target keyword set corresponding to the image.

[0094] Example 3: This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the following steps are implemented: Acquire collected data of a target website; the collected data of the target website includes at least one of images, texts, and tables.

[0095] If the collected data of the target website includes pictures, the blur degree of the pictures included in the collected data of the target website is obtained.

[0096] If the blurriness of the image is less than or equal to a preset blurriness threshold, OCR recognition technology is used to obtain the first keyword set corresponding to the image, and a multimodal large model is used to obtain the second keyword set corresponding to the image, and the union of the first keyword set and the second keyword set corresponding to the image is determined as the target keyword set corresponding to the image.

[0097] If the blurriness of the image is greater than a preset blurriness threshold, the multimodal large model is used to obtain the initial keyword set corresponding to the image, and the keyword set obtained by processing the initial keyword set is determined as the target keyword set corresponding to the image.

[0098] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0099] Although some specific embodiments of the present invention have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A method for processing collected data, characterized in that: The method comprises the following steps: Acquire collected data of a target website; the collected data of the target website includes at least one of images, texts, and tables; If the collected data of the target website includes pictures, obtaining the blur level of the pictures included in the collected data of the target website; If the blurriness of the image is less than or equal to a preset blurriness threshold, then using OCR recognition technology to obtain a first keyword set corresponding to the image, and using a multimodal large model to obtain a second keyword set corresponding to the image, and determining the union of the first keyword set and the second keyword set corresponding to the image as the target keyword set corresponding to the image; If the blurriness of the image is greater than a preset blurriness threshold, the multimodal large model is used to obtain the initial keyword set corresponding to the image, and the keyword set obtained by processing the initial keyword set is determined as the target keyword set corresponding to the image.

2. The method for processing collected data according to claim 1, characterized in that: Processing the initial keyword set includes: Obtaining the confidence of each keyword in the initial keyword set corresponding to the image; the confidence of any keyword in the initial keyword set corresponding to the image is obtained through the multimodal large model; If the collected data of the target website includes text and tables, extracting keywords from the text and tables included in the collected data of the target website; Determining the context relevance of each keyword in the initial keyword set corresponding to the image based on keywords extracted from text and tables included in the collected data of the target website; Determining a comprehensive value of each keyword in the initial keyword set corresponding to the image according to the confidence and context relevance of each keyword in the initial keyword set corresponding to the image; Keywords with comprehensive values ​​less than a preset comprehensive value threshold in the initial keyword set corresponding to the image are deleted.

3. The method for processing collected data according to claim 2, characterized in that: Determining the context relevance of each keyword in the initial keyword set corresponding to the image by extracting keywords from text and tables included in the collected data of the target website includes: Obtain the number of co-occurrences between keywords extracted from the collected data of the target website, including text and tables; Constructing a keyword association graph based on the co-occurrence counts; wherein the nodes of the keyword association graph are keywords, and the weights of the edges of the keyword association graph are the co-occurrence counts of the two keywords connected by the edge; Obtaining the total number of co-occurrences of each keyword in the keyword association graph; the total number of co-occurrences of any keyword in the keyword association graph is the sum of the weights of all edges connected to the keyword; Obtain the correlation between any keyword in the initial keyword set corresponding to the image and each candidate keyword in the keyword association graph, and use the maximum correlation as the context correlation of the keyword in the initial keyword set corresponding to the image; the candidate keyword is a keyword whose total number of co-occurrences is greater than or equal to a preset total threshold.

4. The method for processing collected data according to claim 2, wherein: Determining the comprehensive value of each keyword in the initial keyword set corresponding to the image according to the confidence and context relevance of each keyword in the initial keyword set corresponding to the image includes: Obtaining a first value of any keyword in the initial keyword set corresponding to the image according to the confidence of the keyword and a preset confidence weight; Obtaining a second value of any keyword in the initial keyword set corresponding to the image according to the context relevance of the keyword and a preset context relevance weight; The sum of the first value and the second value of any keyword in the initial keyword set corresponding to the picture is determined as the comprehensive value of the keyword.

5. The method for processing collected data according to claim 1, wherein: The blur degree of the image is determined according to the edge and texture of the image.

6. The method for processing collected data according to claim 1, characterized in that: Using OCR recognition technology to obtain a first keyword set corresponding to the image includes: Use OCR recognition technology to obtain the text corresponding to the image; A keyword set extracted from the text corresponding to the picture is determined as a first keyword set corresponding to the picture.

7. The method for processing collected data according to claim 1, characterized in that: The method further comprises: If the collected data of the target website includes text, a target keyword set of the text included in the collected data of the target website is obtained.

8. The method for processing collected data according to claim 1, characterized in that: The method further includes: if the collected data of the target website includes a table, obtaining a target keyword set of the table included in the collected data of the target website.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for processing collected data according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for processing collected data according to any one of claims 1 to 8 is implemented.

Images