Method and device for acquiring OG data of multiple social media platforms through links

Abstract

The invention provides a method and device for acquiring OG data of multiple social media platforms through links, and belongs to the technical field of data acquisition and processing. The method comprises the following steps: acquiring URL links from a plurality of social media platforms and common webpages as source data, wherein the source data correspond to webpage OG data lt; carrying out metagt; the label form is embedded into the lt of the HTML; carrying out headt; a label; performing validity check and standardization on the URL link, capturing webpage content by using an asynchronous HTTP (Hyper Text Transport Protocol) request library, and loading a complete HTML (Hyper Text Markup Language) on a dynamic webpage through a headless browser; the HTML positioning result is analyzed; carrying out metagt; tag: extracting an OG standard field and a platform extension field, and deducing a missing field from other HTML tags; mapping each platform OG field into a unified standard field; and outputting the OG data containing fields such as title and the like through the standardized API interface. According to the method, the multi-platform OG data acquisition efficiency, accuracy and safety are improved, good expansibility is achieved, and the cross-platform data integration requirement is met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data collection and processing, and particularly relates to a method and device for obtaining OG data of multiple social media platforms through links and electronic equipment. BACKGROUND

[0002] With the diversified development of social media platforms, obtaining Open Graph (OG) data of a webpage through a link has become a core requirement in scenarios such as content aggregation, SEO analysis, etc. OG data is usually embedded in the tag of an HTML document in the form of a tag, containing title, description, image link, etc. key information, which is an important basis for realizing link preview, content recommendation, and plays a key role in cross-platform data integration and application. <meta>

[0003] However, the existing technology for obtaining OG data of multiple social media platforms through links still has many defects, and it is difficult to meet the actual application requirements of high efficiency, accuracy and safety. The main problems are as follows:

[0004] 1. Cross-platform data standards are not unified: although the OG protocol has basic specifications, there are significant differences in the implementation of each social media platform. Some platforms follow the standard OG tags (such as og:title, og:description), some platforms add custom extension tags (such as twitter:card, al:ios:url), and some platforms do not fully comply with the OG protocol, only implement part of the fields or have inconsistent field naming, resulting in large differences in OG data structures between different platforms, greatly increasing the difficulty of data parsing.

[0005] 2. High complexity of data parsing: the extraction of OG data depends on the parsing of the HTML structure of the webpage, and the HTML structures of different social media platforms are significantly different. Traditional static HTML parsing tools (such as BeautifulSoup) can only handle statically generated content and cannot handle webpages that dynamically generate OG data through JavaScript; if you want to obtain OG data from dynamic webpages, you need to introduce browser simulation tools (such as Selenium) for page rendering, which not only increases the complexity of technical implementation, but also consumes more computing resources.

[0006] 3. Poor data integrity and consistency: not all webpages fully implement the OG standard, and some webpages have empty OG field content or do not conform to the standard format; at the same time, some social media platforms only provide part of the OG data to the outside world, rather than complete content description information, resulting in missing and inconsistent OG data, which cannot meet the requirements of subsequent applications for data quality.

[0007] ​4. Batch processing efficiency is low: most of the prior art uses single-threaded or synchronous task model of one by one grabbing, when batch processing multiple links, the data acquisition speed is slow; and in the dynamic web loading process, additional associated resources such as scripts and pictures need to be obtained, which further prolongs the data extraction time, and it is difficult to meet the real-time demand in actual application.

[0008] 5. Anti-crawling restriction and security risk are prominent: most social media platforms set up anti-crawling mechanisms to protect their own data security, such as triggering IP blocking, verification code verification for frequent link crawling behavior, or limiting request frequency and access rights, which leads to failure when large-scale OG data is obtained; in addition, some data acquisition techniques do not encrypt the request process and return results, which may cause sensitive information leakage during transmission, and unauthorized large-scale crawling behavior may also violate platform privacy policies, causing compliance risks.

[0009] The core cause of the above problems is that the OG standards of each platform are not unified, static analysis technology cannot cover dynamic content, there is a lack of intelligent fault tolerance and data standardization mechanism, the synchronous task model is difficult to support high concurrency processing, the anti-crawling strategy is insufficient, and the security transmission mechanism is missing. These defects seriously restrict the efficient integration and application of OG data of multiple social media platforms, and an urgent solution is needed to break through the existing technical bottlenecks. SUMMARY

[0010] Therefore, the embodiments of the present application provide a method, device and electronic equipment for obtaining OG data of multiple social media platforms through links, which at least partially solve the problems in the prior art.

[0011] In a first aspect, the embodiments of the present application provide a method for obtaining OG data of multiple social media platforms through links, including the following content:

[0012] Obtain source data to be processed, the source data is URL links from multiple social media platforms and ordinary web pages, the OG data of the URL links exists in the tags of the HTML document and is embedded in the form of <meta> tags, and the OG data includes standardized OG data, platform customized OG extended data and dynamically generated OG data;

[0013] Perform data collection processing on the obtained URL links, check the validity of the URL and perform standardization processing on the non-standard URL, use an asynchronous HTTP request library to grab web page content, and if the web page is a dynamic web page, use a headless browser to load the complete HTML structure;

[0014] Perform data analysis processing on the collected web page HTML content, analyze the HTML structure of the web page and locate <meta>The label extracts standard fields and platform-specific extension fields conforming to the OG protocol, and if the OG field is missing, infers relevant data from other HTML labels;

[0015] The parsed OG data is subjected to data standardization processing, and the OG fields of each platform are mapped to unified standard fields according to a predefined field mapping table. For missing or inconsistent fields, the fields are filled by intelligent rules or automatically generated field values based on content analysis. Meanwhile, the above data collection, parsing and standardization operations are performed on multiple input URL links in an asynchronous batch processing manner. The input URL queue is divided into multiple batches for asynchronous crawling and parsing. A timeout and retry mechanism is set for each request, and the concurrent quantity is dynamically adjusted according to the server load.

[0016] During the execution of the above data collection, parsing, standardization and asynchronous batch processing, security and anti-crawling protection measures are enabled. A distributed IP proxy pool is used to switch the request source IP. A User-Agent header is dynamically generated to simulate multiple devices and browsers. If a complex anti-crawling mechanism is encountered, a third-party identification service is integrated for processing. Finally, the standardized OG data obtained through data standardization processing is output, including title, description, image_url and other extension fields. The standardized OG data is output in JSON or XML format through a standardized API interface and supports paging and batch return of results.

[0017] According to a specific implementation mode of an embodiment of the application, the platform custom OG extension data includes non-standard OG labels, and the dynamically generated OG data is OG data generated by dynamic rendering through JavaScript and after the web page is completely loaded.

[0018] According to a specific implementation mode of an embodiment of the application, when using an asynchronous HTTP request library to grab web page content, multiple requests are initiated simultaneously through an event loop. When using a headless browser to load dynamic web pages, JavaScript scripts are executed by simulating a browser, and the rendered page is grabbed.

[0019] According to a specific implementation mode of an embodiment of the application, an HTML parsing tool is used to parse the HTML structure of a web page, regular expressions or XPath are used to locate target fields when extracting OG-related fields, and default values are automatically filled for missing fields when inferring relevant data from other HTML labels in the case of missing OG fields.

[0020] According to a specific implementation manner of the embodiment of the application, the dynamic mapping of the OG field and the standard field is implemented through a configuration file; when the intelligent rule filling is performed on the missing or inconsistent field, the corresponding field is supplemented by extracting the web content; when the field value is automatically generated based on the content analysis, the natural language processing technology is combined to generate the supplemented content of the missing field.

[0021] According to a specific implementation manner of the embodiment of the application, the high-concurrency task scheduling is implemented using an asynchronous framework, and the task queue management is implemented through a message queue; when the concurrency is dynamically adjusted, the running state of the server is monitored, and the request rate is optimized according to the monitoring result.

[0022] According to a specific implementation manner of the embodiment of the application, the distributed IP proxy pool periodically detects the validity of the proxy IP; when the User-Agent header is dynamically generated, the identifiers corresponding to different browsers and devices are covered; the third-party identification service is integrated to cope with the complex anti-crawling mechanism.

[0023] According to a specific implementation manner of the embodiment of the application, the standardized API interface is a RESTful API or a GraphQL interface; when the standardized OG data is output, the format and the paging parameter of the returned data are determined according to the request parameter of the client; when the batch return result is returned, the standardized OG data corresponding to multiple URL links is returned.

[0024] In a second aspect, the embodiment of the application further provides a device for obtaining OG data of multiple social media platforms through links, comprising:

[0025] A source data acquisition module is configured to acquire source data to be processed, wherein the source data is URL links from multiple social media platforms and ordinary web pages, the OG data of the URL links exists in a tag of an HTML document and is embedded in the form of a <meta> The OG data includes standardized OG data, platform customized OG extension data and dynamically generated OG data.

[0026] A data acquisition module is configured to perform data acquisition processing on the URL links acquired by the source data acquisition module, check the validity of the URL and perform standardized processing on the non-standard URL, use an asynchronous HTTP request library to grab the web content, and use a headless browser to load the complete HTML structure if the web page is a dynamic web page.

[0027] A data analysis module is configured to perform data analysis processing on the web HTML content collected by the data acquisition module, analyze the HTML structure of the web page and locate <meta>The label extracts standard fields and platform-specific extension fields conforming to the OG protocol, and infers relevant data from other HTML labels if OG fields are missing.

[0028] The data standardization and asynchronous batch processing module is configured to perform data standardization processing on the OG data parsed by the data parsing module, map OG fields of each platform to unified standard fields according to a predefined field mapping table, fill in missing or inconsistent fields by intelligent rules or automatically generate field values based on content analysis, and simultaneously perform asynchronous batch processing on multiple URL links input by the source data acquisition module, control the data acquisition module and the data parsing module to perform corresponding operations, divide the input URL queue into multiple batches, asynchronously grab and parse the batches, set a timeout and a retry mechanism for each request, and dynamically adjust the concurrent quantity according to server load.

[0029] The security and anti-crawling and data output module is configured to enable security and anti-crawling protection measures during the working processes of the data acquisition module, the data parsing module, and the data standardization and asynchronous batch processing module, use a distributed IP proxy pool to switch the request source IP, dynamically generate a User-Agent header to simulate multiple devices and browsers, integrate a third-party identification service to process complex anti-crawling mechanisms, and output standardized OG data obtained by the data standardization and asynchronous batch processing module, the standardized OG data including title, description, image_url, and other extension fields, and the standardized OG data being output in JSON or XML format through a standardized API interface and supporting page-based and batch-based returned results.

[0030] In a third aspect, an electronic device is provided, and the electronic device includes:

[0031] at least one processor; and

[0032] a memory communicatively connected to the at least one processor; and

[0033] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the foregoing first aspect or any implementation manner of the first aspect.

[0034] In a fourth aspect, a non-transitory computer-readable storage medium is provided, and the non-transitory computer-readable storage medium stores computer instructions for causing a computer to perform the method described in the foregoing first aspect or any implementation manner of the first aspect.

[0035] In a fifth aspect, an embodiment of the present application further provides a computer program product, which comprises a computer program stored on a non-transitory computer-readable storage medium, and the computer program comprises program instructions, which, when executed by a computer, cause the computer to perform the method in the first aspect or any implementation manner of the first aspect.

[0036] The present application has the following beneficial effects:

[0037] The field differences and format specifications of different platforms are automatically adapted, and separate parsing logic does not need to be written for a single platform, thereby greatly reducing the technical complexity of multi-platform data acquisition; the integrity and accuracy of OG data extraction of dynamic web pages are ensured; the dynamic OG data that cannot be acquired by traditional static parsing tools is avoided by loading and rendering the dynamically generated content through the headless browser technology, and it is ensured that the extracted data covers various web page scenarios.

[0038] Based on the asynchronous framework, high-concurrency data crawling is realized, the task queue and the dynamic concurrency adjustment mechanism are combined, the multi-link processing time consumption is reduced, and the real-time application requirements are met; the integrity and standardization degree of OG data are improved; through the intelligent verification and field repair mechanism, the missing and inconsistent OG fields are automatically completed or format-corrected, and the standardized data of uniform structure are output, thereby providing high-quality data support for subsequent content aggregation, SEO analysis and other applications.

[0039] With the help of the distributed IP proxy pool and the dynamic User-Agent switching, the platform anti-crawling restrictions are evaded, and the large-scale data acquisition failure probability is reduced; at the same time, the HTTPS encryption and Token authentication are used to ensure the data transmission security, prevent sensitive information leakage, and evade compliance risks.

[0040] The modularization and adapter design are adopted, the core code does not need to be modified when the social media platform parsing logic is added, only the corresponding platform adapter needs to be written, and the update and expansion requirements of different platforms can be quickly integrated, thereby long-term meeting the use requirements of multi-scenario multi-social media platform OG data acquisition. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0042] Figure 1 A method flowchart for acquiring OG data of multiple social media platforms through links provided by an embodiment of the present application is shown in the figure.

[0043] Figure 2A schematic diagram illustrating another method for obtaining OG data from multiple social media platforms via links, provided in an embodiment of the present invention;

[0044] Figure 3 A schematic diagram of the device structure for obtaining OG data from multiple social media platforms via links, provided in an embodiment of the present invention;

[0045] Figure 4 A schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0046] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0047] See Figure 1 and Figure 2 This invention provides a method for obtaining OG data from multiple social media platforms via links, comprising the following steps:

[0048] S101, Obtain the source data to be processed. The source data consists of URL links from multiple social media platforms and ordinary web pages. The OG data corresponding to the URL links exists in the HTML document's `<head>` tag and is defined as follows: <meta> The OG data is embedded in the form of tags and includes standardized OG data, platform-customized OG extended data, and dynamically generated OG data.

[0049] A pre-built source data input interface supports users to input URLs from social media platforms such as Facebook, Twitter, and LinkedIn, as well as ordinary web pages, by batch uploading documents (such as Excel and TXT) or by calling the API interface to pass in URL links. A data classification module is set up in the interface backend to initially classify and label the input URLs according to the source platform, and an OG data type identification rule base is established. When a URL link is input, the rule base is used to match and determine the type of OG data for the corresponding web page: if the HTML document tag of the linked web page contains... <meta> If the tag contains standard fields such as og:title, og:description, and og:image, it is determined to be standardized OG data; if it contains non-standard extended tags such as twitter:card and al:ios:url, it is determined to be platform-customized OG extended data; if the webpage is detected to have JavaScript dynamic loading logic and the OG data needs to be generated after the webpage is loaded, it is determined to be dynamically generated OG data, and the classification results are associated with the URL links and stored to form a set of source data to be processed.

[0050] S102, data collection processing is performed on the acquired URL link, validity of the URL is checked and non-standard URL is standardized, web page content is grabbed using an asynchronous HTTP request library, and if the web page is a dynamic web page, a headless browser is used to load the complete HTML structure.

[0051] A URL preprocessing module is built, a URL validity checking algorithm is used to check the format of the URL link in the source data acquired in S101 (such as checking whether the http / https protocol header is included, whether the domain name format is legal), and invalid links are removed; for non-standard URLs (such as missing protocol headers, redundant parameters), standardized processing is performed according to the HTTP standard format, the protocol header is supplemented, the redundant parameters are cleaned up, and a standard URL is generated. Then, a data collection engine is started, an httpx asynchronous HTTP request library is integrated, and based on an event loop mechanism, multiple URL web page content grabbing requests are simultaneously initiated; during the collection process, a web page feature detection algorithm (such as analyzing the HTML response header, judging whether the JavaScript dynamic rendering identifier is included) is used to identify whether the web page is a dynamic web page, if it is determined to be a dynamic web page, a Puppeteer headless browser is automatically called, a browser environment is simulated to load the web page, JavaScript scripts in the page are executed, after the page is completely rendered, the complete HTML structure data is grabbed, and the collected static HTML or dynamic rendered HTML data is temporarily stored in a temporary database, preparing for the subsequent parsing step.

[0052] S103, data parsing processing is performed on the collected web page HTML content, HTML structure of the web page is parsed and <meta> tags are located, standard fields and platform-specific extended fields conforming to the OG protocol are extracted, and if the OG field is missing, related data is inferred from other HTML tags.

[0053] An HTML parsing engine is started, an lxml parsing tool is integrated, HTML data temporarily stored in S102 is read, and an HTML document tree structure is constructed; all <meta>The tag, extracts the corresponding values of the property attribute and the content attribute in the tag, filters out the standard fields such as og:title, og:image, etc. according to the OG protocol, and platform-specific extension fields such as twitter:card, al:ios:url, etc., and stores the extraction results in a temporary data structure. At the same time, set the field missing detection module to check the integrity of the extracted OG fields. If the key fields (such as title, description) are missing, trigger the supplementary inference mechanism: extract the corresponding values of the title, description, and image tags under the <title>The title of the web page is extracted from the tag and appended to the title field, from< / title> tag from the HTML document The first paragraph of text content in the tag is extracted and supplemented to the description field from The largest size image link at the top of the page in the tag is filtered and supplemented to the image_url field, ensuring that the parsed data has basic integrity.

[0054] S104, the OG data parsed is processed by data standardization, the OG fields of each platform are mapped to unified standard fields according to the pre-defined field mapping table, for the missing or inconsistent fields, the intelligent rule is filled or the field value is automatically generated based on content analysis, and the asynchronous batch processing mode is used to execute the above data collection, parsing and standardization operation on the input multiple URL links, the input URL queue is decomposed into multiple batches, and the asynchronous grabbing and parsing are performed by batch, the timeout and retry mechanism are set for each request, and the concurrent quantity is dynamically adjusted according to the server load.

[0055] The field mapping configuration library is constructed, the mapping relationship between the OG fields of each social media platform and the unified standard fields (such as mapping twitter:title to title and al:ios:image to image_url) is pre-defined in the library; when the OG data is parsed in S103, the data standardization module reads the mapping rules in the configuration library, and automatically converts the OG fields of different platforms into unified format standard fields. For the fields that still exist after conversion and are inconsistent in format, the intelligent repair module is started: if it is a text field (such as description) missing, a natural language processing model (such as a text generation model based on BERT) is called to generate a description text meeting the length requirement according to the web text content; if it is a format inconsistent field (such as date format), it is uniformly converted according to the pre-defined format specification (such as YYYY-MM-DD). At the same time, the asynchronous batch processing engine is started, the asyncio framework and Redis task queue are integrated, the multiple URL links input in S101 are decomposed into multiple task batches according to the pre-set batch size (such as 50 per batch), and the batch tasks are added to the Redis queue; the asynchronous scheduler gets the tasks from the queue, calls the data collection module of S102 and the data parsing module of S103 to execute processing, and sets the timeout time (such as 30 seconds) for each request, which automatically triggers the retry mechanism (up to 3 times) after timeout; through the server monitoring module, the load indicators such as CPU usage and memory occupation rate are collected in real time, when the load exceeds the pre-set threshold (such as CPU usage 80%), the concurrent quantity is automatically reduced, and when the load is lower than the threshold, the concurrent quantity is increased, ensuring the stability and efficiency of batch processing.

[0056] S105, in the execution of the above data collection, analysis, standardization and asynchronous batch processing process, enable security and anti-crawling protection measures, use distributed IP proxy pool to switch request source IP, dynamically generate User-Agent header to simulate multiple devices and browsers, if complex anti-crawling mechanism is encountered, integrate third-party identification service processing, finally output the standardized OG data obtained by data standardization processing, the standardized OG data includes title, description, image_url and other extended fields, output in JSON or XML format through standardized API interface and support paging and batch return results.

[0057] A distributed IP proxy pool is built, the proxy pool includes a plurality of available IP addresses in different regions and different operators, the connectivity and validity of the proxy IP are verified by a timing detection script (once every 5 minutes) to eliminate invalid IP; in the S102 data collection and S104 asynchronous batch processing process, the security module automatically selects an IP from the proxy pool as the request source IP, the IP is switched once every 10 requests to avoid platform IP blocking. At the same time, a User-Agent pool is constructed, the pool includes different browsers such as Chrome and Firefox, and device identifiers of different systems such as Android and iOS, a User-Agent header is randomly selected and added to the request header every time a request is made to simulate real user access. If a complex anti-crawling mechanism such as a verification code is encountered during the request process, a third-party verification code recognition interface (such as a verification code recognition service based on deep learning) is automatically called to upload the verification code picture to obtain the recognition result and automatically fill in the verification code, and the verification process is completed. When the data standardization processing is completed, the data output module reads the standardized OG data (including title, description, image_url and other extended fields), encapsulates the data according to the client request parameter selection JSON or XML format, if the client requests page output, the data is split according to the preset paging parameter (such as 20 data per page) to generate a paging response including the current page number and the total number of pages; finally, the encapsulated data is returned to the client through the RESTful API interface, supporting the client to obtain the standardized OG data corresponding to multiple URL in batches.

[0058] According to a specific implementation manner of the embodiment of the application, the platform custom OG extended data includes non-standard OG tags, and the dynamically generated OG data is OG data generated by dynamic rendering through JavaScript and after the web page is completely loaded.

[0059] Specifically, when processing a URL link from a platform, the tags of the webpage HTML are parsed, and non-standard OG tags specific to the platform can be extracted, which constitute the custom OG extension data of the platform; when processing a URL link of a commodity detail page, the page dynamically loads OG data through a JavaScript script, specifically, after the page is loaded, the JavaScript code requests the core information of the page from the server, and dynamically generates a <meta> tag containing standard OG fields, and such OG data generated after the webpage is completely loaded is dynamic OG data. In actual processing, by detecting whether the JavaScript dynamic rendering identifier is included in the webpage HTML, the dynamic OG data can be quickly identified, and the subsequent headless browser loading process is triggered.

[0060] According to a specific implementation mode of an embodiment of the application, when using an asynchronous HTTP request library to grab webpage content, multiple requests are initiated simultaneously through an event loop; when using a headless browser to load a dynamic webpage, the browser executes a JavaScript script and grabs the rendered page.

[0061] Specifically, when processing a batch of tasks containing 20 URL links, 20 asynchronous tasks are created based on an event loop, each task corresponding to a URL content grabbing request, and the event loop schedules these tasks to be executed concurrently without waiting for the previous request to be completed before initiating the next one, which greatly shortens the grabbing time compared to synchronous requests. When it is detected that a webpage corresponding to a URL is a dynamic webpage, a headless browser is automatically called to create a simulated browser page, the page is waited to be completely idle in terms of network requests to ensure that the JavaScript script is executed, and then the complete HTML content after rendering is grabbed, thereby avoiding the problem that traditional static parsing tools cannot obtain dynamic data.

[0062] According to a specific implementation mode of an embodiment of the application, when parsing the HTML structure of a webpage, an HTML parsing tool is used, the target field is positioned through a regular expression or XPath when extracting OG related fields, and a default value is automatically filled in the missing field when the OG field is missing and related data is inferred from other HTML tags.

[0063] Specifically, after the HTML parsing tool reads the grabbed webpage HTML content, an HTML document tree structure is constructed. When extracting OG related fields, the XPath expression is used for positioning: for standard OG fields, the field value is extracted through the corresponding XPath expression; for platform-specific extension fields, the field value is extracted through the XPath expression matching the extension tag. If it is detected that a key OG field is missing, a supplementary inference mechanism is triggered: the <title>The content is extracted from the tag and appended to the title class field, from< / title> tag under the The first paragraph of text content in the label is extracted and supplemented to the description class field, and from Suitable picture links in the label are screened and supplemented to the picture class field; if it is still impossible to supplement, preset default values are automatically filled in to ensure that the parsed data has basic integrity.

[0064] According to a specific implementation manner of the embodiment of the application, the dynamic mapping of the OG field and the standard field is realized through a configuration file; when the intelligent rule filling is performed on the missing or inconsistent field, the corresponding field is supplemented by extracting the web content; when the field value is automatically generated based on the content analysis, the natural language processing technology is combined to generate the supplemented content of the missing field.

[0065] Specifically, the field mapping configuration file in the JSON format is created, and the mapping relationship between the OG fields of different platforms and the unified standard fields is predefined in the file. When the OG data is parsed, the data standardization module reads the mapping rules in the configuration file, and automatically converts the OG fields of different platforms into the standard fields in the unified format. If a certain text class field is missing after the mapping, the intelligent rule filling module extracts the content from the web text label, and intercepts the specified length of the text to supplement the field; if the text content is short, the natural language processing model is called, the core information of the web page is input, the supplemented text meeting the length requirement is generated, and the corresponding field is automatically filled in to ensure the quality of the field content.

[0066] According to a specific implementation manner of the embodiment of the application, the high-concurrency task scheduling is realized by using an asynchronous framework, and the task queue management is realized by using a message queue; when the concurrency is dynamically adjusted, the running state of the server is monitored, and the request rate is optimized according to the monitoring result.

[0067] Specifically, the asynchronous framework is selected, and the asynchronous batch processing architecture is constructed in combination with the distributed task queue management system: the URL links to be processed are encapsulated into tasks according to a preset batch size, and are sent to the message queue; the task processing node asynchronously obtains the tasks from the queue, and calls the data collection and parsing module to execute the processing, so that the high-concurrency task scheduling is realized. Meanwhile, the server monitoring tool is deployed, and the indexes such as the CPU usage, the memory occupation rate and the network bandwidth usage rate are regularly collected: when the CPU usage or the memory occupation rate exceeds a preset threshold, the number of concurrent tasks is automatically reduced, and the server load is reduced; when the indexes are lower than the threshold, the number of concurrent tasks is increased, and the processing efficiency is improved; if the network bandwidth usage rate is too high, the new tasks are suspended, the task scheduling is resumed after the bandwidth usage rate is reduced to a reasonable range, and the stable operation of the server is ensured.

[0068] According to a specific implementation manner of the embodiment of the application, the distributed IP proxy pool regularly detects the validity of the proxy IP; when the User-Agent header is dynamically generated, the identifiers corresponding to different browsers and devices are covered; and the third-party identification service is integrated to cope with the complex anti-crawling mechanism.

[0069] Specifically, a distributed IP proxy pool is built, a plurality of available IP addresses of different regions and different operators are initially accessed in the pool, the validity of the proxy IP is verified periodically by a timing detection script, the judgment standard is whether the HTTP request to the public test page is successful and returns a normal status code, the invalid IP will be removed, and at the same time, new valid IP is automatically supplemented, so as to ensure that there is always sufficient valid IP in the pool. When dynamically generating the User-Agent header, a random selection is made from a preset User-Agent list, and the list contains the identifiers corresponding to different browsers and different system devices. When processing the URL link triggering the anti-crawling mechanism of the verification code, the third-party verification code recognition API is automatically called, the verification code picture is converted into a specified format and uploaded, the recognition result is received and automatically filled, and the data collection is ensured to be carried out normally.

[0070] According to a specific implementation manner of the embodiment of the application, the standardized API interface is a RESTful API or a GraphQL interface; when the standardized OG data is output, the format and the paging parameter of the returned data are determined according to the request parameter of the client; and the standardized OG data corresponding to a plurality of URL links is returned in batches.

[0071] Specifically, the RESTful API and the GraphQL interface are deployed at the same time for the client to select: the RESTful API and the GraphQL interface are respectively set to corresponding request addresses. When the client requests data through the RESTful API, if the specified format identifier is included in the request parameter, the corresponding format data is returned; if the paging parameter is included, the URL standardized OG data of the corresponding page number and the corresponding number of rows is returned. For example, the client initiates a request including the parameters "format = JSON", "page number = 2", and "number of rows per page = 20", and the server returns a JSON format response including the total data amount, the current page number, the number of rows per page, and 20 pieces of URL standardized OG data. When the client requests data through the GraphQL interface, the return field can be specified as needed, and the server returns the corresponding data according to the request field; when the OG data of a plurality of URLs is requested in batches, the server returns the standardized OG data corresponding to all the requested URLs at one time, without the need for the client to request multiple times, thereby improving the data acquisition efficiency.

[0072] Referring to Figure 3 The application further provides a device for obtaining OG data of a plurality of social media platforms through links, comprising:

[0073] A source data acquisition module is configured to acquire source data to be processed, wherein the source data is URL links from a plurality of social media platforms and ordinary webpages, and the OG data of the URL links exists in a tag of an HTML document and is in a format of <meta> The OG data includes standardized OG data, platform customized OG extension data and dynamically generated OG data;

[0074] The data acquisition module is configured to perform data acquisition processing on the URL links obtained by the source data acquisition module, check validity of the URL and perform standardization processing on non-standard URL, use an asynchronous HTTP request library to grab web page content, and use a headless browser to load complete HTML structure if the web page is a dynamic web page.

[0075] The data parsing module is configured to perform data parsing processing on the HTML content of the web page collected by the data acquisition module, parse the HTML structure of the web page and locate <meta> The tag extracts standard fields and platform specific extension fields conforming to the OG protocol, and infers related data from other HTML tags if the OG field is missing.

[0076] The data standardization and asynchronous batch processing module is configured to perform data standardization processing on the OG data parsed by the data parsing module, map the OG fields of each platform to unified standard fields according to a predefined field mapping table, fill in missing or inconsistent fields through intelligent rules or automatically generate field values based on content analysis, simultaneously use an asynchronous batch processing mode to process multiple URL links input by the source data acquisition module, control the data acquisition module and the data parsing module to perform corresponding operations, and divide the input URL queue into multiple batches to asynchronously grab and parse the batches, set a timeout and a retry mechanism for each request and dynamically adjust the concurrency according to the server load.

[0077] The security and anti-crawling and data output module is configured to enable security and anti-crawling protection measures during working processes of the data acquisition module, the data parsing module and the data standardization and asynchronous batch processing module, use a distributed IP proxy pool to switch request source IP, dynamically generate a User-Agent header to simulate multiple devices and browsers, integrate a third-party identification service to process complex anti-crawling mechanisms, and output standardized OG data obtained by the data standardization and asynchronous batch processing module, the standardized OG data including title, description, image_url and other extension fields, and the standardized OG data being output in JSON or XML format through a standardized API interface and supporting paging and batch return results.

[0078] Referring to Figure 4 The electronic device 60 includes:

[0079] at least one processor; and

[0080] a memory in communication with the at least one processor; wherein

[0081] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of obtaining OG data of multiple social media platforms through link in the foregoing method embodiments.

[0082] The embodiment of the present application also provides a non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method in the foregoing method embodiments.

[0083] The embodiment of the present application also provides a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, cause the computer to perform the method of obtaining OG data of multiple social media platforms through link in the foregoing method embodiments.

[0084] As shown in Figure 4 The electronic device 60 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601 that can perform various appropriate actions and processes in accordance with programs stored in a read-only memory (ROM) 602 or loaded from a storage device 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device 60 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0085] In general, the following devices can be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, a gyro sensor, a geomagnetic sensor, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication devices 609 can allow the electronic device 60 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 Figure 4 The electronic device 60 is shown with various devices, but it should be understood that not all of the shown devices are required to be implemented or present. More or fewer devices can alternatively be implemented or present.

[0086] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program in accordance with embodiments of the application. For example, embodiments of the application include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the methods of the embodiments of the application are performed.

[0087] The above description is merely illustrative of the application and is not intended to limit the scope of the application. Any changes and modifications that can be made to the application in accordance with the principle of the application should be encompassed within the scope of the application. The scope of the application should be determined by the appended claims.

Claims

1. A method for obtaining OG data of a plurality of social media platforms through chaining, characterized in that, Includes the following: Obtaining source data to be processed, the source data being URL links from multiple social media platforms and ordinary web pages, the OG data corresponding to the URL links existing in the form of <meta> embedded in the tag of the HTML document, the OG data including standardized OG data, platform customized OG extension data and dynamically generated OG data; The obtained URLs are processed by data collection, the validity of the URLs is checked and non-standard URLs are standardized, and the web page content is crawled using an asynchronous HTTP request library. If the web page is a dynamic web page, the headless browser is used to load the complete HTML structure. The collected webpage HTML content is subjected to data analysis processing, the HTML structure of the webpage is analyzed and the position of the webpage is located <meta> Tags, extract standard fields and platform-specific extension fields that meet the OG protocol, and if the OG field is missing, infer the relevant data from other HTML tags; The parsed OG data is standardized by mapping OG fields from various platforms to uniform standard fields according to a predefined field mapping table. For missing or inconsistent fields, the data is filled in by intelligent rules or automatically generated based on content analysis. At the same time, the above data collection, parsing and standardization operations are performed on multiple input URLs using an asynchronous batch processing method. The input URL queue is decomposed into multiple batches and asynchronously crawled and parsed in batches. Timeout and retry mechanisms are set for each request and the concurrency is dynamically adjusted according to the server load. During the data collection, parsing, standardization, and asynchronous batch processing described above, security and anti-crawling protection measures are enabled. A distributed IP proxy pool is used to switch the request source IP, and User-Agent headers are dynamically generated to simulate various devices and browsers. If complex anti-crawling mechanisms are encountered, third-party identification services are integrated for processing. Finally, standardized OG data obtained through data standardization processing is output. The standardized OG data includes title, description, image_url, and other extended fields. It is output in JSON or XML format through a standardized API interface and supports pagination and batch return of results.

2. The method of claim 1, wherein, The platform-customized OG extended data includes non-standard OG tags, and the dynamically generated OG data is OG data generated by JavaScript dynamic rendering after the webpage is fully loaded.

3. The method of claim 2, wherein, When using an asynchronous HTTP request library to fetch web page content, multiple requests are initiated simultaneously through an event loop; when using a headless browser to load dynamic web pages, the browser is simulated to execute JavaScript scripts and fetch the rendered page.

4. The method of claim 3, wherein, When parsing the HTML structure of a webpage, an HTML parsing tool is used. When extracting OG-related fields, the target fields are located using regular expressions or XPath. When inferring related data from other HTML tags if an OG field is missing, default values ​​are automatically filled in for the missing fields.

5. The method of claim 4, wherein, The configuration file enables dynamic mapping between OG fields and standard fields; when filling missing or inconsistent fields with intelligent rules, the webpage content is extracted to supplement the corresponding fields; when automatically generating field values ​​based on content analysis, natural language processing technology is used to generate supplementary content for missing fields.

6. The method of claim 5, wherein, High-concurrency task scheduling is implemented using an asynchronous framework, and task queue management is achieved through message queues. When dynamically adjusting the concurrency, the server's operating status is monitored, and the request rate is optimized based on the monitoring results.

7. The method of claim 6, wherein, The distributed IP proxy pool periodically checks the validity of proxy IPs; the dynamically generated User-Agent header includes identifiers for different browsers and devices; and third-party identification services are integrated to deal with complex anti-scraping mechanisms.

8. The method of claim 7, wherein, The standardized API interface is a RESTful API or a GraphQL interface; when outputting standardized OG data, the format and paging parameters of the returned data are determined according to the request parameters of the client; and when returning batch results, the standardized OG data corresponding to multiple URL links is returned.

9. A device for obtaining OG data from multiple social media platforms via links, characterized in that, Comprise: a source data acquisition module, configured to acquire source data to be processed, the source data being URL links from a plurality of social media platforms and general web pages, OG data of a webpage corresponding to the URL links being embedded in a tag of an HTML document in a <meta> tag form, the OG data including standardized OG data, platform customized OG extension data, and dynamically generated OG data; A data acquisition module, configured to perform data acquisition processing on the URL links acquired by the source data acquisition module, check the validity of the URL and perform standardization processing on non-standardized URLs, use an asynchronous HTTP request library to grab web page content, and if the web page is a dynamic web page, use a headless browser to load the complete HTML structure; A data analysis module is configured to analyze the HTML content of the webpage collected by the data collection module, analyze the HTML structure of the webpage and locate <meta> Tags, extract standard fields and platform-specific extension fields that comply with the OG protocol, and if the OG field is missing, infer the relevant data from other HTML tags. A data standardization and asynchronous batch processing module, configured to perform data standardization processing on the OG data parsed by the data parsing module, map the OG fields of each platform to unified standard fields according to a pre-defined field mapping table, fill in missing or inconsistent fields through intelligent rules or automatically generate field values based on content analysis, and simultaneously use an asynchronous batch processing mode to control the data acquisition module and the data parsing module to perform corresponding operations, divide the input URL queue into multiple batches, asynchronously grab and parse the batches, set a timeout and a retry mechanism for each request, and dynamically adjust the concurrency according to the server load; A security and anti-crawling and data output module, configured to enable security and anti-crawling protection measures during the working processes of the data acquisition module, the data parsing module and the data standardization and asynchronous batch processing module, use a distributed IP proxy pool to switch the request source IP, dynamically generate a User-Agent header to simulate multiple devices and browsers, integrate a third-party identification service to handle complex anti-crawling mechanisms, and output the standardized OG data obtained by the data standardization and asynchronous batch processing module, wherein the standardized OG data includes title, description, image_url and other extended fields, and is output in JSON or XML format through a standardized API interface and supports paging and batch return of results.

10. An electronic device, comprising: The electronic device comprises: At least one processor; and A memory connected in communication with the at least one processor; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of preceding claims 1 to 8.

Images