Data processing method and device, equipment and storage medium

Abstract

A data processing method, device and equipment and storage medium, relating to the technical field of Internet. The method comprises: obtaining configuration information of a user account, the configuration information comprising identification information of a social media platform; obtaining link information of web content of the social media platform according to the configuration information; obtaining text data of the web content according to the link information, the text data being data describing the web content in text form; analyzing the text data to obtain summary data of the web content, the summary data being data used to summarize the web content; and providing the summary data to the user account. The embodiment of the application can automatically obtain summary data of web content of a social media platform specified by a user account, which is conducive to improving the efficiency and convenience of obtaining data of the social media platform.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a data processing method, apparatus, device, and storage medium. Background Technology

[0002] Social media platforms are content production and interaction platforms based on user relationships, allowing users to share opinions, insights, experiences, and perspectives via the internet. Due to regional diversity and wide reach, popular social media platforms often vary from region to region.

[0003] In related technologies, to quickly and effectively understand local public opinion hotspots and discussion topics, personnel often identify these hotspots through manual observation and recording. For example, for different social media platforms, personnel need to manually check and summarize the relevant websites for trending topics, especially videos. However, this manual method is time-consuming, labor-intensive, and inefficient in data acquisition. Summary of the Invention

[0004] This application provides a data processing method, apparatus, device, and storage medium. The technical solutions provided by this application can be as follows:

[0005] According to one aspect of the embodiments of this application, a data processing method is provided, the method comprising:

[0006] Obtain the configuration information of the user account, including the identification information of the social media platform;

[0007] Based on the configuration information, obtain the link information of the web page content of the social media platform;

[0008] Based on the link information, obtain the text data of the webpage content, wherein the text data is data describing the webpage content in text form;

[0009] Analyze the text data to obtain summary data of the webpage content, wherein the summary data is used to summarize and generalize the webpage content;

[0010] The summarized data is provided to the user account.

[0011] According to one aspect of the embodiments of this application, a data processing apparatus is provided, the apparatus comprising:

[0012] The configuration information acquisition module is used to acquire the configuration information of the user account, including the identification information of the social media platform;

[0013] The link information acquisition module is used to acquire link information of the web page content of the social media platform according to the configuration information;

[0014] The text data acquisition module is used to acquire text data of the webpage content based on the link information, wherein the text data is data describing the webpage content in text form;

[0015] The summary data acquisition module is used to analyze the text data and obtain summary data of the webpage content. The summary data is data used to summarize and generalize the webpage content.

[0016] The summary data push module is used to provide the summary data to the user account.

[0017] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described data processing method.

[0018] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the above-described data processing method.

[0019] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including a computer program stored in a computer-readable storage medium, and a processor reading from the computer-readable storage medium and executing the computer program to implement the above-described data processing method.

[0020] The technical solutions provided in this application have at least the following beneficial effects:

[0021] By leveraging the identifiers of social media platforms configured in a user's account, the system automatically retrieves the webpage content and summary data of those platforms, eliminating the need for users to manually review and summarize the content. This significantly improves the ease and efficiency of data acquisition from social media platforms. Furthermore, it avoids the time-consuming and labor-intensive nature of manual methods, thereby reducing the cost of data acquisition. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of a computer system provided in one embodiment of this application;

[0023] Figure 2This is a schematic diagram of a data processing method provided in one embodiment of this application;

[0024] Figure 3 This is a schematic diagram of webpage content provided in one embodiment of this application;

[0025] Figure 4 This is a schematic diagram of webpage content provided in another embodiment of this application;

[0026] Figure 5 This is a schematic diagram of a task configuration interface provided in one embodiment of this application;

[0027] Figure 6 This is a schematic diagram of the prompt words provided in one embodiment of this application;

[0028] Figure 7 This is a schematic diagram of summary data provided in one embodiment of this application;

[0029] Figure 8 This is a schematic diagram of a data recording interface provided in one embodiment of this application;

[0030] Figure 9 This is a schematic diagram of the social media information acquisition stage provided in one embodiment of this application;

[0031] Figure 10 This is a schematic diagram of the data standardization processing stage provided in one embodiment of this application;

[0032] Figure 11 This is a schematic diagram of a base class provided in one embodiment of this application;

[0033] Figure 12 This is a schematic diagram illustrating the content summary stage provided in one embodiment of this application;

[0034] Figure 13 This is a schematic diagram illustrating the mechanism of an error retry and upgrade model provided in one embodiment of this application;

[0035] Figure 14 This is a schematic diagram of the summary data push stage provided in one embodiment of this application;

[0036] Figure 15 This is a schematic diagram of a one-click summary page provided in one embodiment of this application;

[0037] Figure 16 This is a block diagram of a data processing apparatus provided in one embodiment of this application;

[0038] Figure 17 This is a block diagram of a data processing apparatus provided in another embodiment of this application;

[0039] Figure 18This is a structural block diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0041] Before introducing the technical solutions of this application, some terms involved in this application will be explained. The following related explanations are optional solutions and can be arbitrarily combined with the technical solutions of the embodiments of this application, all of which fall within the protection scope of the embodiments of this application. The embodiments of this application include at least some of the following contents.

[0042] Social media platforms: These are content production and interaction platforms based on user relationships, allowing users to share opinions, insights, experiences, and perspectives via the internet. The social media platforms in this application include overseas social media platforms and domestic social media platforms.

[0043] Token: A token or word unit is the basic unit of text processing by artificial intelligence (AI). It can be a word or a fragment of characters.

[0044] AI illusion: The content produced by AI contradicts the logic of the physical world; it is a product of AI's deduction based solely on training content.

[0045] The implementation environment of the technical solutions provided in the embodiments of this application will be described below.

[0046] Please refer to Figure 1 This diagram illustrates a computer system provided in one embodiment of this application. The computer system can implement the technical solutions provided in the embodiments of this application within a specific environment. The computer system may include: a terminal device 10 and a server 20.

[0047] Terminal device 10 can be an electronic device such as a mobile phone, tablet computer, multimedia playback device, PC (Personal Computer), wearable device, in-vehicle terminal device, intelligent robot, VR (Virtual Reality) device, AR (Augmented Reality) device, MR (Mixed Reality) device, MP3 (Moving Picture Experts Group Audio Layer III) player, MP4 (Moving Picture Experts Group Audio Layer IV) player, laptop computer, etc. Terminal device 10 can install and run a client application for the target application. The target application can be an application with the function of summarizing web page content from social media platforms. Summarizing web page content from social media platforms refers to the process of automatically acquiring web page content from social media platforms and automatically summarizing and generalizing the web page content. For example, the target application can include at least one of the following: data processing application, information collection application, information analysis application, data push application, engine application, simulation application, design and development application; this application embodiment does not limit this.

[0048] The embodiments of this application do not limit the implementation form of the target application described above. For example, it can be an application that needs to be downloaded and installed, a mini-program that does not need to be installed, a web application, a browser, etc.

[0049] Server 20 can be used to provide backend services for clients of the aforementioned target application (such as a data processing application). For example, server 20 can be a backend server for the target application. Server 20 can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center.

[0050] Terminal device 10 can communicate with server 20 via network 30, such as a wireless or wired network.

[0051] The technical solutions provided in this application are applicable to any scenario requiring the summarization of webpage content on social media platforms, such as application delivery feedback analysis scenarios (e.g., delivery feedback analysis of game applications, map navigation applications, shopping applications, simulation applications, etc.), public opinion hotspot identification and analysis scenarios, and information collection scenarios. For example, taking the delivery feedback analysis scenario of game applications as an example, the technical solutions provided in this application are applicable to: information collection of trending games on social media platforms, and related information of games delivered on social media platforms.

[0052] Optionally, the aforementioned game applications can be any of the following: Massive Multiplayer Online Role-Playing Game (MMORPG), casual games, party games, sandbox games, tower defense games, action-adventure games, Multiplayer Online Battle Arena (MOBA) games, first-person shooter (FPS) games, multiplayer shooting survival games, third-person shooter (TPS) games, strategy games (SLG), simulation management games, survival building games, real-time strategy games, etc., and this application embodiment does not limit this.

[0053] Optionally, for the computation work of the technical solution provided in the embodiments of this application, server 20 undertakes the main computation work and terminal device 10 undertakes the secondary computation work; or, server 20 undertakes the secondary computation work and terminal device 10 undertakes the main computation work; or, server 20 and terminal device 10 adopt a distributed computing architecture for collaborative computation.

[0054] For example, refer to Figure 1 Taking terminal device 10 as an example, after obtaining the user account's configuration information, the client in terminal device 10 retrieves the text data of the social media platform's webpage content based on the social media platform's identifier information included in the configuration information. This text data describes the webpage content in text form. Then, the client analyzes the text data to obtain summary data of the webpage content, which is used to summarize and generalize the webpage content. Finally, the client pushes the summary data to the user account.

[0055] The technical solutions provided in this application will be described below through method embodiments.

[0056] Please refer to Figure 2 The diagram illustrates a flowchart of a data processing method provided in one embodiment of this application. The execution entity for each step of the method is a computer device; however, the execution entity for each step of the method may be... Figure 1 The terminal device 10 in the computer system shown is a client of the target application in the terminal device 10. The method may include at least one of the following steps 210 to 250.

[0057] Step 210: Obtain the user account configuration information, which includes the identification information of the social media platform.

[0058] The aforementioned user accounts are those registered by the user in the target application and can be used to refer to the user of the target application. Configuration information refers to the information generated when the user performs configuration operations on the target application. Configuration operations can specify the social media platform for the user. Social media platform identification information is used to uniquely identify the social media platform.

[0059] Optionally, users can specify at least one social media platform through the client of the target application to obtain the required data from the web content of the at least one social media platform, according to their actual usage needs. For example, users can choose different types of social media platforms to obtain data for different regions within the same country; or, users can choose different types of social media platforms to obtain data for different countries. This application embodiment does not limit this.

[0060] In one example, the client provides a task configuration interface through which users can create tasks to obtain summary data of web content from a social media platform. After the task is created, the client executes the task to obtain the summary data of web content from that social media platform that the user needs.

[0061] For example, embodiments of this application may also include the following:

[0062] 1. In response to the operation used to initiate task creation, display the task configuration interface.

[0063] Optionally, after the client is launched, a task creation control is displayed in the client's main user interface. In response to the user's operation on the task creation control, the client displays a task configuration interface. The operation on the task creation control is an operation used to trigger the function of the task creation control, which is used to open the task configuration interface. For example, when the terminal device displays the user interface through a touch screen, the operation on the task creation control can be at least one of the following: click operation, press operation, double-click operation, or swipe operation.

[0064] Optionally, the client's main user interface also displays a task list, which includes at least one task created by the user. Different tasks can correspond to different social media platforms. The at least one task can be sorted sequentially according to the order of its creation.

[0065] Optionally, the task configuration interface displays at least one of the following options: task name, source channel, country / region, summary enable, output language, push time, task scheduling, task public, and email push.

[0066] The options include: Task Name, Source Channel, Country / Region, Summary Enable, Output Language, Push Time, Task Scheduler, Task Execution Cycle, Public / Public, and Email Push.

[0067] 2. In response to actions related to source channel options, display options for at least one social media platform.

[0068] Optionally, each option may correspond to a type of social media platform or to different functional pages within a social media platform; this embodiment of the application does not limit this.

[0069] For example, social media platform A has at least three different functional pages: a page for viewing currently trending videos, a page for viewing currently trending games, and a page for viewing keyword search results. For example, refer to... Figure 3 Function page 300 displays a list of videos for currently popular games, which can be used to view currently popular games; for example, see reference Figure 4 The 400 function page displays a list of videos obtained based on keyword searches, which is used to view the search results for keywords, such as search results for games.

[0070] 3. In response to an operation on an option for at least one target social media platform, generate configuration information based on the identification information of the target social media platform.

[0071] The target social media platform can be any one of at least one social media platform. Optionally, the configuration information may also include at least one of the following: task name, country / region, output language, push time, task execution cycle, and push method.

[0072] After a task is created, the client names it and adds it to the task list. The client then executes the task periodically, specifically by periodically retrieving summary data of web content from a target social media platform in a specific country / region, and periodically pushing the summary data in that output language to the user's account based on the push time and method.

[0073] For example, refer to Figure 5 The client displays a task configuration interface 501, which includes a source channel option 502. In response to the user's selection of the source channel option 502, the client displays an option 503 for at least one social media platform. In response to the selection of the option 503 for the target social media platform among the at least one social media platform, the client generates configuration information based on the identification information of the target social media platform.

[0074] For example, a certain configuration can achieve the following tasks: retrieve the top 5 trending videos on social media platform B in country A every day, and push the summary data to the user via email at 12:48 UTC+08:00 (East 8 time zone) every day. The summary data is displayed in both Japanese and Chinese.

[0075] This application embodiment supports users in creating personalized tasks, and the tasks can be adapted to different social media platforms in different regions, which helps to improve the applicability and flexibility of the technical solution provided by this application embodiment.

[0076] In one example, the configuration information described above may also include at least one keyword. Optionally, the task configuration interface may also include a keyword option, through which the user can enter or select at least one keyword. Keywords can be used to guide the client to obtain summary data of web page content related to the keyword on social media platforms.

[0077] This application does not limit the form of keywords; they can be at least one word, at least one character, at least one symbol, etc. In this application, anything entered by the user can be called a keyword.

[0078] Step 220: Based on the configuration information, obtain the link information of the web page content of the social media platform.

[0079] Link information within webpage content is used to identify and locate webpage content. For example, link information can be the URL (Uniform Resource Locator) of the webpage content, also known as the web address. Webpage content is the fundamental component of a webpage, including text, images, audio, and video.

[0080] In one example, after obtaining the user account's configuration information, the client can retrieve the link information of the web page content indicated by the configuration information from the social media platform, based on the platform's identification information.

[0081] For example, a client can use a headless browser (such as a web browser in headless mode) to obtain link information of web page content from a social media platform and store the link information in a database.

[0082] Optionally, if the configuration information includes the identifier of a social media platform, the client obtains the link information of the webpage content of that social media platform; if the configuration information includes the identifier of a feature page of a social media platform, the client obtains the link information of the webpage content of that feature page. For example, taking the aforementioned social media platform A as an example, for the feature page for viewing currently popular videos and the feature page for viewing currently popular games, since the URLs are unique, the client obtains the link information of each feature page through a headless browser.

[0083] In one example, if the above configuration information includes at least one keyword, for any one of the at least one keyword, the client determines the link information of the web page content related to the keyword on the social media platform.

[0084] Optionally, each keyword may correspond to a link on a social media platform, pointing to web page content related to that keyword on the social media platform. A combination of multiple keywords may also correspond to a link on a social media platform, pointing to web page content related to those multiple keywords on the social media platform.

[0085] For example, taking the aforementioned social media platform A as an example, for the function page used to view search results for keywords, each keyword can correspond to one URL, and multiple keywords can correspond to multiple URLs.

[0086] This application embodiment supports obtaining summary information of web page content related to a configured keyword, which helps to expand the application scenarios and scope of application of the technical solution provided in this application embodiment.

[0087] In a feasible example, after the user completes the selection of a social media platform, the client can automatically create a task for each feature page of that social media platform to obtain summary data of the page content of that feature page.

[0088] Step 230: Based on the link information, obtain the text data of the webpage content. The text data is data that describes the webpage content in text form.

[0089] Optionally, the client can obtain the web page content of the social media platform based on the link information, and extract text data from the web page content.

[0090] Optionally, social media platforms can be divided into two categories. For the first category, clients can directly access the webpage corresponding to the link information through a headless browser to obtain the webpage content. For the second category, clients need to obtain the webpage content based on the link information through the social media platform's application programming interface (API) or toolkit.

[0091] In one example, if obtaining webpage content from a social media platform requires authorization from the social media platform, then the process of obtaining text data may include the following:

[0092] 1. Obtain access tokens for social media platforms.

[0093] The aforementioned access token is an authorization code for the social media platform. This authorization code, denoted as a token, is a code used to verify or authorize a user's identity; it can also be referred to as an access ticket. For example, the terminal device first runs the social media platform to display its login interface. After logging in with their account, the user obtains an authorization code. In response to the user entering the authorization code, the client retrieves it. The client can then send the authorization code to the server.

[0094] For example, when using a toolkit to obtain web page content from a social media platform, the toolkit must obtain an authorization code before it can obtain authorization from the social media platform and thus acquire the web page content.

[0095] In this embodiment of the application, user login authorization for the social media platform is only required when the web page content of the social media platform is accessed for the first time.

[0096] 2. Obtain the webpage content based on the link information and access token.

[0097] Optionally, the client obtains authorization from the social media platform based on the access token, and can then access the web page content of the social media platform based on the link information.

[0098] In one example, containerization (Docker) technology can be used to solve licensing issues on social media platforms. Docker is an open-source containerization technology designed to simplify the development, deployment, and operation of applications.

[0099] For example, the process may include the following: building a Docker image file with an access token as a startup parameter; obtaining authorization information from the social media platform based on the Docker image file; and obtaining web page content based on the authorization information and link information.

[0100] A Docker image file is essentially a read-only file that contains the file system, source code, libraries, dependencies, tools, and other files necessary for running the application. In this embodiment, the Docker image file also includes an access token.

[0101] Optionally, the client can obtain the authorization information of the social media platform, i.e., the authorization code (access token), from the Docker image file. This allows the client to obtain the authorization of the social media platform by reconstructing the Docker image file each time it accesses web content from the social media platform, without requiring the user to log in and authorize again for that social media platform. This avoids the authorization login issue and enables login-free access to web content on both the terminal device and the server side, thereby effectively expanding the applicability of the technical solution provided in this application embodiment. It also reduces operational complexity and improves the efficiency and convenience of accessing web content from social media platforms.

[0102] In addition, the technical solutions provided in this application embodiment can be connected to multiple social media platforms, further expanding the application scenarios of the technical solutions provided in this application embodiment.

[0103] For example, before building an online service (i.e., a server-side service for retrieving web page content), an access token for the social media platform can be obtained through the client. When the server builds the Docker image file in the pipeline, the access token sent by the client can be passed as a startup parameter. Once the server is running, it can obtain the access token from the Docker image file, thus bypassing the authorization login issue of the social media platform and enabling the online service to successfully retrieve web page content from the social media platform, such as text, audio, video, and images.

[0104] Optionally, before the access token expires, the social media platform's toolkit will automatically refresh the login state, ensuring that the login state is maintained. Login state refers to a state created by the social media platform for a user after successful login, designed to ensure that every request from the user is recognized as a logged-in state, thereby granting access to protected resources.

[0105] In one example, when the executing entity in this application embodiment is a client, the tool library of the social media platform can write the access token to a local token.json file after obtaining the access token. The next time the tool library needs to obtain the web page content of the social media platform, it can use the access token in the token.json file to obtain the web page content of the social media platform without requiring the user to log in and authorize the social media platform again.

[0106] 3. Extract basic text data and media text data from the webpage content. Basic text data is used to indicate the text content in the webpage content, while media text data is data that describes the audio and video content in the webpage content in text form.

[0107] Optionally, the data for webpage content can be divided into basic data and media data. The basic data for webpage content may include at least one of the following: links, titles, descriptions, video duration, and pageviews. The media data for webpage content may include at least one of the following: subtitles, images, audio, and video. The aforementioned audio and video content may include at least one of the following: subtitle data, audio data, and video data.

[0108] The client can extract text content from the basic data of the web page content and define it as basic text data, and extract text content from the media data of the web page content and define it as media text data.

[0109] In one example, extracting basic text data and media text data from web page content can include the following:

[0110] (1) Extract basic text data from the web page content.

[0111] Optionally, the client iterates through the basic data of the webpage content to obtain the text content within the webpage content, thereby forming the basic text data.

[0112] (2) When the webpage content includes subtitle data of audio and video content, extract media text data from the subtitle data.

[0113] Optionally, if a subtitle file exists to record the subtitles corresponding to the subtitle data, the subtitles recorded in that file can be directly identified as media text data. If no subtitle file exists, subtitles (i.e., text content) can be extracted from each video frame corresponding to the audio and video content using text extraction techniques to obtain media text data. These text extraction techniques may include at least one of the following: OCR (Optical Character Recognition) technology, deep learning models (such as image recognition models), and Video Text Recognition (VTR) technology.

[0114] For example, obtain the video frame sequence corresponding to the audio and video content, input each video frame in the video frame sequence into the image recognition model in turn to obtain the subtitles in each video frame, form a subtitle set, delete the duplicate subtitles in the subtitle set to obtain the filtered subtitle set, and determine the filtered subtitle set as media text data.

[0115] Since the subtitle data contains the complete text content of the audio and video content, the text content extracted from the subtitle data can be directly identified as media text data without the need to acquire audio or video data. This avoids the acquisition and processing of audio or video data, which helps reduce the workload of acquiring media text data and thus improves the efficiency of text data acquisition.

[0116] Furthermore, since subtitle data itself contains text content, it does not require further conversion. Compared to audio or video data, which needs to be converted to text, this avoids errors introduced during the conversion process. Therefore, directly extracting media text data from subtitle data improves the accuracy of media text data acquisition. Additionally, the subtitle data itself accurately represents the text content, which further enhances the accuracy of media text data acquisition.

[0117] (3) Extract media text data from the audio data of the audio and video content when the web page content does not include subtitle data of audio and video content.

[0118] Optionally, the client can download the audio data of the audio and video content to its local machine, and then extract the text content from the audio data to determine it as media text data.

[0119] For example, audio recognition technology can be used to convert audio data into text data to obtain media text data. For instance, the audio data is compiled to obtain audio modal features; the audio modal features are input into a trained large language model, which processes the audio modal features to obtain the text data corresponding to the audio data; and the text content corresponding to the text data is determined as media text data.

[0120] Since audio data is generated based on subtitle data, it can accurately reflect the subtitle data to a certain extent. Furthermore, the processing complexity of audio data is lower than that of video data. Extracting media text data from audio data is beneficial to improving the efficiency of text data acquisition.

[0121] (4) Extract media text data from the video data of audio and video content when the webpage content does not include subtitle data and audio data of audio and video content.

[0122] Optionally, the client can download the video data of the audio and video content to its local machine, and then extract the text content from the video data to determine it as media text data.

[0123] For example, video processing techniques can be used to convert video data into text data to obtain media text data. For instance, video data is compiled to obtain video modal features; these features are then input into a trained large language model, which processes them to obtain text data corresponding to the video data. This text data can be used to describe the audio and video content; the text content corresponding to the text data is then identified as media text data.

[0124] In this embodiment, the acquisition priority of data with faster processing speed, lower processing difficulty, and higher processing accuracy among subtitle data, audio data, and video data is set higher. For example, the acquisition priority of subtitle data is higher than that of audio data, and the acquisition priority of audio data is higher than that of video data. This helps to reduce the workload of acquiring media text data and improve the acquisition efficiency and accuracy of media text data.

[0125] In one example, where the webpage content includes images, the client can also extract image text data from the images. Image text data is text data that describes the content of the image in text form, and the text data includes image text data.

[0126] Optionally, the client can define the text content extracted from the image as image-text data. Obtaining image-text data enriches the text data, which helps improve the accuracy of the summarized data, especially for summaries of web page content with multiple images.

[0127] 4. Based on basic text data and media text data, obtain the text data of the webpage content.

[0128] Optionally, the text data of the webpage content can be obtained by combining the basic text data and the media text data.

[0129] Optionally, the text data of the webpage content can be obtained on a per-video (i.e., audio and video content) basis. For example, each video may correspond to a portion of text data (including basic text data and media text data), and the text data of multiple videos can be combined to form the text data of the webpage content. The number of videos can also be set and adjusted by the user according to their needs. For example, the configuration information mentioned above includes the number of videos configured by the user. The number of videos can also be the same as the number of videos present in the webpage content; this embodiment of the application does not limit this.

[0130] This application embodiment extracts both basic text data and media text data of web page content simultaneously, resulting in more comprehensive text data. This improves the comprehensiveness and richness of the text data, thereby enhancing the accuracy of the summarized data.

[0131] Step 240: Analyze the text data to obtain summary data of the webpage content. The summary data is used to summarize and generalize the webpage content.

[0132] Analyzing text data refers to the process of summarizing and generalizing text content. Summary data of webpage content can be used to reflect the main content of the webpage. For example, if the webpage content includes at least one audio or video element, the summary data can include summary content (such as abstract information) corresponding to each of those audio or video elements (e.g., video).

[0133] Optionally, artificial intelligence tools or open-source tools can be used to analyze text data and obtain summary data of web page content. The artificial intelligence tools can be tools built based on artificial intelligence, while the open-source tools can be tools built based on a programming framework that helps use large language models in applications.

[0134] In one example, because there are size limitations on the text preceding and following the API call to an AI tool, the text data needs to be adjusted according to its length. For instance, this process might include the following:

[0135] 1. Get the text length of the text data.

[0136] Optionally, the number of tokens included in the text data can be determined as the text length of the text data. For example, the number of words included in the text data can be determined as the text length of the text data.

[0137] 2. If the text length is greater than the length threshold, divide the text data into at least two sub-text data, and the text length of the sub-text data is less than or equal to the length threshold.

[0138] The length threshold can be set and adjusted according to the context size limitations of the AI ​​tool. For example, for a big data model used to analyze text data, if its maximum supported token count is 16385, considering that the maximum supported token count is the sum of the input and output content of the big data model, 10000 tokens can be set as the length threshold to reserve more tokens for the output content. If the text length exceeds the length threshold, it can be determined that the text data is too long and needs to be segmented using open-source tools.

[0139] Optionally, during the analysis of text data, open-source tools can automatically analyze the text data based on the segmentation results. They can pass the summary data of each sub-text data to the prompt word as input data in the analysis process of the next sub-text data, and so on, thus avoiding the problem of token quantity limitations.

[0140] 3. Analyze at least two sub-text data in sequence according to the prompt words to obtain summary data. The prompt words are used to indicate the analysis method of the web page content.

[0141] The prompt words can be used to guide the large language model to analyze text data according to the analysis method indicated by the prompt words. For example, the prompt words can be used to specify at least one of the following: the number of tokens in the summary data, the output language of the summary data, and the file format of the summary data.

[0142] For example, refer to Figure 6 In prompt word 600, "content" refers to the text data of the webpage content, such as the text data of audio and video content. Prompt words "English" and "Simple Chinese" in prompt word 600 instruct the large language model to output analysis data in both English and Simplified Chinese. Prompt word "Json" in prompt word 600 instructs the large language model to output analysis data in JSON format.

[0143] Optionally, after the last of the at least two subtext data has been analyzed, the output data of the large language model is determined as the summary data of the webpage content.

[0144] Optionally, if the text length of the text data is less than or equal to the length threshold, the analysis data of the text data can be obtained directly using the aforementioned open-source tools without segmenting the text data.

[0145] For example, refer to Figure 7 The summary data 700 for the webpage content includes summary data for five audio and video content items, with the summary data 701 for each audio and video content item displayed in two languages. These five audio and video content items can be the five most popular videos on the social media platform in that country, reflecting the country's public opinion hotspots.

[0146] This application utilizes AI capabilities to analyze the text data of web page content, eliminating the need for manual text data analysis and thus improving the accuracy and efficiency of text data analysis.

[0147] Step 250: Provide the summarized data to the user account.

[0148] Optionally, the client pushes summary data corresponding to a specific task to the user account that created the task. For example, the client can push the summary data to the user account via the user's email address, such as generating an email message based on the summary data and sending the email message to the user account's email address. The client can also push summary data to the user account via third-party applications such as social entertainment applications or instant messaging applications; this embodiment of the application does not limit this approach.

[0149] For example, the client periodically checks whether a task has reached its push time. Once the push time has arrived, the client sends the summary data corresponding to the task to the user account that created the task via email. Optionally, if the same user creates multiple tasks, the client pushes the summary data corresponding to each task to the user account's email address in the order set by the user account.

[0150] Through the embodiments of this application, users only need to perform simple operations on a single client to automatically obtain various types of information from the configured social media platforms on a timely basis, which greatly improves the efficiency of obtaining information from social media platforms in daily life.

[0151] In one example, task summary data is presented as data records, meaning the client supports storing summary data for each task. This allows users to view historical task summary data, enabling them to effectively review and summarize the differences between historical and current public opinion hotspots (such as the similarities and differences in their impact on a particular game version). This summary data can also be shared with other user accounts based on permissions and whether it is public or private.

[0152] Optionally, for each summary data point, in response to a user's action to view the corresponding webpage content, the client displays the webpage content to the user, allowing the user to quickly view the webpage content in person.

[0153] For example, refer to Figure 8 The data recording interface 800 presents summary data of historical tasks in a data recording format. Users can view the summary data of historical tasks in the data recording interface 800 by searching for time, task name, and creator. Optionally, in response to a user's viewing operation of Video 1 in the data recording interface 800, the client displays the content of Video 1 to the user.

[0154] In summary, the technical solution provided in this application automatically retrieves the webpage content and summary data of the social media platform specified by the user's account by obtaining the platform's webpage content based on the platform's identifier information configured in the user's account. This eliminates the need for the user to manually view and summarize the social media platform's webpage content, thus improving the convenience and efficiency of data acquisition. Furthermore, it avoids the time-consuming and labor-intensive nature of manual methods, thereby reducing the cost of acquiring social media platform data.

[0155] In some embodiments, the media data of the aforementioned audio and video content includes audio data or video data. For the media data of audio and video content, audio-to-text technology can be used to extract the text content from the media data to obtain media text data. Audio-to-text technology is a technique that converts human speech into editable and searchable text. The process of acquiring media text data may also include the following in embodiments of this application.

[0156] 1. Obtain the duration of audio and video content.

[0157] Optionally, the duration of any one of the subtitle data, audio data, and video data of the audio / video content can be determined as the duration of the audio / video content.

[0158] For example, the playback duration of the video data in the audio and video content can be determined as the duration of the audio and video content.

[0159] 2. Determine the target text conversion model that matches the duration. The target text conversion model is for media data with a processing duration of less than a set threshold.

[0160] In this embodiment, the text conversion model can be a neural network model built using audio-to-text technology, which can be used to convert audio data into text content.

[0161] Optionally, the aforementioned text conversion model can refer to the text conversion model provided by a speech recognition tool. Speech recognition tools include several text conversion models with different parameter sizes: base, small, medium, and large. Among them, base has the smallest parameter size, the fastest processing speed, and the lowest accuracy; large has the largest parameter size, the slowest processing speed, and the highest accuracy. For the same audio segment, the processing speed of base, small, medium, and large decreases in that order, and the conversion quality increases in that order. This text conversion model is a general-purpose speech recognition model that can perform multilingual speech recognition and speech translation.

[0162] The processing time of the text conversion model increases with the duration of the audio and video. In order to balance the conversion effect and processing speed, this application divides the audio and video content into four categories: longer audio and video (>32 min), long audio and video (12 to 32 minutes), medium audio and video (2 to 12 minutes), and short audio and video (<2 minutes).

[0163] Longer audio and video files can be processed using the "base" setting, longer audio and video files using the "small" setting, medium-length audio and video files using the "medium" setting, and short audio and video files using the "large" setting, ensuring that the media data of the audio and video content can be processed within a set threshold. The set threshold can be configured and adjusted according to actual usage needs, such as 3 minutes.

[0164] In one example, if the duration of the audio / video content exceeds a first threshold, the first text conversion model is determined as the target text conversion model.

[0165] Optionally, the first threshold can be implemented as the aforementioned 32 minutes, and the first text conversion model can be implemented as the aforementioned base, that is, the parameter scale of the first text conversion model is minimized to ensure that the processing time of audio and video content is less than the set threshold.

[0166] If the duration of the audio / video content is less than or equal to the first threshold and greater than the second threshold, the second text conversion model is determined as the target text conversion model, wherein the first threshold is greater than the second threshold, and the parameter size of the second text conversion model is greater than the parameter size of the first text conversion model.

[0167] Optionally, the second threshold can be implemented as 12min as described above, and the second text conversion model can be implemented as small as described above.

[0168] If the duration of the audio / video content is less than or equal to the second threshold and greater than or equal to the third threshold, the third text conversion model is determined as the target text conversion model, wherein the second threshold is greater than the third threshold and the parameter size of the third text conversion model is greater than the parameter size of the second text conversion model.

[0169] Optionally, the third threshold can be implemented as the above-mentioned 2min, and the third text conversion model can be implemented as the above-mentioned medium.

[0170] If the duration of the audio / video content is less than the third threshold, the fourth text conversion model is determined as the target text conversion model. The third threshold is greater than the fourth threshold, and the parameter size of the fourth text conversion model is greater than the parameter size of the third text conversion model.

[0171] Alternatively, the third text conversion model can be implemented as described above (large).

[0172] For example, if the audio / video content is longer than 32 minutes, `base` can be used as the target text conversion model. If the audio / video content is shorter than 2 minutes, `large` can be used as the target text conversion model. This ensures that the media data for each audio / video content can be processed completely in a shorter time.

[0173] In a feasible example, the target text conversion model can be determined based on the duration of the audio / video content and the parameter size of the text conversion model.

[0174] For example, for any text conversion model, a weighted sum of the duration of the audio / video content and the parameter size of the text conversion model is obtained to obtain the first parameter of the text conversion model; text conversion models whose first parameter falls within a set range are determined as target text conversion models. The set range can be set and adjusted according to actual usage requirements. The set range can be based on the aforementioned set threshold, so that the processing time of the target text conversion model for media data is less than the set threshold.

[0175] Optionally, in the weighted summation process, the weight parameter for the duration of the audio and video content is greater than the weight parameter for the parameter size of the text conversion model. This helps to improve the processing speed of media data while ensuring the processing effect of the media data.

[0176] 3. Input the media data into the target text conversion model, and the target text conversion model extracts the media text data from the media data.

[0177] Optionally, the client extracts text content from the media data using a target text conversion model, and identifies the text content as media text data. For example, if the media data includes audio data, the client extracts text content from the audio data using a target text conversion model; if the media data includes video data, the client extracts text content from the video data using a target text conversion model.

[0178] Optionally, the aforementioned image text data can be obtained through large language models in artificial intelligence tools or other graph-to-text technologies, and this application embodiment does not limit this.

[0179] In one example, where the aforementioned media text data is extracted by a target text conversion model that matches the duration of the audio and video content, this application embodiment also supports a mechanism for error retry and model upgrade to improve the success rate of data acquisition. The implementation process of this mechanism may include the following.

[0180] 1. In cases where the summary data contains anomalies, determine a replacement text conversion model to replace the target text conversion model. The parameter size of the replacement text conversion model should be larger than that of the target text conversion model.

[0181] An anomaly can be determined when at least one of the following conditions is present: the summary data is empty, the summary data is in an incorrect format, the summary data is meaningless, the summary data contains garbled characters, the summary data acquisition failed, or meaningless text appears in the summary data.

[0182] The replacement text conversion model can be regarded as an upgraded model of the target text conversion model. Since the parameter scale of the replacement text conversion model is larger than that of the target text conversion model, the conversion effect of the replacement text conversion model is better than that of the target text conversion model.

[0183] For example, when using the aforementioned base method to extract media text data, if anomalies are found in the summarized data, the base method can be upgraded to the medium method, which has a higher accuracy. If anomalies still exist in the summarized data, the medium method can be further upgraded to the large method, which has an even higher accuracy. If anomalies still exist in the summarized data, it can be determined that the audio / video content is meaningless, and the task should be handled as a failure. Through this mechanism of error retry and model upgrade, the impact of instability in AI tools and text conversion models can be minimized, which helps improve the accuracy of the summarized data and reduces the influence of AI illusions, thereby improving the overall success rate of summarizing the data.

[0184] In a feasible example, the process of determining the replacement text transformation model can also be as follows:

[0185] (1) When there are anomalies in the summary data, obtain the degree of anomaly in the summary data. The degree of anomaly is used to indicate whether the anomaly can be optimized.

[0186] Optionally, the greater the probability that an anomaly can be optimized, the lower the level of anomaly. For example, if the summary data is meaningless, the anomaly level of the summary data can be determined as level three; if the summary data is in an incorrect format, the anomaly level of the summary data can be determined as level two; if the summary data is garbled, the anomaly level of the summary data can be determined as level one. This application does not limit this.

[0187] (2) Determine the replacement text conversion model to replace the target text conversion model based on the degree of anomaly.

[0188] Optionally, a replacement text conversion model matching the anomaly level can be selected from the remaining text conversion models whose parameter size is larger than that of the target text conversion model. For example, the higher the anomaly level, the larger the parameter size of the replacement text conversion model. For instance, when using the above-described base to extract media text data, if the anomaly level of the summarized data is level one, the base can be upgraded to medium, which has a higher processing accuracy. If the anomaly level of the summarized data is level three, medium can be directly upgraded to large; this embodiment of the application does not limit this.

[0189] This application's embodiments select a replacement text conversion model by summarizing the degree of anomalies in the data, without having to adjust the replacement text conversion model step by step through trial and error. This is beneficial to improving the convenience and accuracy of determining the replacement text conversion model, as well as the efficiency of generating summary data.

[0190] 2. Input the media data of audio and video content into the replacement text conversion model, and the replacement text conversion model extracts new media text data from the media data.

[0191] Optionally, the text content in the media data of audio and video content can be extracted again by replacing the text conversion model, and the newly obtained text content can be identified as new media text data.

[0192] 3. Analyze basic text data and new media text data to obtain new summary data of webpage content.

[0193] Optionally, by using basic text data and new media text data as new text data for web page content, and by using the aforementioned artificial intelligence tools and open-source tools to analyze the new text data, new summary data of the web page content can be obtained.

[0194] 4. The new summary data is determined as the summary data of the webpage content.

[0195] Optionally, if the new summary data is free of anomalies, it is identified as the summary data of the webpage content. If the new summary data contains anomalies, execution continues from the step of "determining the replacement text transformation model used to replace the target text transformation model".

[0196] Optionally, in the absence of a replacement text transformation model, the client analyzes the basic text data to obtain summary data of the webpage content. This means that if the media text data is deemed meaningless, the client can directly determine the summary data of the webpage content based on the basic text data.

[0197] Optionally, in the absence of audio or video content, the client analyzes the basic text data to obtain a summary of the webpage content. That is, the client can also directly determine the summary data of the webpage content based on the basic text data of the webpage content.

[0198] This application's embodiments support obtaining summary data of webpage content through basic text data, which helps improve the success rate of obtaining summary data.

[0199] In summary, the technical solution provided in this application extracts media text data by selecting a text conversion model with corresponding parameter scale based on the duration of the audio and video content. This is beneficial for balancing the extraction speed and quality of media text data.

[0200] In one example, taking a social media platform whose webpage content mainly includes audio and video content as an example, the technical solution provided by the embodiments of this application will be described. The embodiments of this application may also include the following contents.

[0201] refer to Figures 9 to 14 The technical solution provided in this application can be divided into four stages: social media information acquisition stage, data standardization processing stage, AI content summarization stage, and summary data push stage. Each stage will be described in detail below.

[0202] 1. Social media information acquisition stage.

[0203] Social media information can refer to the web page content of social media platforms.

[0204] like Figure 9 As shown, user 901 selects at least one social media platform through the task configuration interface. For any one of the at least one social media platform, the client creates one or more scheduled tasks based on the platform's identification information. For example, for social media platform 1, the client creates three scheduled tasks, each corresponding to a function page on social media platform 1. As another example, for social media platform 2, the client creates only one scheduled task. A scheduled task refers to a task that is executed at a set time, such as a task executed daily.

[0205] Optionally, for any scheduled task, the client uses a headless browser to retrieve the corresponding link information (such as URL) from the social media platform associated with the scheduled task each day, and uses the application programming interface (API) and toolkit of the headless browser or the social media platform to retrieve the web page content under that link information. The client stores the link information and web page content in database 902.

[0206] Optionally, for each social media platform, the client can achieve login-free authorization by building a Docker image file using the social media platform's access token as a startup parameter, and then periodically retrieve the web page content of the social media platform.

[0207] 2. Data standardization processing stage.

[0208] The data standardization processing stage is used to obtain the text data of web page content from social media platforms. This process is used by the client for each social media platform. This facilitates subsequent expansion to other social media platforms, thereby improving the scalability of the technical solution provided in this application's embodiments.

[0209] like Figure 10 As shown, for webpage content on various social media platforms, after obtaining the basic text data of the webpage content, client 1000 determines whether the webpage content contains media data. If the webpage content does not contain media data, the process of obtaining the text data of the webpage content ends. Here, media data refers to audio and video content. Optionally, the aforementioned audio and video content can be implemented as videos for game applications, such as game videos posted by users.

[0210] When media data exists in the webpage content, client 1000 prioritizes extracting media text data from the subtitle data of the audio / video content. When no subtitle data exists, client 1000 prioritizes extracting media text data from the audio data of the audio / video content. When neither subtitle nor audio data exists, client 1000 extracts media text data from the video data of the audio / video content.

[0211] Optionally, for audio and video data, the client 1000 selects a target text conversion model from the base, small, medium, and large options included in the speech recognition tool based on the duration of the audio and video content, so as to extract media text data through the target text conversion model.

[0212] Optionally, if the webpage content does not contain media data, the client 1000 determines the basic text data as the text data of the webpage content; if the webpage content contains media data, the client 1000 combines the basic text data and the media text data to form the text data of the webpage content.

[0213] In one example, for future scalability, this application embodiment provides a base class BaseChannel, which is used to extract common attributes and methods of various social media platforms, such as basic video information, and methods for downloading audio and video data.

[0214] For example, refer to Figure 11 The base class 1100 (BaseChannel) includes attributes such as URL, name, and video_info, as well as abstract methods such as get_info and download_local. The get_info method is used to retrieve basic data of the webpage content, while the download_local method is used to download media data (such as audio and video content) of the webpage content. video_info refers to basic information about the audio and video content, including the video ID (Identity Document), video link, video duration, number of views, number of likes, number of comments, and audio / video URLs.

[0215] Each social media platform inherits from this base class 1100 to implement these two abstract methods. Clients can use these methods to retrieve the webpage content of each social media platform based on the link provided. For example... Figure 11 As shown, Channel A, Channel B, and Channel C each correspond to a social media platform, and all three social media platforms inherit from base class 1100.

[0216] 3. AI content summary stage.

[0217] The AI ​​content summary stage is the stage of acquiring summary data of web page content.

[0218] like Figure 12 As shown, after obtaining the text data of the webpage content, client 1000 uses artificial intelligence tools and open-source tools to analyze the text data and obtain summary data of the webpage content.

[0219] When the text data's length exceeds a length threshold, client 1000 uses open-source tools to divide the text data into at least two sub-text data, and constructs a summary chain based on these at least two sub-text data. When the text data's length is less than or equal to the length threshold, client 1000 can directly identify the text data as a summary chain. When the text data is divided into at least two sub-text data, the summary chain can be used to indicate the processing order of each sub-text data; for example, by sorting each sub-text data sequentially according to its order of arrangement within the text data, the summary chain can be obtained.

[0220] Open-source tools can automatically summarize summary chains to obtain summary data of web page content. For example, by using a large language model to analyze the summary chain in the guidance phase of prompt words, summary data of web page content can be obtained.

[0221] like Figure 13As shown, in the event of a failed summary (i.e., the summarized data is abnormal), client 1000 executes a mechanism for error retry and model upgrade. If a replacement text conversion model exists with a parameter size larger than the target text conversion model, client 1000 replaces the target text conversion model with the replacement model and re-acquires the media text data using the replacement model, then analyzes the text data again to obtain new summary data. If the new summary data is normal (i.e., the summary is successful), the AI ​​content summary phase ends; otherwise, the model continues to be upgraded until the summary is successful or no replacement text conversion model is found.

[0222] In the absence of a replacement text conversion model, client 1000 directly analyzes the basic text data of web page content using artificial intelligence tools and open-source tools to obtain summary data of the web page content.

[0223] 4. Summary of the data push phase.

[0224] like Figure 14 As shown, for each scheduled task, when the scheduled task's push time arrives, the client 1000 determines whether the scheduled task has been summarized (i.e., whether the summary data of the scheduled task has been successfully obtained). If the scheduled task has been summarized, the client 1000 constructs push content based on the summary data and pushes the push content to user 901, such as by pushing the push content to user 901 via email.

[0225] If the scheduled task's push time has not arrived or the scheduled task has not been completed, client 1000 continues to execute the scheduled task. Optionally, if the scheduled task's execution time has arrived, the client executes the scheduled task.

[0226] In one example, this application embodiment also supports users manually creating temporary tasks so that users can better analyze web page content of interest, such as the web page containing a game video.

[0227] like Figure 15As shown, the client displays a one-click summary page 1500. Users can input the video's URL via option 1501 and control the client to summarize the video's content via trigger control 1502, obtaining the video's summary data 1503. The client retrieves the webpage content containing the video based on its URL. This webpage content includes not only the video's media data but also basic data. The client extracts the text data from the webpage content, analyzes it, and obtains the webpage content's summary data (including the video's summary data). If the webpage content contains only one video, the webpage content's summary data is the same as that video's summary data. Optionally, the client's method for obtaining the video's summary data is the same as in the above embodiment, and will not be repeated here.

[0228] Optionally, if the user selects both English and Chinese as the output languages, the summary data 1503 includes summary data described in English and summary data described in Chinese.

[0229] In summary, by using the technical solution provided in this application, users do not need to access social media platforms separately, nor do they need to view the details of each link, post, or video. They can conveniently obtain public opinion hotspots from different social media platforms and countries simply by clicking to create a task on the client.

[0230] Furthermore, by employing the social media information acquisition stage and the data standardization processing stage, the technical solution provided in this application embodiment can be adapted to multiple social media platforms, which helps to improve the versatility and applicability of the technical solution provided in this application embodiment.

[0231] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0232] Please refer to Figure 16 This diagram illustrates a block diagram of a data processing apparatus according to an embodiment of this application. The apparatus has the function of implementing the above-described data processing method; this function can be implemented in hardware or by hardware executing corresponding software. The apparatus can be a computer device or can be installed within a computer device. The apparatus 1600 may include: a configuration information acquisition module 1601, a link information acquisition module 1602, a text data acquisition module 1603, a summary data acquisition module 1604, and a summary data push module 1605.

[0233] The configuration information acquisition module 1601 is used to acquire the configuration information of the user account, including the identification information of the social media platform.

[0234] The link information acquisition module 1602 is used to acquire link information of the web page content of the social media platform according to the configuration information.

[0235] The text data acquisition module 1603 is used to acquire text data of the webpage content based on the link information, wherein the text data is data describing the webpage content in text form.

[0236] The summary data acquisition module 1604 is used to analyze the text data to obtain summary data of the webpage content, wherein the summary data is data used to summarize and generalize the webpage content.

[0237] The summary data push module 1605 is used to provide the summary data to the user account.

[0238] In some embodiments, such as Figure 17 As shown, the text data acquisition module 1603 includes: an access token acquisition submodule 1603a, a web page content acquisition submodule 1603b, a web page content extraction submodule 1603c, and a text data acquisition submodule 1603d.

[0239] Access token acquisition submodule 1603a is used to acquire the access token of the social media platform.

[0240] The webpage content acquisition submodule 1603b is used to acquire the webpage content based on the link information and the access token.

[0241] The webpage content extraction submodule 1603c is used to extract basic text data and media text data from the webpage content. The basic text data is used to indicate the text content in the webpage content, and the media text data is data that describes the audio and video content in the webpage content in text form.

[0242] The text data acquisition submodule 1603d is used to acquire the text data of the webpage content based on the basic text data and the media text data.

[0243] In some embodiments, the webpage content extraction submodule 1603c is further configured to:

[0244] Extract the basic text data from the webpage content;

[0245] If the webpage content includes subtitle data of the audio and video content, extract the media text data from the subtitle data;

[0246] If the webpage content does not include subtitle data for the audio / video content, the media text data is extracted from the audio data of the audio / video content.

[0247] If the webpage content does not include subtitle data and audio data of the audio and video content, the media text data is extracted from the video data of the audio and video content.

[0248] In some embodiments, the web page content extraction submodule 1603c is further configured to extract image text data from the image when the web page content includes an image, wherein the image text data is text data describing the content of the image in text form, and the text data includes the image text data.

[0249] In some embodiments, the media data of the audio and video content includes audio data or video data; the webpage content extraction submodule 1603c is further configured to:

[0250] Obtain the duration of the audio and video content;

[0251] Determine a target text conversion model that matches the duration, wherein the target text conversion model is for media data whose processing duration is less than a set threshold;

[0252] The media data is input into the target text conversion model, and the target text conversion model extracts the media text data from the media data.

[0253] In some embodiments, the webpage content extraction submodule 1603c is further configured to:

[0254] If the duration of the audio and video content exceeds a first threshold, the first text conversion model is determined as the target text conversion model;

[0255] If the duration of the audio and video content is less than or equal to the first threshold and greater than the second threshold, the second text conversion model is determined as the target text conversion model, wherein the first threshold is greater than the second threshold and the parameter size of the second text conversion model is greater than the parameter size of the first text conversion model.

[0256] If the duration of the audio and video content is less than or equal to the second threshold and greater than or equal to the third threshold, the third text conversion model is determined as the target text conversion model, wherein the second threshold is greater than the third threshold, and the parameter size of the third text conversion model is greater than the parameter size of the second text conversion model.

[0257] If the duration of the audio / video content is less than the third threshold, the fourth text conversion model is determined as the target text conversion model, wherein the third threshold is greater than the fourth threshold, and the parameter size of the fourth text conversion model is greater than the parameter size of the third text conversion model.

[0258] In some embodiments, the webpage content acquisition submodule 1603b is further configured to:

[0259] Use the access token as a startup parameter to build a Docker image file;

[0260] Based on the Docker image file, obtain the authorization information of the social media platform;

[0261] The webpage content is obtained based on the authorization information and the link information.

[0262] In some embodiments, the summary data acquisition module 1604 is further configured to:

[0263] Obtain the text length of the text data;

[0264] If the text length is greater than a length threshold, the text data is divided into at least two sub-text data, and the text length of the sub-text data is less than or equal to the length threshold.

[0265] The summary data is obtained by sequentially analyzing the at least two sub-text data based on the prompt words, wherein the prompt words are used to indicate the analysis method of the webpage content.

[0266] In some embodiments, the media text data is extracted by a target text conversion model that matches the duration of the audio and video content; the summary data acquisition module 1604 is further configured to:

[0267] In the event of anomalies in the summarized data, an alternative text conversion model is determined to replace the target text conversion model, wherein the parameter size of the alternative text conversion model is larger than that of the target text conversion model.

[0268] The media data of the audio and video content is input into the replacement text conversion model, and the replacement text conversion model extracts new media text data from the media data;

[0269] By analyzing the basic text data and the new media text data, new summary data of the webpage content is obtained;

[0270] The new summary data is determined as the summary data of the webpage content.

[0271] In some embodiments, the summary data acquisition module 1604 is further configured to:

[0272] In the absence of the replacement text conversion model, the basic text data is analyzed to obtain summary data of the webpage content;

[0273] Alternatively, in the absence of the audio and video content, the basic text data can be analyzed to obtain a summary of the webpage content.

[0274] In some embodiments, the configuration information further includes at least one keyword; the link information acquisition module 1602 is further configured to:

[0275] The step of obtaining the link information of the web page content of the social media platform according to the configuration information includes:

[0276] For any one of the at least one keywords, determine the link information of the web page content related to the keyword on the social media platform.

[0277] In summary, the technical solution provided in this application automatically retrieves the webpage content and summary data of the social media platform specified by the user's account by obtaining the platform's webpage content based on the platform's identifier information configured in the user's account. This eliminates the need for the user to manually view and summarize the social media platform's webpage content, thus improving the convenience and efficiency of data acquisition. Furthermore, it avoids the time-consuming and labor-intensive nature of manual methods, thereby reducing the cost of acquiring social media platform data.

[0278] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0279] Please refer to Figure 18 This illustrates a structural block diagram of a computer device provided in one embodiment of this application. The computer device 1800 can be implemented as... Figure 1 The terminal device 10 or server 20 shown.

[0280] Typically, computer device 1800 includes a processor 1801 and a memory 1802.

[0281] Processor 1801 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1801 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1801 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1801 may integrate a GPU, which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 1801 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0282] The memory 1802 may include one or more computer-readable storage media, which may be tangible and non-transitory. The memory 1802 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1802 stores a computer program that is loaded and executed by the processor 1801 to implement the data processing method described above.

[0283] Those skilled in the art will understand that Figure 18 The structure shown does not constitute a limitation on the computer device 1800, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0284] In some embodiments, a computer-readable storage medium is also provided, wherein a computer program is stored therein, the computer program being loaded and executed by a processor to implement the above-described data processing method.

[0285] Optionally, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random-Access Memory), SSD (Solid State Drives), or optical disc, etc. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0286] In some embodiments, a computer program product is also provided, the computer program product including a computer program stored in a computer-readable storage medium, and a processor reading from the computer-readable storage medium and executing the computer program to implement the above-described data processing method.

[0287] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0288] The above are merely exemplary embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims

1. A data processing method, characterized by, The method includes: Obtain the configuration information of the user account, including the identification information of the social media platform; Based on the configuration information, obtain the link information of the web page content of the social media platform; Based on the link information, obtain the text data of the webpage content, wherein the text data is data describing the webpage content in text form; Analyze the text data to obtain summary data of the webpage content, wherein the summary data is used to summarize and generalize the webpage content; The summarized data is provided to the user account.

2. The method of claim 1, wherein, The step of obtaining the text data of the webpage content based on the link information includes: Obtain the access token for the social media platform; Based on the link information and the access token, obtain the webpage content; Extract basic text data and media text data from the web page content. The basic text data is used to indicate the text content in the web page content, and the media text data is data that describes the audio and video content in the web page content in text form. Based on the basic text data and the media text data, obtain the text data of the webpage content.

3. The method of claim 2, wherein, The extraction of basic text data and media text data from the webpage content includes: Extract the basic text data from the webpage content; If the webpage content includes subtitle data of the audio and video content, extract the media text data from the subtitle data; If the webpage content does not include subtitle data for the audio / video content, the media text data is extracted from the audio data of the audio / video content. If the webpage content does not include subtitle data and audio data of the audio and video content, the media text data is extracted from the video data of the audio and video content.

4. The method according to claim 2 or 3, characterized in that, The method further includes: When the webpage content includes images, image text data is extracted from the images. The image text data is text data that describes the content of the image in text form, and the text data includes the image text data.

5. The method according to any one of claims 2 to 4, characterized in that, The media data of the audio and video content includes audio data or video data; the method further includes: Obtain the duration of the audio and video content; Determine a target text conversion model that matches the duration, wherein the target text conversion model is for media data whose processing duration is less than a set threshold; The media data is input into the target text conversion model, and the target text conversion model extracts the media text data from the media data.

6. The method according to claim 5, characterized in that, The determination of the target text conversion model that matches the duration includes: If the duration of the audio and video content exceeds a first threshold, the first text conversion model is determined as the target text conversion model; If the duration of the audio and video content is less than or equal to the first threshold and greater than the second threshold, the second text conversion model is determined as the target text conversion model, wherein the first threshold is greater than the second threshold and the parameter size of the second text conversion model is greater than the parameter size of the first text conversion model. If the duration of the audio and video content is less than or equal to the second threshold and greater than or equal to the third threshold, the third text conversion model is determined as the target text conversion model, wherein the second threshold is greater than the third threshold, and the parameter size of the third text conversion model is greater than the parameter size of the second text conversion model. If the duration of the audio / video content is less than the third threshold, the fourth text conversion model is determined as the target text conversion model, wherein the third threshold is greater than the fourth threshold, and the parameter size of the fourth text conversion model is greater than the parameter size of the third text conversion model.

7. The method according to any one of claims 2 to 6, characterized in that, The step of obtaining the webpage content based on the link information and the access token includes: Use the access token as a startup parameter to build a Docker image file; Based on the Docker image file, obtain the authorization information of the social media platform; The webpage content is obtained based on the authorization information and the link information.

8. The method according to any one of claims 1 to 7, characterized in that, The analysis of the text data to obtain summary data of the webpage content includes: Obtain the text length of the text data; If the text length is greater than a length threshold, the text data is divided into at least two sub-text data, and the text length of the sub-text data is less than or equal to the length threshold. The summary data is obtained by sequentially analyzing the at least two sub-text data based on the prompt words, wherein the prompt words are used to indicate the analysis method of the webpage content.

9. The method according to any one of claims 2 to 8, characterized in that, The media text data is extracted by a target text conversion model that matches the duration of the audio and video content; the method further includes: In the event of anomalies in the summarized data, an alternative text conversion model is determined to replace the target text conversion model, wherein the parameter size of the alternative text conversion model is larger than that of the target text conversion model. The media data of the audio and video content is input into the replacement text conversion model, and the replacement text conversion model extracts new media text data from the media data; By analyzing the basic text data and the new media text data, new summary data of the webpage content is obtained; The new summary data is determined as the summary data of the webpage content.

10. The method according to claim 9, characterized in that, The method further includes: In the absence of the replacement text conversion model, the basic text data is analyzed to obtain summary data of the webpage content; or, In the absence of the audio and video content, the basic text data is analyzed to obtain a summary of the webpage content.

11. The method according to any one of claims 1 to 10, characterized in that, The configuration information also includes at least one keyword; The step of obtaining the link information of the web page content of the social media platform according to the configuration information includes: For any one of the at least one keywords, determine the link information of the web page content related to the keyword on the social media platform.

12. A data processing apparatus, characterized in that, The device includes: The configuration information acquisition module is used to acquire the configuration information of the user account, including the identification information of the social media platform; The link information acquisition module is used to acquire link information of the web page content of the social media platform according to the configuration information; The text data acquisition module is used to acquire text data of the webpage content based on the link information, wherein the text data is data describing the webpage content in text form; The summary data acquisition module is used to analyze the text data and obtain summary data of the webpage content. The summary data is data used to summarize and generalize the webpage content. The summary data push module is used to provide the summary data to the user account.

13. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the method as claimed in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the method as described in any one of claims 1 to 11.

15. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, which a processor reads from and executes to implement the method as described in any one of claims 1 to 11.

Images