Social media data processing method and device, computer equipment and medium

Abstract

The invention belongs to the technical field of data processing, and discloses a social media data processing method and device, computer equipment and a medium. The method comprises the following steps: acquiring various types of media data from a plurality of social media platforms through an interface adapter; converting the media data into structured data according to a preset data conversion rule, and storing the structured data in a database; receiving a data analysis instruction, and extracting to-be-processed target data from the database based on a screening condition in the data analysis instruction; inputting the target data into a pre-trained topic analysis model for topic extraction to obtain a topic recognition result; inputting the target data into a pre-trained sentiment analysis model for sentiment analysis to obtain a sentiment classification result; and combining the subject recognition result and the sentiment classification result to obtain a final data analysis result. The media data can be comprehensively collected and deeply analyzed, so that the value of the media data is fully utilized.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, computer equipment and medium for processing social media data. Background Art

[0002] Currently, most of the data collection tools on the market have a single function. Most of them only support the collection of a single type of data. For example, some tools only collect text data, but cannot collect image data, video data, etc. At the same time, there is also the problem of platform limitations. Existing data collection tools can only obtain data from a single social media platform. For example, some data collection tools only support the collection of data from media platform A, and cannot obtain data from other media platforms, resulting in a single data sample; if data from multiple media platforms is required, it needs to be collected through multiple single data collection tools, which increases resource consumption and operational complexity. In addition, after obtaining data, existing data collection methods often lack in-depth data analysis and insight capabilities, so that the potential value of the data is not fully utilized. Summary of the invention

[0003] To this end, the embodiments of the present application provide a method, apparatus, computer device, and medium for processing social media data, which achieve comprehensive collection and in-depth analysis of media data and fully utilize the value of media data.

[0004] In a first aspect, the present application provides a method for processing social media data.

[0005] This application is achieved through the following technical solutions:

[0006] A method for processing social media data, comprising:

[0007] Acquire media data from multiple social media platforms through an interface adapter, wherein the data types of the media data include text data, image data, audio data, and video data;

[0008] Converting the media data into structured data according to a preset data conversion rule, and storing the structured data in a database, wherein the structured data is data organized by the same field structure;

[0009] Receive a data analysis instruction, and extract target data to be processed from a database based on screening conditions in the data analysis instruction;

[0010] Inputting the target data into a pre-trained topic analysis model to extract topics and obtain topic recognition results;

[0011] Input the target data into a pre-trained sentiment analysis model to perform sentiment analysis and obtain a sentiment classification result;

[0012] The topic identification result and the sentiment classification result are combined to obtain the final data analysis result.

[0013] In a preferred example of the present application, it can be further configured to include:

[0014] Encapsulate the interface adapters of all social media platforms in the interface management module for unified deployment, where the interface adapters are developed based on the API documents of the social media platforms;

[0015] When the authentication requirements of the social media platform are updated, the interface management module adjusts the interface adapter according to the current authentication requirements of the social media platform and the current API documentation.

[0016] In a preferred example of the present application, it can be further configured that before converting the media data into structured data according to a preset data conversion rule, the method further includes:

[0017] Normalize the data formats of media data from different social media platforms;

[0018] Use mean filling method or median filling method to fill the missing values ​​in the normalized media data;

[0019] A hash function or Bloom filter is used to deduplicate the padded media data.

[0020] In a preferred example of the present application, it can be further configured that the media data is converted into structured data according to a preset data conversion rule, including:

[0021] When the media data is text data, the text data is stored as structured data according to a preset field structure.

[0022] In a preferred example of the present application, it can be further configured that the media data is converted into structured data according to a preset data conversion rule, including:

[0023] When the media data is image data, the description information in the image data is identified by an image recognition algorithm, and the description information is stored as structured data according to a preset field structure.

[0024] In a preferred example of the present application, it can be further configured that the media data is converted into structured data according to a preset data conversion rule, including:

[0025] When the media data is audio data, a speech transcription result is obtained from the audio data through a speech-to-text service, and the speech transcription result is stored as structured data according to a preset field structure.

[0026] In a preferred example of the present application, it can be further configured that the media data is converted into structured data according to a preset data conversion rule, including:

[0027] When the media data is video data, splitting the video data into audio data and image data;

[0028] Obtaining a speech transcription result from the audio data through a speech-to-text service;

[0029] Identify the description information in the image data through the image recognition algorithm;

[0030] The speech transcription result and the description information are stored as structured data according to a preset field structure.

[0031] In a preferred example of the present application, it can be further configured that the topic analysis model is obtained by training the LDA topic model, the target data is input into the pre-trained topic analysis model for topic extraction, and a topic recognition result is obtained, including:

[0032] Perform word segmentation on the target data to obtain a word segmentation list;

[0033] Use the TF-IDF algorithm to extract keywords from the word list and obtain multiple predicted keywords;

[0034] Performing data cleaning on multiple predicted keywords based on a preset stop word list to obtain multiple keywords after cleaning;

[0035] The cleaned keywords are input into the pre-trained LDA topic model to obtain the probability values ​​of the keywords belonging to the preset topics. The keywords are sorted from large to small according to the probability values, and the topics corresponding to the preset number of keywords are selected as the topic recognition results.

[0036] In a preferred example of the present application, it can be further configured that the sentiment analysis model is obtained by training the BERT module and the LSTM module, and the target data is input into the pre-trained sentiment analysis model for sentiment analysis to obtain a sentiment classification result, including:

[0037] Input the target data into the BERT module to extract text features and obtain the context vector representation of the target data;

[0038] The context vector representation is input into the LSTM module to extract features through convolution and pooling operations, and the extracted text features are input into the linear layer to obtain the predicted sentiment classification results.

[0039] In a preferred example of the present application, it can be further configured to obtain media data from multiple social media platforms through an interface adapter, including:

[0040] Use multi-threading technology or asynchronous calling technology to obtain media data from multiple social media platforms.

[0041] In a second aspect, the present application provides a device for processing social media data.

[0042] This application is achieved through the following technical solutions:

[0043] A device for processing social media data, the device being used to execute the method for processing social media data according to the first aspect, the device comprising:

[0044] A data acquisition module, used to acquire media data from multiple social media platforms through an interface adapter, wherein the data types of the media data include text data, image data, audio data and video data;

[0045] A data conversion module, used to convert the media data into structured data according to a preset data conversion rule, and store the structured data in a database, wherein the structured data is data organized by the same field structure;

[0046] A target data extraction module is used to receive a data analysis instruction and extract the target data to be processed from the database based on the screening conditions in the data analysis instruction;

[0047] The first analysis module is used to extract themes from target data and obtain theme recognition results;

[0048] The second analysis module is used to perform sentiment analysis on the target data to obtain sentiment classification results;

[0049] The result summary module is used to combine the topic identification result and the sentiment classification result to obtain the final data analysis result.

[0050] In a preferred example of the present application, it can be further configured that the data acquisition module also includes an interface management unit, which is used to encapsulate the interface adapters of all social media platforms in the interface management module for unified deployment, wherein the interface adapters are developed corresponding to the API documents of the social media platform; when the authentication requirements of the social media platform are updated, the interface management module adjusts the interface adapter according to the current authentication requirements of the social media platform and the current API documents.

[0051] In a preferred example of the present application, it can be further configured that the data conversion module also includes:

[0052] The data preprocessing unit is used to normalize the data formats of media data from different social media platforms; fill in missing values ​​in the normalized media data using the mean filling method or the median filling method; and deduplicate the filled media data using a hash function or a Bloom filter.

[0053] On the third aspect, the present application is realized through the following technical solutions:

[0054] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned methods for processing social media data when executing the computer program.

[0055] In a fourth aspect, the present application provides a computer-readable storage medium.

[0056] This application is achieved through the following technical solutions:

[0057] A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of any one of the above-mentioned methods for processing social media data.

[0058] In summary, compared with the prior art, the technical solution provided in the embodiment of the present application has at least the following beneficial effects:

[0059] This application obtains various types of media data from multiple social media platforms through an interface adapter; converts the media data into structured data according to preset data conversion rules, and stores the structured data in a database, wherein the structured data is data organized by the same field structure; receives data analysis instructions, and extracts target data to be processed from the database based on the screening conditions in the data analysis instructions; inputs the target data into a pre-trained topic analysis model for topic extraction to obtain topic identification results; inputs the target data into a pre-trained sentiment analysis model for sentiment analysis to obtain sentiment classification results; combines the topic identification results with the sentiment classification results to obtain the final data analysis results. Comprehensive collection of media data from different platforms can capture a wider range of user perspectives, and in-depth analysis and parsing of these data can be performed through deep learning models to mine the deep-level information behind the data, thereby understanding the user's true emotional tendencies and discussion trends. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1A flowchart of a method for processing social media data provided in one embodiment of the present application;

[0061] Figure 2 A schematic diagram of the structure of a device for processing social media data provided in one embodiment of the present application;

[0062] Description of reference numerals:

[0063] Data acquisition module-101, data conversion module-102, target data extraction module-103, first analysis module-104, second analysis module-105, result summary module-106. DETAILED DESCRIPTION

[0064] This specific embodiment is merely an explanation of the present application and is not a limitation of the present application. After reading this specification, those skilled in the art may make modifications to the present embodiment without any creative contribution as needed, but such modifications are protected by the patent law as long as they are within the scope of the claims of the present application.

[0065] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0066] In addition, the term "and / or" in this application is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application, unless otherwise specified, generally indicates that the associated objects before and after are in an "or" relationship.

[0067] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with basically the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there any limitation on quantity and execution order.

[0068] The following is a further detailed description of the embodiments of the present application in conjunction with the accompanying drawings. Figure 1 As shown, in order to overcome the deficiencies in the prior art, the social media data processing method proposed in this application includes the following steps:

[0069] S10: Acquire media data from multiple social media platforms through an interface adapter, where the data types of the media data include text data, image data, audio data, and video data.

[0070] Specifically, social media platforms include Weibo, WeChat, Douyin, Xiaohongshu, Zhihu, Kuaishou, Baidu Tieba, Douyin, Douban and other social media. Develop specific interface adapters for different social media platforms, and associate the interface adapters with social media platforms. When it is necessary to obtain media data from certain social media platforms, call the corresponding interface adapter to obtain the media data. It should be noted that media data of all data types can be obtained from a social media platform according to data acquisition requirements, or media data of some data types can be obtained according to requirements.

[0071] S20: Convert the media data into structured data according to a preset data conversion rule, and store the structured data in a database, wherein the structured data is data organized by the same field structure.

[0072] Because different social media platforms and data sources may use different data formats, in order to unify management and analysis, it is necessary to convert these media data into data in a unified format with the same field structure. Specifically, structured data can be stored in a NoSQL database that supports multiple data types. NoSQL databases can provide high-performance data reading and writing capabilities, especially when processing large amounts of distributed data, such as MongoDB databases. When storing data, in order to improve the efficiency of data retrieval, a dual indexing strategy based on data type index, data source index, or data type and data source index is adopted. Among them, based on the data type index, media data is stored in different collections according to data type, so that media data of the same type can be quickly located and retrieved. Based on the data source index, media data is stored in different collections according to source, so that media data from the same social media platform can be quickly located and retrieved. Based on the dual indexing of data type and data source, sub-collections can be further set according to data source on the basis of indexing according to data type; sub-collections can also be further set according to data type on the basis of indexing according to data source.

[0073] S30: Receive a data analysis instruction, and extract target data to be processed from a database based on screening conditions in the data analysis instruction.

[0074] The data analysis instruction may come from a user who needs to perform media data analysis, or from a third-party system that needs to perform media data analysis. The data analysis instruction includes the range of media data that needs to be analyzed (i.e., screening conditions), specifically, the time range, data type, and data source. Based on the above screening conditions, the target data is extracted from the database in step S20.

[0075] S40: Input the target data into the pre-trained topic analysis model to extract topics and obtain topic recognition results.

[0076] The acquired target data is used as input data and input into the pre-trained topic analysis model. The topic analysis model analyzes and extracts the target data and obtains the topic recognition result. Specifically, the topic recognition result is the main topic contained in the target data, such as "hot news", "entertainment news", "social events", "technological trends", "technological development", etc. Through these topic recognition results, we can better understand the content distribution of the target data and provide support for content management, information retrieval, user recommendations, etc. The topic analysis model is trained based on the deep learning algorithm.

[0077] S50: Input the target data into the pre-trained sentiment analysis model for sentiment analysis to obtain the sentiment classification result.

[0078] Load the pre-trained sentiment analysis model and input the target data into the sentiment analysis model. The sentiment analysis model learns the text features of the target data and obtains the sentiment tendency in the target data. After analyzing the sentiment tendency, the sentiment classification result is obtained. The sentiment classification result includes the positive sentiment, neutral sentiment and negative sentiment contained in the target data, as well as the corresponding probability values. The sentiment analysis results in the target data are directly displayed to the user in the form of sentiment labels and probability values. The sentiment analysis model is obtained by training the BERT module and LSTM module using public media data and corresponding sentiment labels.

[0079] S60: Combine the topic recognition results and the sentiment classification results to obtain the final data analysis results.

[0080] Specifically, the topic recognition results and sentiment classification results obtained from the above analysis can be combined to generate analysis reports according to user needs. The analysis reports can provide data overviews in the form of bar charts, line charts, pie charts, and heat maps. At the same time, an API interface for data access and query is provided for third-party systems to support further use and integration of data.

[0081] The processing method proposed in this application obtains various types of media data from multiple social media platforms through an interface adapter, converts the media data into a unified format, and then analyzes the media data through multiple pre-trained deep learning models to obtain topic recognition results and sentiment classification results, and finally combines multiple analysis results to obtain the final data analysis results. Comprehensive collection of media data from different platforms can capture a wider range of user perspectives, and conduct in-depth analysis and parsing of these data through deep learning models, digging out the deep-level information behind the data, and understanding the user's real emotional tendencies and discussion trends, which can help companies adjust strategies and improve products and services in a timely manner to adapt to market changes, thereby improving user experience.

[0082] In some feasible embodiments, all interface adapters of social media platforms are encapsulated in an interface management module for unified deployment, wherein the interface adapter is developed based on the API document of the social media platform; when the authentication requirements of the social media platform are updated, the interface management module adjusts the interface adapter according to the current authentication requirements and current API document of the social media platform. Specifically, the API document of the target social media platform is obtained, and the corresponding interface adapter is developed for the API document. The interface adapter defines the interaction mode between systems, including the data calling mode and the error handling mechanism. All developed interface adapters are encapsulated in an interface management module for unified deployment. The interface management module is responsible for initializing each interface adapter, managing the life cycle of the interface adapter, and providing a unified interface for external calls. Encapsulation can be achieved through the adapter mode. At the same time, the authentication requirements of the social media platform may change. The interface management module is responsible for listening to the update notification of the social media platform, regularly checking the changes in the API document, and once a change of a social media platform is detected, the interface adapter is updated according to the new authentication requirements (i.e., the current authentication requirements) and the new API document (i.e., the current API document). Through a flexible and maintainable interface management method, seamless connection with the social media platform is guaranteed.

[0083] In some feasible embodiments, before converting the media data into structured data according to a preset data conversion rule, the method further includes: normalizing the data formats of media data from different social media platforms; filling missing values ​​in the normalized media data using a mean filling method or a median filling method; and deduplicating the filled media data using a hash function or a Bloom filter. In the actual implementation process, media data from different social media platforms have different data formats, and the data formats need to be converted to the same data format first. For example, the original image data may be in the data format of BMP, GIF, TIFF, RAW, SVG, WebP, PSD, ICO, JPEG, PNG, etc. Before data conversion, the data format of the image data needs to be normalized to JPEG format through open source image processing tools; the original text data may be in the format of TXT, CSV, JSON, XML, HTML, RTF, DOCX, PDF, etc., and the data format of the text data can be normalized to the same data format, such as JSON format, through open source text editing services; the original audio data may be in the data format of MP3, WAV, AAC, FLAC, OGG, AMR, etc., and before data conversion, it must be normalized to MP3 format through open source audio editing software; the original video data may be in the format of MP4, AVI, MOV, FLV, MKV, etc., and before data conversion, it must be normalized to MP4 format through open source video editing software. Furthermore, for the numerical data in the media data, it is necessary to fill in the missing values ​​and remove the duplicate values ​​to improve the quality of the media data and the accuracy of the subsequent model analysis.

[0084] In some feasible embodiments, the media data is converted into structured data according to a preset data conversion rule, including: when the media data is text data, the text data is stored as structured data according to a preset field structure. The structured data includes a number of fields, S = [s1, s2, s3, ...s i …,s n ],s i Represents the i-th field. Each field has a corresponding field name, corresponding data type and constraints.

[0085] In some feasible embodiments, the media data is converted into structured data according to a preset data conversion rule, including: when the media data is image data, the description information in the image data is identified by an image recognition algorithm, and the description information is stored as structured data according to a preset field structure. Specifically, the description information includes the URL, subtitles, metadata, etc. of the image data.

[0086] In some feasible embodiments, the media data is converted into structured data according to a preset data conversion rule, including: when the media data is audio data, a speech transcription result is obtained from the audio data through a speech-to-text service, and the speech transcription result is stored as structured data according to a preset field structure. In the actual implementation process, the storage of the speech transcription result is the same as the processing of text data.

[0087] In some feasible embodiments, media data is converted into structured data according to preset data conversion rules, including: when the media data is video data, the video data is split into audio data and image data; a speech transcription result is obtained from the audio data through a speech-to-text service; descriptive information in the image data is identified through an image recognition algorithm; and the speech transcription result and the descriptive information are stored as structured data according to a preset field structure.

[0088] In some feasible embodiments, the topic analysis model is obtained by training the LDA topic model, and the target data is input into the pre-trained topic analysis model for topic extraction to obtain a topic identification result, including: performing word segmentation processing on the target data to obtain a word segmentation list; using the TF-IDF algorithm to extract keywords from the word segmentation list to obtain multiple predicted keywords; performing data cleaning on multiple predicted keywords based on a preset stop word list to obtain multiple cleaned keywords; inputting the cleaned multiple keywords into the pre-trained LDA topic model to obtain the probability values ​​of multiple keywords belonging to the preset topic, sorting them from large to small according to the probability values, and selecting the topics corresponding to the preset number of keywords as the topic identification results. Specifically, in the actual implementation process, firstly, social media posts and news articles and other training data are collected from public channels, and the media data to be processed is subjected to word segmentation processing when the topic analysis model is trained; the word segmentation list after word segmentation is input into the TF-IDF algorithm to extract keywords to obtain multiple predicted keywords, and the preset stop word list is used to clean the multiple predicted keywords to obtain multiple cleaned keywords. The keywords and corresponding topic tags are input into the LDA topic model for training, and the parameter values ​​of the model are optimized by propagating errors through the back propagation algorithm to achieve the preset training cycle.

[0089] In some feasible embodiments, the sentiment analysis model is obtained by training the BERT module and the LSTM module, and the target data is input into the pre-trained sentiment analysis model for sentiment analysis to obtain the sentiment classification result, including: inputting the target data into the BERT module for text feature extraction to obtain the context vector representation of the target data; inputting the context vector representation into the LSTM module to extract features through convolution and pooling operations, and inputting the extracted text features into the linear layer to obtain the predicted sentiment classification result. According to the prediction results, the text can be divided into different sentiment categories, such as positive, negative or neutral. During the training process, public data is obtained, the data is divided into a training set and a test set, the model is trained using the training set data, the model parameters are updated by back propagation and gradient descent, and the cross entropy loss function and the Adam optimizer are used. During the training process, the model can be evaluated using a validation set to avoid overfitting.

[0090] The topic recognition results and sentiment classification results are input into the pre-trained sentiment dictionary for sentiment analysis to obtain the sentiment tendency results; the sentiment tendency results are analyzed to obtain the final data analysis results.

[0091] In some feasible embodiments, obtaining media data from multiple social media platforms through an interface adapter includes: obtaining media data from multiple social media platforms using multi-threading technology or asynchronous calling technology, which can improve the efficiency of obtaining social media data.

[0092] Another embodiment of the present application further provides a social media data processing device, which is used to execute the above-mentioned processing method, such as Figure 2 As shown, the processing device includes: a data acquisition module 101, a data conversion module 102, a target data extraction module 103, a first analysis module 104, a second analysis module 105, and a result summary module 106;

[0093] The data acquisition module 101 is used to acquire media data from multiple social media platforms through an interface adapter, and the data types of the media data include text data, image data, audio data and video data;

[0094] A data conversion module 102, configured to convert the media data into structured data according to a preset data conversion rule, and store the structured data in a database, wherein the structured data is data organized by the same field structure;

[0095] The target data extraction module 103 is used to receive the data analysis instruction and extract the target data to be processed from the database based on the screening conditions in the data analysis instruction;

[0096] The first analysis module 104 is used to extract themes from the target data to obtain theme recognition results;

[0097] The second analysis module 105 is used to perform sentiment analysis on the target data to obtain sentiment classification results;

[0098] The result aggregation module 106 is used to combine the topic identification result and the sentiment classification result to obtain the final data analysis result.

[0099] In some feasible embodiments, the data acquisition module 101 also includes an interface management unit, which is used to develop corresponding interface adapters based on the API documents of each social media platform; the interface adapters of all social media platforms are encapsulated in the interface management module for unified deployment; when the authentication requirements of the social media platform are updated, the interface management module adjusts the interface adapter according to the current authentication requirements of the social media platform and the current API documents.

[0100] In some feasible embodiments, the data conversion module 102 also includes: a data preprocessing unit, which is used to normalize the data format of media data from different social media platforms; use the mean filling method or the median filling method to fill the missing values ​​in the normalized media data; use a hash function or a Bloom filter to deduplicate the filled media data.

[0101] The specific definition of the social media data processing device provided in this embodiment can be found in the embodiment of the social media data processing device described above, and will not be repeated here. Each module in the above social media data processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0102] An embodiment of the present application provides a computer device, which may include a processor, a memory, a network interface, and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the processor executes the steps of the method for processing social media data as in any of the above embodiments.

[0103] The working process, working details and technical effects of the computer device provided in this embodiment can be found in the embodiment of the method for processing social media data above, and will not be described in detail here.

[0104] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the social media data processing method as described in any of the above embodiments are implemented. The computer-readable storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive, and / or a memory stick, etc. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0105] The working process, working details and technical effects of the computer-readable storage medium provided in this embodiment can be found in the embodiments of the method for processing social media data above, and will not be described in detail here.

[0106] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0107] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0108] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system described in the present application is divided into different functional units or modules to complete all or part of the functions described above.

Claims

1. A method for processing social media data, characterized in that: include: Acquire media data from multiple social media platforms through an interface adapter, wherein the data types of the media data include text data, image data, audio data, and video data; Converting the media data into structured data according to a preset data conversion rule, and storing the structured data in a database, wherein the structured data is data organized by the same field structure; Receive a data analysis instruction, and extract target data to be processed from a database based on screening conditions in the data analysis instruction; Inputting the target data into a pre-trained topic analysis model to extract topics and obtain topic recognition results; Input the target data into a pre-trained sentiment analysis model to perform sentiment analysis and obtain a sentiment classification result; The topic identification result and the sentiment classification result are combined to obtain the final data analysis result.

2. The method for processing social media data according to claim 1, characterized in that: Also includes: Encapsulate the interface adapters of all social media platforms in the interface management module for unified deployment, where the interface adapters are developed based on the API documents of the social media platforms; When the authentication requirements of the social media platform are updated, the interface management module adjusts the interface adapter according to the current authentication requirements of the social media platform and the current API documentation.

3. The method for processing social media data according to claim 1, characterized in that: Before converting the media data into structured data according to a preset data conversion rule, the method further includes: Normalize the data formats of media data from different social media platforms; Use mean filling method or median filling method to fill the missing values ​​in the normalized media data; A hash function or Bloom filter is used to deduplicate the padded media data.

4. The method for processing social media data according to claim 3, characterized in that: Converting the media data into structured data according to a preset data conversion rule includes: When the media data is text data, the text data is stored as structured data according to a preset field structure.

5. The method for processing social media data according to claim 3, characterized in that: Converting the media data into structured data according to a preset data conversion rule includes: When the media data is image data, the description information in the image data is identified by an image recognition algorithm, and the description information is stored as structured data according to a preset field structure.

6. The method for processing social media data according to claim 3, characterized in that: Converting the media data into structured data according to a preset data conversion rule includes: When the media data is audio data, a speech transcription result is obtained from the audio data through a speech-to-text service, and the speech transcription result is stored as structured data according to a preset field structure.

7. The method for processing social media data according to claim 3, characterized in that: Converting the media data into structured data according to a preset data conversion rule includes: When the media data is video data, splitting the video data into audio data and image data; Obtaining a speech transcription result from the audio data through a speech-to-text service; Identify the description information in the image data through the image recognition algorithm; The speech transcription result and the description information are stored as structured data according to a preset field structure.

8. The method for processing social media data according to claim 1, characterized in that: The topic analysis model is obtained by training the LDA topic model. The target data is input into the pre-trained topic analysis model to extract topics and obtain topic recognition results, including: Perform word segmentation on the target data to obtain a word segmentation list; Use the TF-IDF algorithm to extract keywords from the word list and obtain multiple predicted keywords; Performing data cleaning on multiple predicted keywords based on a preset stop word list to obtain multiple keywords after cleaning; The cleaned keywords are input into the pre-trained LDA topic model to obtain the probability values ​​of the keywords belonging to the preset topics. The keywords are sorted from large to small according to the probability values, and the topics corresponding to the preset number of keywords are selected as the topic recognition results.

9. The method for processing social media data according to claim 1, characterized in that: The sentiment analysis model is obtained by training the BERT module and the LSTM module. The target data is input into the pre-trained sentiment analysis model for sentiment analysis to obtain sentiment classification results, including: Input the target data into the BERT module to extract text features and obtain the context vector representation of the target data; The context vector representation is input into the LSTM module to extract features through convolution and pooling operations, and the extracted text features are input into the linear layer to obtain the predicted sentiment classification results.

10. The method for processing social media data according to any one of claims 1 to 9, characterized in that: Get media data from multiple social media platforms through interface adapters, including: Use multi-threading technology or asynchronous calling technology to obtain media data from multiple social media platforms.

11. A device for processing social media data, characterized in that: The processing device is used to execute the processing method according to any one of claims 1 to 10, comprising: A data acquisition module, used to acquire media data from multiple social media platforms through an interface adapter, wherein the data types of the media data include text data, image data, audio data and video data; A data conversion module, used to convert the media data into structured data according to a preset data conversion rule, and store the structured data in a database, wherein the structured data is data organized by the same field structure; A target data extraction module is used to receive a data analysis instruction and extract the target data to be processed from the database based on the screening conditions in the data analysis instruction; The first analysis module is used to extract themes from target data and obtain theme recognition results; The second analysis module is used to perform sentiment analysis on the target data to obtain sentiment classification results; The result summary module is used to combine the topic identification result and the sentiment classification result to obtain the final data analysis result.

12. The device for processing social media data according to claim 11, characterized in that: The data acquisition module also includes an interface management unit, which is used to encapsulate the interface adapters of all social media platforms in the interface management module for unified deployment, wherein the interface adapters are developed based on the API documents of the social media platforms; When the authentication requirements of the social media platform are updated, the interface management module adjusts the interface adapter according to the current authentication requirements of the social media platform and the current API documentation.

13. The device for processing social media data according to claim 12, characterized in that: The data conversion module also includes: The data preprocessing unit is used to normalize the data formats of media data from different social media platforms; fill in missing values ​​in the normalized media data using the mean filling method or the median filling method; and deduplicate the filled media data using a hash function or a Bloom filter.

14. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 10.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images