Unstructured data management method based on big data and AI technology

Abstract

The invention discloses an unstructured data governance method based on big data and an AI technology, and belongs to the technical field of data governance, and the method comprises the steps: firstly, through training and deploying an AI emotion model, scanning a data source, obtaining target data after unstructured data is recognized, and carrying out data analysis; according to the method, enterprise development strategies are guided through emotional tendencies and in combination with superior and disadvantage analysis results of products, new user evaluation data are continuously collected, the analysis process is repeated, the strategies are continuously adjusted, maintained and improved according to new analysis results, and target user portraits are constructed, so that the user experience is improved. More accurate market strategies, product strategies and marketing channels are formulated for target user groups, the product attraction is improved to attract potential users, meanwhile, the stickiness of the target users is enhanced, finally, through sentiment analysis on channels such as social media, enterprises are helped to discover potential public crisis or brand risks in time, response measures are taken in time, and the enterprise experience is improved. And valuable commercial insights are provided for enterprises.

Description

Technical Field

[0001] The present invention relates to the field of data governance technology, and more specifically, to an unstructured data governance method based on big data and AI technology. Background Art

[0002] Unstructured data refers to data that does not have a fixed or predefined data model. It is not expressed in a two-dimensional table structure logic like structured data, but is irregular or incomplete, and is not convenient to be expressed in a two-dimensional database logical table. Unstructured data usually includes text files, pictures, audio, video, social media posts, emails, log files, XML, JSON, etc. The mainstream application environment of unstructured data governance covers advertising and marketing, retail, education, finance, manufacturing and other fields. In these fields, data governance can help organizations achieve standardized data management, improve data quality, protect data security, optimize decision-making processes, and promote business innovation.

[0003] For companies that have settled in e-commerce platforms and invested in online marketing, consumers will generate a large amount of unstructured text data, such as comments, posts, feedback, etc. in channels such as social media, online forums, and e-commerce platforms. These text data contain users' emotional attitudes towards a certain product, service, or event. These emotional attitudes are very likely to provide shopping guidance for potential users who see the above comments and posts. The emotional tendencies in these unstructured data can be mined, which can provide companies with valuable business insight management.

[0004] Therefore, in response to the above technical problems, it is necessary to provide an unstructured data governance method based on big data and AI technology. Summary of the invention

[0005] The purpose of the present invention is to provide an unstructured data governance method based on big data and AI technology to solve the above-mentioned problems.

[0006] In order to achieve the above purpose, the technical solution provided by an embodiment of the present invention is as follows:

[0007] An unstructured data governance method based on big data and AI technology includes the following steps:

[0008] S1. Training and deploying AI emotion models: Feeding large-scale data sets with sufficient samples covering various emotion combinations and nuances for model training, extracting emotion feature results from samples, and enabling them to perform emotion classification prediction in actual applications after deployment;

[0009] S2. Obtain target data: Use text mining and natural language processing technology to scan various preset data sources and identify unstructured data;

[0010] S3. Data cleaning and standardization: Use text processing and data cleaning techniques to clean unstructured data, remove noise data, convert unstructured data into structured or semi-structured data, and obtain target data;

[0011] S4. Data classification and analysis: Perform text preprocessing steps such as text segmentation, part-of-speech tagging, and stop word removal on the target data, and then use the sentiment analysis algorithm provided by the AI ​​training model to identify the sentiment tendency in the text;

[0012] S5. Data visualization: The data analysis results are displayed in a visual manner, and the sentiment analysis results are converted into visual charts including bar charts, heat maps, clustering charts, etc.

[0013] As a further improvement of the present invention, the AI ​​training emotion model in step S1 optimizes the model parameters, such as learning rate, batch size, number of iterations, etc., by iteratively updating the model parameters while using cross-validation to enable the model to continuously learn to extract more accurate emotion features from text data.

[0014] As a further improvement of the present invention, the data source in step S2 includes user evaluations and product feedback on e-commerce platforms (such as Pinduoduo, Taobao, JD.com, Amazon and other platforms), user comments and product-related posts on social media platforms (such as Weibo, Twitter), and product discussions and questions and answers on online forums (such as Zhihu, Douban).

[0015] As a further improvement of the present invention, the unstructured data in step S2 is data such as documents, pictures, video links and video files extracted from user reviews on e-commerce platforms, social media platforms and online forums.

[0016] As a further improvement of the present invention, the noise data in step S3 includes advertisements, repeated information or other junk data irrelevant to the product and the evaluation of the product.

[0017] As a further improvement of the present invention, the target data in step S3 also includes metadata extraction, specifically collecting user evaluations of products and product after-sales services from various publishing channels and user publishing time and other information, and adding the direction of time trends.

[0018] As a further improvement of the present invention, the sentiment tendency in step S4 includes three types: positive, negative and neutral. When performing sentiment analysis on the target data, the sentiment dictionary method can also be combined to judge the sentiment tendency by matching the sentiment words in the text.

[0019] As a further improvement of the present invention, the step S4 also includes S41, advantages and disadvantages analysis and feedback loop and S42, building a target user portrait.

[0020] As a further improvement of the present invention, the step S41, advantages and disadvantages analysis and feedback loop is specifically based on sentiment analysis, further using topic modeling technology (such as LDA, NMF, etc.) to identify different topics or aspects mentioned in the evaluation, analyzing the topic content reflecting the advantages of the product or user satisfaction in the positive evaluation, analyzing the topic content reflecting the shortcomings of the product or user dissatisfaction in the negative evaluation, and extracting and quantifying these advantages and disadvantages through word frequency statistics and association rule mining, and finally comparing the neutral evaluation with the positive evaluation and the negative evaluation, observing the similarities and differences between them, identifying the positive or negative factors that may be implied in the neutral evaluation, for the advantages, ensure that these advantages are maintained in product updates and iterations, and consider how to further improve the user experience, for the disadvantages, formulate a specific improvement plan, and give priority to solving the problems mentioned by users' high-frequency words, in this process continuously collect new user evaluation data, and repeat the above analysis process to monitor whether the product advantages are maintained and whether the disadvantages are improved, and continuously adjust the maintenance and improvement strategies according to the new analysis results.

[0021] As a further improvement of the present invention, the step S42, constructing a target user portrait, specifically comprises conducting a comprehensive analysis of the region, age, gender, occupation, income, education level, etc. of users with positive emotional tendencies, obtaining a comprehensive model of user characteristics, behaviors, preferences and other information, constructing a character portrait of the target user, and formulating more accurate market strategies, product strategies and marketing channels for this target user group.

[0022] Compared with the prior art, the advantages of the present invention are:

[0023] This solution first trains and deploys an AI sentiment model, then scans the data source and performs data cleaning and standardization after identifying unstructured data to obtain the target data. Then, it performs sentiment tendency data analysis on the target data. The sentiment tendency analysis results are combined with the product's strengths and weaknesses analysis results to guide the company's development strategy, maintain the product's strengths and improve the weaknesses, and continuously collect new user evaluation data. Repeat the above analysis process, continuously adjust the retention and improvement strategies based on the new analysis results, and formulate more accurate market strategies, product strategies and marketing channels for the target user groups by building target user portraits. Finally, through sentiment analysis of channels such as social media and online forums, it helps companies to promptly discover potential public relations crises or brand risks and take corresponding countermeasures in a timely manner, which can effectively enhance the attractiveness of products to attract potential users and enhance the stickiness of target users at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flow chart of the unstructured data management method based on big data and AI technology of the present invention. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention; it is obvious that the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments, and all other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making creative work are within the scope of protection of the present invention.

[0026] Embodiment 1:

[0027] Please refer to the figure, an unstructured data governance method based on big data and AI technology, including the following steps:

[0028] S1. Training and deploying AI emotion models: Feeding large-scale data sets with sufficient samples covering various emotion combinations and nuances for model training, extracting emotion feature results from the samples, and making emotion classification predictions in actual applications after deployment. AI emotion model training iteratively updates model parameters while using cross-validation to optimize model parameters, such as learning rate, batch size, number of iterations, etc., so that the model can continuously learn to extract more accurate emotion features from text data;

[0029] S2. Obtain target data: Use text mining and natural language processing technology to scan various preset data sources, including user reviews and product feedback on e-commerce platforms (such as Pinduoduo, Taobao, JD.com, Amazon, etc.), user comments and product-related posts on social media platforms (such as Weibo, Twitter), and product discussions and Q&A on online forums (such as Zhihu, Douban), and identify unstructured data. Unstructured data refers to documents, pictures, video links, video files, etc. extracted from user reviews on e-commerce platforms, social media platforms, and online forums;

[0030] S3. Data cleaning and standardization: Use text processing and data cleaning technology to clean unstructured data and remove noise data. Noise data includes advertisements, repeated information or other junk data that is irrelevant to the product and product evaluation. Convert unstructured data into structured or semi-structured data to obtain target data. The target data includes metadata extraction. Specifically, collect the evaluation of users on the product and product after-sales service from each publishing channel, as well as the user publishing time and other information, and add the direction of time trend;

[0031] S4. Data classification and analysis: Perform text preprocessing steps such as word segmentation, part-of-speech tagging, and stop word removal on the target data, and then use the sentiment analysis algorithm provided by the AI ​​training model to identify the sentiment tendency in the text. The sentiment tendency includes positive, negative, and neutral. When performing sentiment analysis on the target data, the sentiment dictionary method can also be combined to judge the sentiment tendency by matching the sentiment words in the text;

[0032] S41. Strengths and weaknesses analysis and feedback loop: Based on sentiment analysis, further use topic modeling techniques (such as LDA, NMF, etc.) to identify different topics or aspects mentioned in the reviews, analyze the topics that reflect the advantages of the product or user satisfaction in positive reviews, analyze the topics that reflect the shortcomings of the product or user dissatisfaction in negative reviews, and extract and quantify these strengths and weaknesses through word frequency statistics and association rule mining. Finally, compare neutral reviews with positive and negative reviews, observe the similarities and differences between them, and identify the positive or negative factors that may be implied in neutral reviews. For strengths, ensure that these strengths are maintained in product updates and iterations, and consider how to further improve the user experience. For weaknesses, formulate specific improvement plans and give priority to solving problems mentioned by users' high-frequency words. In this process, continuously collect new user evaluation data and repeat the above analysis process to monitor whether the product advantages are maintained and whether the disadvantages are improved, and continuously adjust the maintenance and improvement strategies based on the new analysis results.

[0033] S42. Construct target user portraits: For users with positive emotional tendencies, a comprehensive analysis is conducted based on their region, age, gender, occupation, income, education level, etc. Based on the analysis of user geographical distribution, we can try to infer through indirect information such as the content posted by users or IP addresses, to understand the preferences and feedback of users in different regions on products. For user age and gender tendency analysis, we can try to infer through indirect characteristics such as the language style, word usage habits, and type of content posted by users. Income and education level are inferred based on the basis of purchase decisions and expectations for products, so as to obtain a comprehensive model of user characteristics, behaviors, preferences and other information, construct a character portrait of the target user, and formulate more precise market strategies, product strategies and marketing channels for this target user group.

[0034] S5. Data visualization: Display data analysis results in a visual way, convert sentiment analysis results into visual charts including bar charts, heat maps and cluster charts, and convey analysis results and suggestions to product teams and management.

[0035] This solution first trains and deploys an AI sentiment model, then scans the data source and performs data cleaning and standardization after identifying unstructured data to obtain the target data. Then, it performs sentiment tendency data analysis on the target data. The sentiment tendency analysis results are combined with the product's strengths and weaknesses analysis results to guide the company's development strategy, maintain the product's strengths and improve the weaknesses, and continuously collect new user evaluation data. The above analysis process is repeated, and the retention and improvement strategies are continuously adjusted according to the new analysis results. By building a target user portrait, more accurate market strategies, product strategies and marketing channels are formulated for the target user group. Finally, through sentiment analysis of channels such as social media and online forums, it helps companies to promptly discover potential public relations crises or brand risks and take corresponding countermeasures in a timely manner. This can effectively enhance the attractiveness of products to attract potential users and enhance the stickiness of target users at the same time, providing companies with valuable business insights.

[0036] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0037] In addition, it should be understood that although the present specification is described according to embodiments, not every embodiment contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that those skilled in the art can understand.

Claims

1. A method for managing unstructured data based on big data and AI technology, characterized by: The following steps are involved: S1. Training and deploying AI emotion models: Feeding large-scale data sets with sufficient samples covering various emotion combinations and nuances for model training, extracting emotion feature results from samples, and enabling them to perform emotion classification prediction in actual applications after deployment; S2. Obtain target data: Use text mining and natural language processing technology to scan various preset data sources and identify unstructured data; S3. Data cleaning and standardization: Use text processing and data cleaning techniques to clean unstructured data, remove noise data, convert unstructured data into structured or semi-structured data, and obtain target data; S4. Data classification and analysis: Perform text preprocessing steps such as text segmentation, part-of-speech tagging, and stop word removal on the target data, and then use the sentiment analysis algorithm provided by the AI ​​training model to identify the sentiment tendency in the text; S5. Data visualization: The data analysis results are displayed in a visual manner, and the sentiment analysis results are converted into visual charts including bar charts, heat maps, clustering charts, etc.

2. The unstructured data management method based on big data and AI technology according to claim 1 is characterized by: In step S1, the AI ​​training emotion model iteratively updates the model parameters while using cross-validation to optimize the model parameters, such as learning rate, batch size, number of iterations, etc., so that the model can continuously learn to extract more accurate emotion features from text data.

3. The unstructured data management method based on big data and AI technology according to claim 1 is characterized by: The data sources in step S2 include user reviews and product feedback on e-commerce platforms (such as Pinduoduo, Taobao, JD.com, Amazon, etc.), user comments and product-related posts on social media platforms (such as Weibo, Twitter), and product discussions and questions and answers on online forums (such as Zhihu, Douban).

4. The unstructured data management method based on big data and AI technology according to claim 3 is characterized by: The unstructured data in step S2 are data such as documents, pictures, video links, and video files extracted from user reviews on e-commerce platforms, social media platforms, and online forums.

5. The unstructured data management method based on big data and AI technology according to claim 1 is characterized by: The noise data in step S3 includes advertisements, repeated information or other junk data irrelevant to the product and the evaluation of the product.

6. The unstructured data governance method based on big data and AI technology according to claim 3 is characterized by: The target data in step S3 also includes metadata extraction, specifically collecting information such as users' evaluations of products and product after-sales services from various publishing channels and user publishing time, and adding the direction of time trends.

7. The unstructured data management method based on big data and AI technology according to claim 1 is characterized by: The sentiment tendency in step S4 includes three types: positive, negative and neutral. When performing sentiment analysis on the target data, the sentiment dictionary method can also be combined to judge the sentiment tendency by matching the sentiment words in the text.

8. The unstructured data management method based on big data and AI technology according to claim 1 is characterized by: The step S4 also includes S41, advantages and disadvantages analysis and feedback loop and S42, building a target user portrait.

9. The unstructured data management method based on big data and AI technology according to claim 8 is characterized by: The step S41, advantages and disadvantages analysis and feedback loop is specifically to further use topic modeling technology (such as LDA, NMF, etc.) to identify different topics or aspects mentioned in the evaluation on the basis of sentiment analysis, analyze the themes reflected in the positive evaluations that reflect the advantages of the product or user satisfaction, analyze the themes reflected in the negative evaluations that reflect the shortcomings of the product or user dissatisfaction, and extract and quantify these advantages and disadvantages through word frequency statistics and association rule mining, and finally compare the neutral evaluations with the positive evaluations and the negative evaluations to observe the similarities and differences between them, identify the positive or negative factors that may be implied in the neutral evaluations, and for the advantages, ensure that these advantages are maintained in product updates and iterations, and consider how to further enhance the user experience. For the disadvantages, formulate specific improvement plans and give priority to solving the problems mentioned by users' high-frequency words. In this process, continuously collect new user evaluation data and repeat the above analysis process to monitor whether the product advantages are maintained and whether the disadvantages are improved, and continuously adjust the maintenance and improvement strategies according to the new analysis results.

10. The unstructured data governance method based on big data and AI technology according to claim 8, characterized in that: The step S42, constructing the target user portrait, specifically comprises conducting a comprehensive analysis of the region, age, gender, occupation, income, education level, etc. of users with positive emotional tendencies, obtaining a comprehensive model of user characteristics, behaviors, preferences and other information, constructing a character portrait of the target user, and formulating more precise market strategies, product strategies and marketing channels for this target user group.

Images