An Algorithm for Analyzing Consumer Information Data

Abstract

This invention provides a consumer information data analysis algorithm, comprising: S1, dividing the collected consumer information data into text data and video data according to data modality, and performing preliminary screening on the text data and video data using a preset keyword matching algorithm to obtain preliminary consumer information data; S2, parsing the text data and video data in the preliminary consumer information data using text content parsing methods and video content parsing methods to obtain parsed consumer information data; S3, performing statistical analysis on the obtained parsed consumer information data according to static and dynamic indicators. The beneficial effects of this invention are: a consumer information data analysis algorithm, designed based on the actual business direction of consumer-related matters, incorporates five categories of consumer-related keyword matching preliminary screening schemes, and allows for preliminary screening of video data through extraction of short video titles and description text, quickly narrowing down the scope of consumer information data and significantly improving algorithm efficiency.

Description

Technical Field

[0001] The invention belongs to the field of information technology, and in particular relates to a consumption-related information data analysis algorithm. Background Art

[0002] The scope of fire incidents has been broadened to include not only fire fighting, emergency rescue, social assistance, fire science popularization, fire prevention supervision and other business contents. Under the current fire data system, the traditional fire accident consequence evaluation indicators start from the accident itself and can reflect the severity of a fire accident;

[0003] In the prior art, there are the following problems:

[0004] Due to the limitations of technology and computing resources, most current information monitoring and analysis systems use keyword or tag matching technology to associate information. The advantages of this method are fast matching speed and strong versatility, but it is prone to semantic ambiguity and a lot of irrelevant content, which brings unnecessary interference to information monitoring. In addition, for the currently popular short video data, existing information monitoring products have problems such as incomplete data crawling and poor matching capabilities. Currently, most of the publicly available technical methods are to achieve information association through technologies such as video title keyword matching and video cover text extraction. This method has poor accuracy and is prone to a large amount of irrelevant data, affecting the application effect of information monitoring and analysis. Summary of the invention

[0005] In view of this, the present invention aims to propose a consumption information data analysis algorithm to at least solve one of the problems in the background technology.

[0006] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0007] A consumption information data analysis algorithm, comprising:

[0008] S1. The collected consumer information data is divided into text data and video data according to the data modality, and the text data and the video data are preliminarily screened using a preset keyword matching algorithm to obtain preliminarily screened consumer information data;

[0009] S2. Analyze the text data and video data in the preliminary screening data of the consumer information using a text content analysis method and a video content analysis method to obtain consumer information analysis data;

[0010] S3. Conduct statistical analysis on the obtained consumer information analysis data according to static and dynamic indicators.

[0011] Furthermore, in the step S1, the collected consumer information data is divided into text data and video data according to the data modality. First, the video data is matched with keywords by extracting the text content of the short video title and introduction. Then, after the text data and video data are matched with keywords respectively, the successfully matched data is used as initial data for preliminary screening using the keyword matching algorithm.

[0012] Furthermore, in step S1, according to the firefighting business category, a keyword matching algorithm based on five major firefighting-related topics, namely, fire accidents, official duties, emergency rescue, social assistance and natural disasters, is used for preliminary data screening;

[0013] The keyword matching algorithm includes keyword determination relationships. When the relationship between keywords meets the conditions, the corresponding text data or video data is identified as consumer-related content, and the consumer-related content is aggregated to construct preliminary screening data for consumer-related information.

[0014] Furthermore, in step S2, the text content parsing method includes a text summarization algorithm, which quickly analyzes the newly released news, obtains the hot topics on the current network, and matches them with the relevant consumer-related content, so as to timely discover the newly released consumer-related information, specifically including:

[0015] 1) First, integrate the consumer news reports into text format data;

[0016] 2) Split text-formatted data into individual sentences;

[0017] 3) Find word vector representation for each sentence;

[0018] 4) Construct a similarity matrix and calculate the similarity between sentence vectors;

[0019] 5) Convert the similarity matrix into a graph structure with sentences as nodes and similarity scores as edges, and use the TextRank algorithm for calculation;

[0020] 6) The sentence with the highest score is taken as the text summary of this text data.

[0021] Furthermore, in step S2, the text content parsing method includes an event association algorithm:

[0022] Firstly, the text content of the consumption events is preprocessed, data is cleaned and formatted, the hot words of the events are extracted using the TextRank algorithm, and the co-occurrence frequency of multiple events is calculated;

[0023] Extract interactive relationships from interactive user information of events, and use Apriori algorithm to find item set relationships using layer-by-layer search iteration method;

[0024] With reference to the two-dimensional result indicators, the degree of correlation is finally comprehensively analyzed and the correlation score is output.

[0025] Furthermore, in step S2, the video content analysis method includes:

[0026] For video data, video frame image data is extracted every n seconds, and the key elements appearing in the video frame image data are identified through the consumer-related target recognition algorithm. When the key elements appear in the image, the video is judged to be consumer-related content, and the key elements are used as video tags.

[0027] Furthermore, the key elements include firefighters, vehicles, flames, and thick smoke. In order to solve the problem that firefighters, vehicles, flames, and thick smoke in video frame image data are easily confused due to their consistent colors, the HSV color space is first used to quickly locate the red objects in the image, and color recognition is performed using the HSV color space parameters in OpenCV. The pixels other than the H value of (0,8)∪(160,180) are set to black, and the boundary pixel positions of the connected area are selected and framed in the original image as the target recognition area.

[0028] Furthermore, in step S3, static indicators and dynamic indicators are constructed according to event category, release time, region, user behavior, and dissemination behavior. The static indicators include event category, content type, and region.

[0029] The dynamic indicators include user behavior indicators and communication indicators.

[0030] Furthermore, the present solution discloses an electronic device, including a processor and a memory that is communicatively connected to the processor and is used to store executable instructions of the processor, wherein the processor is used to execute a consumption information data analysis algorithm.

[0031] Furthermore, the present solution discloses a computer-readable storage medium storing a computer program, which implements a consumption information data analysis algorithm when executed by a processor.

[0032] Compared with the prior art, the consumption information data analysis algorithm described in the present invention has the following beneficial effects:

[0033] (1) The consumer information data analysis algorithm described in the present invention designs five categories of consumer-related keyword matching preliminary screening schemes according to the actual business direction of consumer-related information, and also enables the preliminary screening of video data by extracting short video titles and introduction texts, thereby quickly narrowing the scope of consumer-related information data and greatly improving the efficiency of the algorithm;

[0034] (2) The consumer information data analysis algorithm described in the present invention designs text and video content analysis algorithms according to the information data category, thus broadening the application scope of the consumer information analysis algorithm. In-depth content analysis helps to further filter out irrelevant information and improve the accuracy of the results.

[0035] (3) The consumer information data analysis algorithm described in the present invention constructs a consumer information short video training data set, uses this data set to perform transfer learning on the target recognition algorithm, improves the accuracy of the algorithm in the field of consumer short video information analysis, and solves the problem of poor application effect and low accuracy of information monitoring in the consumer field;

[0036] (4) The present invention discloses a consumer information data analysis algorithm and designs a consumer information influence statistical indicator system, which helps to collect and count useful information and can provide data support for tasks such as tracking the popularity of consumer events, information data statistics, and information management and control. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0038] Figure 1 A schematic diagram of a technical roadmap for monitoring and analyzing consumer information according to an embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram of the information data initial screening flow chart (first step) according to an embodiment of the present invention;

[0040] Figure 3 It is a schematic diagram of a multimodal consumption information data analysis flow chart (second step, third step) according to an embodiment of the present invention;

[0041] Figure 4 A schematic diagram of a flow chart of a text summarization algorithm according to an embodiment of the present invention;

[0042] Figure 5 A schematic diagram of a consumption-related information event correlation algorithm according to an embodiment of the present invention;

[0043] Figure 6 This is a schematic diagram of a model training flow chart according to an embodiment of the present invention;

[0044] Figure 7 This is a schematic diagram of the consumer information monitoring and analysis indicator system architecture described in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0046] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0047] In the field of information analysis, we focus on the theme of fire rescue, carry out data analysis work in a targeted manner, and propose a complete "monitoring-analysis-statistics" process and indicator system for fire-related information. The technical route flow chart is as follows: Figure 1 shown.

[0048] Step 1: To optimize information processing efficiency and improve recognition accuracy and speed, we first use the keyword matching algorithm to screen the target platform information data. The process is as follows: Figure 2 As shown:

[0049] For various media platforms such as social media, portals, and short video platforms, the information data is first divided into two categories according to the data modality: text and short video. Considering that the processing of short video data is labor-intensive and time-consuming, this study first screens the data by extracting text content such as short video titles and introductions for keyword matching. After the text and short video data are matched by keywords, the successfully matched data are saved as the consumer information screening data set, which serves as the initial data for the second step of the deep analysis algorithm.

[0050] According to the actual firefighting business categories, a keyword matching algorithm based on five major firefighting-related topics, namely fire accidents, official duties, emergency rescue, social assistance and natural disasters, was designed for preliminary data screening, for example:

[0051] The target keywords for the fire accident topic include: (ignition|fire|on fire|flame|flame) + (rescue|help|transfer|reinforcement) etc.

[0052] The target keywords for the official duty topic include: (festivals | events | celebrations | performances | meetings) + (security | patrols | stationing);

[0053] The target keywords for emergency rescue topics include: (fire|flood|earthquake|mudslide|blizzard) + (rescue|help|transfer);

[0054] The target keywords for the social assistance topic include: (search and rescue|treatment|rescue|material support|high-rise building) + (rescue|case|rescue);

[0055] The target keywords for the natural disaster topic include: (fire|fire condition|flood|land collapse|earthquake) + (protection|disaster|disaster relief);

[0056] In the designed keyword contents of each topic, “|” represents the relationship of “or”, and “+” represents the relationship of “and”. For example, “community+fire” means that when the text contains both the two groups of keywords “community” and “fire”, the text can be identified as fire-related content; “fire|fire” means that when the text contains any one of the keywords “fire” or “fire”, the text can be identified as fire-related content, constituting the initial screening data of fire-related information.

[0057] Step 2: After obtaining the preliminary screening data of consumer information, in order to more accurately analyze the data content and further determine whether it is consumer information, the data is divided into two categories according to the data modality: text and video. Data analysis is performed for the characteristics of these two types of data respectively. The flow chart is as follows Figure 3 The key technical contents are mainly divided into the following two parts:

[0058] 1. Consumer information text content analysis technology

[0059] (1) Text Summarization Algorithm

[0060] By quickly analyzing the newly released news, we can obtain the hottest topics on the entire network and match them with the relevant consumer topics, so as to timely discover the newly released consumer information and achieve the purpose of monitoring the entire network. The input of the text summarization task is a long text document. The goal of the task is to convert the long text into a short, fluent and accurate text summary. The algorithm construction idea is as follows Figure 4 As shown:

[0061] 1) First, integrate consumer news reports into text data;

[0062] 2) Segment the text into individual sentences;

[0063] 3) Find word vector representation for each sentence;

[0064] 4) Construct a similarity matrix and calculate the similarity between sentence vectors;

[0065] 5) Convert the similarity matrix into a graph structure with sentences as nodes and similarity scores as edges for sentence TextRank calculation;

[0066] 6) Finally, TextRank calculates the sentence with the highest score as the text summary of this article.

[0067] Note: TextRank calculation method is public technology

[0068] (2) Event association algorithm

[0069] The event association algorithm is used to realize the clustering function of consumption information based on specific events:

[0070] First, the text content involved in the consumption events is preprocessed, data cleaned and format converted to ensure the accuracy and completeness of the data. The TextRank algorithm is used to extract event hot words and calculate the co-occurrence frequency of multiple events.

[0071] The interactive relationship is extracted from the interactive user information of the event, and the Apriori algorithm is used to find the item set relationship using a layer-by-layer search iterative method.

[0072] Refer to the two-dimensional result indicators, and finally comprehensively analyze the degree of correlation and output the correlation score. Figure 5 One dimension is to extract hot words of events, and the other dimension is to extract interactive information of events. Analyze these two dimensions separately and finally get the event correlation degree score.

[0073] 2. Short video content analysis technology

[0074] Since short video analysis occupies a large amount of computing resources and takes a long time, in order to further improve computing efficiency and meet the real-time requirements of information monitoring, a collection scheme is designed to extract video frame image data every 5 seconds. After obtaining the video frame image data, the fire-related target recognition algorithm is used to identify key elements such as firefighters, vehicles, flames, and thick smoke that appear in the image. When key elements appear in the image, the video is determined to be fire-related content, and the key elements are used as video tags.

[0075] Taking into account the real-time requirements of the algorithm for information monitoring tasks, and since firefighters, fire trucks, flames, fire hydrants and other fire-related targets are all red, in order to more quickly identify fire-related targets in video image frames, it is proposed to first quickly locate the red objects in the image through HSV color space processing, and perform color recognition through the HSV color space parameters in OpenCV. The pixels outside the H value of (0,8)∪(160,180) are set to black, and the boundary pixel positions of the connected area are selected, and a box is selected in the original image as the target recognition area.

[0076] The target recognition algorithm is an improved YOLOv5 (public technology, no further description), and the model training process is as follows: Figure 6 As shown in the figure, the COCO dataset is used for pre-training of the YOLOv5 algorithm model, and then the weight file generated by the pre-training is migrated to the new YOLOv5 model network as the initial value for the training of the intelligent target detection algorithm model for consumer short videos. Then, the model is trained for multiple rounds using the manually labeled consumer short video training dataset to obtain the final model weights.

[0077] Step 3: After conducting intelligent analysis on text and video fire-related information data through the above algorithms, a fire-related information data analysis indicator system was designed in combination with the actual business needs of fire protection, such as Figure 7 According to the designed indicator system, the event category, release time, region, user behavior, dissemination behavior and other indicators are collected and counted, which can realize the quantitative statistics of the information influence of specific consumption-related events.

[0078] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0079] In the several embodiments provided in the present application, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the division of the units described above is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The above-mentioned units may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A consumption information data analysis algorithm, characterized in that: include: S1. The collected consumer information data is divided into text data and video data according to the data modality, and the text data and the video data are preliminarily screened using a preset keyword matching algorithm to obtain preliminarily screened consumer information data; S2. Analyze the text data and video data in the preliminary screening data of the consumer information using a text content analysis method and a video content analysis method to obtain consumer information analysis data; S3. Statistically analyze the obtained consumer information analysis data according to static indicators and dynamic indicators; In the step S1, the collected consumer information data is divided into text data and video data according to the data modality. First, the video data is subjected to keyword matching by extracting the text content of the short video title and introduction. Then, after the text data and the video data are matched by keywords respectively, the successfully matched data is used as the initial data for preliminary screening using the keyword matching algorithm. In step S1, according to the firefighting business categories, a keyword matching algorithm based on five major firefighting-related topics, namely, fire accidents, official duties, emergency rescue, social assistance and natural disasters, is used to perform preliminary data screening; The keyword matching algorithm includes keyword determination relationships. When the relationship between keywords meets the conditions, the corresponding text data or video data is identified as consumer-related content, and the consumer-related content is aggregated to construct the consumer-related information preliminary screening data; In step S2, the text content analysis method includes a text summarization algorithm, which analyzes the newly released news to obtain the hot topics on the current Internet and matches them with the consumption-related special content, specifically including: 1) First, integrate the consumer news reports into text format data; 2) Split text-formatted data into individual sentences; 3) Find word vector representation for each sentence; 4) Construct a similarity matrix and calculate the similarity between sentence vectors; 5) Convert the similarity matrix into a graph structure with sentences as nodes and similarity scores as edges, and use the TextRank algorithm for calculation; 6) The sentence with the highest score is used as the text summary of this text data; In step S2, the text content parsing method includes an event association algorithm: Firstly, the text content of the consumption events is preprocessed, data is cleaned and formatted, the hot words of the events are extracted using the TextRank algorithm, and the co-occurrence frequency of multiple events is calculated; Extract interactive relationships from interactive user information of events, and use Apriori algorithm to find item set relationships using layer-by-layer search iteration method; Refer to the co-occurrence frequency and item set relationship, and finally comprehensively analyze the degree of association and output the association score; In step S2, the video content analysis method includes: For video data, video frame image data is extracted every n seconds, and the key elements appearing in the video frame image data are identified through the consumption-related target recognition algorithm. When the key elements appear in the image, the video is determined to be consumption-related content, and the key elements are used as video tags; The key elements include firefighters, vehicles, flames, and thick smoke. To address the problem that firefighters, vehicles, flames, and thick smoke in video frame image data are easily confused due to their consistent colors, the HSV color space is first used to quickly locate the red objects in the image, and color recognition is performed using the HSV color space parameters in OpenCV. The pixels other than those with H values ​​of (0,8)∪(160,180) are set to black, and the boundary pixel positions of the connected area are selected and framed in the original image as the target recognition area.

2. The consumption information data analysis algorithm according to claim 1 is characterized in that: In step S3, static indicators and dynamic indicators are constructed according to event category, release time, region, user behavior, and dissemination behavior. The static indicators include event category, content type, and region. The dynamic indicators include user behavior indicators and communication indicators.

3. An electronic device, comprising a processor and a memory connected to the processor for storing instructions executable by the processor, characterized in that: The processor is used to execute a consumption information data analysis algorithm as described in any one of claims 1-2.

4. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements a consumption information data analysis algorithm as described in any one of claims 1-2.

Images