Information processing device, information processing program, and information processing method

Abstract

To solve the problem that no effective mechanism for nicely feeding back and collecting hidden gas-related needs of consumers has been built because of the nature of services provided in the gas industry.SOLUTION: An information processing device according to a technique disclosed herein comprises a scraping unit (110) for automatically extracting information necessary for users from big data, a morpheme analysis unit (120) configured to divide sentences into morphemes and perform language processing, and a clustering unit (130) for clustering posted articles by topic.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing program, and an information processing method. [Background technology]

[0002] Since gas lamps first came on in Yokohama in 1872, gas has rapidly spread to various areas of Japan. In particular, since 1970, the consumption of electricity and city gas has increased rapidly, making them an important infrastructure in people's lives.

[0003] As the number of gas service providers and usage scenarios increases, it is natural that complaints and opinions will arise from users. For example, according to a report by the National Consumer Affairs Center of Japan, complaints are on the rise, and most of the complaints are related to contracts with gas service providers. If we consider that the complaints reported to the National Consumer Affairs Center of Japan are just the tip of the iceberg, there may be many unreported negative opinions, as well as positive opinions about beneficial gas usage. In the technical field of gas services, there is a need to easily and quickly collect and visualize unexpressed consumer needs regarding gas.

[0004] Meanwhile, there is known a mechanism (for example, a posting server that provides a posting service) that allows users to post reviews of products they have purchased on e-commerce sites that conduct electronic commerce. For example, Patent Document 1 discloses an information processing device that includes an estimation unit that estimates a tendency regarding the life cycle of a product purchased by a user based on the stage of the life cycle of each product purchased by the user and the usage pattern of each product, and a provision unit that provides the user with information according to the estimated tendency. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2024-10552 Summary of the Invention [Problem to be solved by the invention]

[0006] On the other hand, in the gas industry, due to the nature of the services provided, there is currently no effective system in place to effectively provide feedback and collect consumer needs regarding gas, which are not yet explicitly stated.

[0007] The disclosed technology aims to build a system in the technical field of the gas business that can easily and quickly collect and visualize consumer needs regarding gas that are not yet expressed. [Means for solving the problem]

[0008] The information processing device according to the disclosed technology includes a scraping unit that automatically extracts information necessary for users from big data, a morphological analysis unit that divides sentences into morphemes and performs language processing, and a clustering unit that clusters posted articles by topic. [Effects of the Invention]

[0009] The information processing device according to the disclosed technique has the above-described configuration, and therefore can easily and quickly collect and visualize unexpressed consumer needs regarding gas. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a functional block diagram showing the functional configuration of an information processing device 100 according to the first embodiment. [Figure 2] FIG. 2 is a flowchart showing a processing flow of the information processing device 100 according to the first embodiment. [Figure 3] FIG. 3 is a flowchart showing detailed processing steps of the information processing device 100 according to the first embodiment. [Figure 4] FIG. 4 is a diagram (part 1) showing an image of an output result of information processing device 100 according to the first embodiment. [Figure 5] FIG. 5 is a diagram (part 2) showing an image of an output result of information processing device 100 according to the first embodiment. [Figure 6] FIG. 6 is a hardware configuration diagram showing the hardware configuration of the information processing device 100 according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Embodiment 1 1 is a functional block diagram showing a functional configuration of an information processing device 100 according to embodiment 1. As shown in FIG. 1, the information processing device 100 according to embodiment 1 includes a scraping unit 110 (big data acquisition unit 110), a morphological analysis unit 120, and a clustering unit 130.

[0012] Fig. 2 is a flowchart showing a processing flow of the information processing device 100 according to Embodiment 1. As shown in Fig. 2, the processing flow of the information processing device 100 according to Embodiment 1 includes a text data acquisition process (ST10), a morphological analysis process (ST20), and a clustering process (ST30). Here, the text data acquisition process (ST10) is a processing step performed by the scraping unit 110. The morphological analysis process (ST20) is a processing step performed by the morphological analysis unit 120. The clustering process (ST30) is a processing step performed by the clustering unit 130.

[0013] FIG. 3 is a flowchart showing detailed processing steps of the information processing device 100 according to the first embodiment. ST11 to ST14 appearing in FIG. 3 are detailed processing steps of the text data acquisition process (ST10) performed by the scraping unit 110. ST21 to ST23 appearing in FIG. 3 are detailed processing steps of the morphological analysis process (ST20) performed by the morphological analysis unit 120. ST31 to ST34 appearing in FIG. 3 are detailed processing steps of the clustering process (ST30) performed by the clustering unit 130.

[0014] As illustrated in FIG. 3, the information processing device 100 may be programmed to handle, for example, a GUI file, a scraping file, a morphological analysis file, and a BERTopic file. Here, GUI stands for Graphical User Interface. BERTopic, simply put, is a technology that combines BERT, which stands for Bidirectional Encoder Representations from Transformer, and Topic analysis. The role of each file will become clear from the explanation below.

[0015] <<Scraping unit 110 constituting information processing device 100>> The scraping unit 110 of the information processing device 100 is a component that automatically extracts information required by users from big data on the Internet. The word "scraping" in the name comes from the English word "scrape," which literally means "to scrape off or scrape off." To clarify that scraping is performed from websites, "scraping" is often referred to as "web scraping."

[0016] An important part of the scraping unit 110 is the "data selector" used to find the data to be extracted from the HTML file. The data selector may be, for example, XPath, CSS, regular expressions, or a combination thereof.

[0017] When the information processing device 100 is started up, a screen is displayed on the output interface 240 (see FIG. 6) of the information processing device 100 so that the user can input necessary information into each component of the GUI file. The user inputs information via the input interface 210 (see FIG. 6) of the information processing device 100. Details of the hardware 200 of the information processing device 100, such as the input interface 210 and the output interface 240, will become clear from the explanation given below with reference to FIG.

[0018] The scraping unit 110 transfers information input by the user to the GUI file, such as "designated keywords" and "number of pages to be scraped," to the scraping file (ST11 in FIG. 3).

[0019] The scraping unit 110 opens a website such as Yahoo! Answers (Yahoo! is a registered trademark) based on the information written in the scraping file, and starts a search for the specified keyword (ST12 in FIG. 3). The scraping unit 110 may access a predetermined website using a framework that can automate the operation of a web browser, such as Selenium.

[0020] The scraping unit 110 acquires information including the posting link, posting date and time, posting title, and posting content from the website (ST13 in FIG. 3). The posting link refers to the link destination of each post displayed on the search result page. The posting date and time refers to the date and time when each posted content was created. The posting title refers to the title or heading of each posted content. The posting content refers to the main text information content of each posted content.

[0021] The scraping unit 110 stores the acquired information in a storage medium, for example, as a CSV file (ST14 in FIG. 3). The path of the file stored in the storage medium (referred to as a "file path") is sent to the morphological analysis unit 120 (ST21 in FIG. 3).

[0022] Some websites accessed by the scraping unit 110 have an upper limit on the number of search results that can be displayed. For example, if a website displays 10 search results per page and the upper limit of search result pages is 100, the upper limit of the number of search results that can be searched is 1,000. In addition, some websites do not allow scraping at all due to reasons such as limiting the number of accesses or copyright protection.

[0023] In addition, when handling big data without performing web scraping, the scraping unit 110 may be given a name (big data acquisition unit 110) that reflects the processing content and suggests that it acquires big data.

[0024] <<Morphological analysis unit 120 configuring information processing device 100>> The morphological analysis unit 120 included in the information processing device 100 is, simply put, a component that performs morphological analysis, which is a type of natural language processing. The term "morpheme" included in the name of the morphological analysis unit 120 refers to a unit smaller than a word, and is the smallest unit that carries meaning in a language. For example, the English word "book" and its plural form "books" are different words, but the latter is divided into smaller morphemes, such as "book" and "s," and each word shares the morpheme "book." In the case of Japanese, due to the nature of the language, the identification of "words" itself is not clear. Therefore, in natural language processing for Japanese, there is no strict distinction between word analysis and morphological analysis. Specifically, Japanese morphological analysis includes the process of dividing text into words (or morphemes) (also known as "word segmentation processing"), the process of identifying parts of speech, and the process of inflection (the process of finding the root form).

[0025] When the morphological analysis unit 120 receives the file path of the CSV file to be subjected to morphological analysis, it acquires the CSV file from the storage medium based on the file path (ST22 in FIG. 3). The text data acquisition process (ST10) performed by the scraping unit 110 and the morphological analysis process (ST20) performed by the morphological analysis unit 120 may be a series of processes that are continuous in time, or may be separate processes that are not continuous in time. The user can specify in the GUI file whether or not to have the scraping unit 110 perform the text data acquisition process (ST10) as a series of operations (ST21 in FIG. 3).

[0026] The morphological analysis process (ST20) performed by the morphological analysis unit 120 may use a morphological analysis system in which the dictionary and the system are separated, such as MeCab. When the morphological analysis unit 120 uses MeCab, it can use multiple dictionaries, such as ipendic, UnitDic, and JUMAN dictionary. The morphological analysis process (ST20) may use the MeCab library to extract words that have been learned as important from the text of the CSV file (ST22 in Figure 3).

[0027] The morphological analysis unit 120 may visualize and output text data consisting of words extracted as important (ST22 in FIG. 3). The morphological analysis unit 120 may visualize the text data using, for example, a word cloud. FIG. 4 shows an image of the output result when the morphological analysis unit 120 uses a word cloud.

[0028] The morphological analysis unit 120 may count the words extracted as important by type and output the frequent word list as a text file (ST23 in FIG. 3). The frequent word list output by the morphological analysis unit 120 may be saved, for example, in a subfolder for output files under the current directory.

[0029] <<Clustering unit 130 constituting information processing device 100>> The clustering unit 130 constituting the information processing device 100 is, simply put, a component that clusters posted articles by topic. The clustering unit 130 does not use the analysis results from the morphological analysis unit 120. Therefore, a user can use the information processing device 100 according to the disclosed technology to skip the morphological analysis process (ST20) and perform only the clustering process (ST30). Alternatively, a user can use the information processing device 100 according to the disclosed technology to perform the morphological analysis process (ST20) and skip the clustering process (ST30).

[0030] When clustering for each topic, the clustering unit 130 scores the importance of keywords related to each topic. The clustering unit 130 may perform this scoring using, for example, the Topic Word Scores function of BERTopic (see ST32 in FIG. 3). In this case, the clustering unit 130 may obtain the number of topics in the Topic Word Scores via a GUI file (ST31 in FIG. 3). Thereafter, the clustering unit 130 calls the BERTopic method (ST32 in FIG. 3) and outputs the Topic Word Scores to the browser (ST33 in FIG. 3).

[0031] The clustering unit 130 maps the distances between topics and visualizes how and to what extent different topics are related. The clustering unit 130 may perform this mapping using, for example, the Intertopic Distance Map function of BERTopic (see ST32 in FIG. 3). In this case, the clustering unit 130 may acquire the number of maps in the Intertopic Distance Map via a GUI file (ST31 in FIG. 3). The clustering unit 130 then outputs the Intertopic Distance Map to the browser (ST34 in FIG. 3).

[0032] Fig. 5 shows an image of the intertopic distance map that the clustering unit 130 outputs to the browser. As shown in Fig. 5, the clustering unit 130 performs clustering processing (ST30) for each article. The intertopic distance map can be considered as a reduced-dimensional feature space.

[0033] As exemplified by the Intertopic Distance Map, the clustering process (ST30) performed by the clustering unit 130 can be considered to be performed in a reduced-dimensional feature space. From this perspective, it is possible to understand the processing process in which the clustering unit 130 performs clustering through the processing procedures of vectorization and dimensional reduction. The clustering unit 130 converts the target text data into a feature vector, which is a numerical vector. The clustering unit 130 performs the conversion into the feature vector using a multilingual model, for example, the paraphrase-multilingual-MiniLM-L12-v2 model. The clustering unit 130 may appropriately reduce the dimension of the feature vector to make it easier to handle, for example, by using techniques such as singular value decomposition and principal component analysis. When BERTopic is used, UMAP (Uniform Manifold Approximation and Projection) is provided to perform dimension reduction. The clustering unit 130 may perform the clustering process (ST30) based on an algorithm such as HDBSCAN or K-means. HDBSCAN is a clustering algorithm that is provided by default in BERTopic. K-means has the advantage of being able to classify data into a specified number of clusters, making it highly convenient. The specified number of clusters may be determined as appropriate depending on the situation, purpose, etc. in which the disclosed technology is used.

[0034] FIG. 5 shows an example of articles classified into three classes as a result of the clustering process (ST30) performed by the clustering unit 130. The first class is about "consultations about stomach aches, money, children, accidents, energy issues, nuclear power plants, gas, and baths." The second class is about "discussions about cars, engines, and motorcycles." The third class is about "consultations about gas stove operation and air conditioners, and questions about propane gas and city gas." In this example, the gas company could improve the efficiency of its analysis by focusing its analysis on articles classified into the third class.

[0035] 6 is a hardware configuration diagram showing the hardware configuration of information processing device 100 according to embodiment 1. As shown in FIG. 6, hardware 200 of information processing device 100 includes an input interface 210, a processor 220, a memory 230, and an output interface 240.

[0036] The functions of the scraping unit 110, morphological analysis unit 120, and clustering unit 130 in the information processing device 100 are realized by processing circuits. That is, the scraping unit 110 includes processing circuits for performing text data acquisition processing (ST10), morphological analysis processing (ST20), and clustering processing (ST30). The processing circuit is a processor 220 (also referred to as a CPU, central processing unit, processing device, arithmetic unit, microprocessor, microcomputer, or DSP) that executes programs stored in a memory 230.

[0037] The functions of the scraping unit 110, morphological analysis unit 120, and clustering unit 130 are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in memory 230. The processor 220 realizes the functions of each unit by reading and executing the programs stored in memory 230. That is, the information processing device 100 includes memory 230 for storing programs that, when executed by the processor 220, result in the execution of the text data acquisition process (ST10), the morphological analysis process (ST20), and the clustering process (ST30). These programs can also be said to cause a computer to execute the procedures or methods of the scraping unit 110, the morphological analysis unit 120, and the clustering unit 130. Here, the memory 230 may be, for example, a non-volatile or volatile semiconductor memory such as RAM, ROM, flash memory, or EPROM. The memory 230 may also be in the form of a disk such as a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, a DVD, etc. The memory 230 may also be in the form of an HDD or an SSD.

[0038] The program stored in the memory 230 (hereinafter referred to as the "information processing program") may execute the detailed processing steps shown in Tables 1 to 4 below.

[0039] Table 1 below shows an example of detailed processing steps of an information processing program related to a GUI file. Table 1. Detailed processing steps of information processing programs related to GUI files TIFF2025133127000002.tif62139

[0040] Table 2 below shows an example of detailed processing steps of the information processing program associated with the scraped file. Table 2. Detailed processing steps of the information processing program associated with the scraped file TIFF2025133127000003.tif67163

[0041] Table 3 below shows an example of detailed processing steps of the information processing program associated with the morphological analysis file. Table 3. Detailed processing steps of the information processing program related to the morphological analysis file TIFF2025133127000004.tif99159

[0042] Table 4 below is an example of detailed processing steps of an information processing program associated with a BERTopic file. Table 4. Detailed processing steps of the information processing program related to BERTopic files TIFF2025133127000005.tif73162

[0043] As described above, the information processing device 100 according to embodiment 1 has the above-mentioned characteristics, and therefore has the effect of being able to easily and quickly collect and visualize consumer needs regarding unexpressed gases. [Industrial Applicability]

[0044] The disclosed technology can be applied to gas company systems that collect consumer gas-related needs from information available on the web, and has industrial applicability. [Explanation of symbols]

[0045] 100 information processing device, 110 scraping unit (big data acquisition unit), 120 morphological analysis unit, 130 clustering unit, 200 hardware, 210 input interface, 220 processor, 230 memory, 240 output interface.

Claims

1. A scraping unit that automatically extracts information necessary for users from big data, a morphological analysis unit that divides sentences into morphemes and performs language processing; A clustering unit that clusters the posted articles by topic. Information processing device.

2. An information processing method for an information processing device including a scraping unit, a morphological analysis unit, and a clustering unit, The scraping unit automatically extracts information necessary for the user from the big data, the morphological analysis unit divides the sentence into morphemes and performs language processing; The clustering unit clusters the posted articles by topic. Information processing methods.

3. An information processing program for causing a computer to execute the method according to claim 2.

Images