Anomaly detection method, detection device, and computer program

Abstract

To provide an anomaly detection method, a detection device, and a computer program capable of detecting comments regarding an anomaly in a game in a condition of 24 hours and 365 days.SOLUTION: An anomaly detection method collects data that is contributed regarding a game application, inputs the collected data to a learned model such that the data outputs data regarding an anomaly in the game application when the data is input, and stores data determined to be data regarding an anomaly from among collected data on the basis of data output from the model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an anomaly detection method, a detection device, and a computer program for detecting comments related to an anomaly in a predetermined game. [Background technology]

[0002] Elements of video games, such as scenarios, actions, graphics, and background music, are becoming increasingly complex and require more and more data, resulting in a wider variety of options. Patent Document 1 mentions that if the testing process for open-world-based games is left to humans, it would require a considerable number of man-months. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-108733 Summary of the Invention [Problem to be solved by the invention]

[0004] In online games, updates and new events are often implemented from time to time. For example, when such updates or new events are added, anomalies such as game crashes or bugs may occur in the game. If the game operator does not respond to these anomalies promptly, the game may become unpopular and users may abandon the game. Such anomalies can occur at times other than when updates or new events are added or around those times. When an anomaly occurs, the operator is required to recognize the occurrence of the anomaly as soon as possible.

[0005] The present invention aims to provide an anomaly detection method, detection device, and computer program that can detect comments regarding anomalies in a specified game on a specified website (e.g., an electronic bulletin board or a social networking service (SNS)) 24 hours a day, 365 days a year. [Means for solving the problem]

[0006] An anomaly detection method according to an embodiment of the present disclosure includes collecting data posted about a game application, inputting the collected data into an extraction model that has been trained to output data about an anomaly of the game application when the data is input, storing data among the collected data that is determined to be data about an anomaly based on the data output from the extraction model, and, when the data is input, extracting the data from a preset Bugs, glitches, and crashes The data on the stored anomalies is input to a classification model that has been trained to classify the anomalies, and the data on the stored anomalies is then extracted from the classification model. The aforementioned An anomaly detection method that outputs classification data indicating an anomaly classification, calculates the number of overlaps for each anomaly classification based on the output classification data indicating the anomaly classification, and stores each of the classification data indicating the anomaly classification in association with the corresponding number of overlaps.

[0007] The detection device according to an embodiment of the present disclosure includes a collection unit that collects data posted about a game application, a trained extraction model that, when data is input, outputs data about an anomaly of the game application for the input data, and a pre-set extraction model that, when data is input, outputs the data about an anomaly of the game application. Bugs, glitches, and crashes a first storage unit that stores a classification model that has been trained to classify the anomalies; and The aforementionedThe system includes a calculation unit that calculates the number of overlaps for each anomaly classification based on classification data that indicates the classification of the anomaly, and a second storage unit that inputs collected data into the extraction model and, based on the data output from the extraction model, stores data that is determined to be data related to an anomaly among the collected data and each of the classification data that indicates the classification of the anomaly, in association with the corresponding number of overlaps.

[0008] A computer program according to an embodiment of the present disclosure includes a computer program for collecting data posted on a game application, inputting the collected data into a trained extraction model so that when data is input, the data is output as data related to an anomaly of the game application, storing data among the collected data that is determined to be data related to an anomaly based on the data output from the extraction model, and, when data is input, extracting the data from a preset Bugs, glitches, and crashes The data on the stored anomalies is input to a classification model that has been trained to classify the anomalies, and the data on the stored anomalies is then extracted from the classification model. The aforementioned Classification data indicating the classification of anomalies is output, and a process is executed in which the number of overlaps for each classification of anomalies is calculated based on the output classification data indicating the classification of anomalies, and each of the classification data indicating the classification of anomalies is associated with the corresponding number of overlaps and stored.

[0009] The anomaly detection method, detection device, and computer program disclosed herein automatically collect data posted by game users regarding the game application to be detected, and use a trained model to store data that is likely to be data related to an anomaly.

[0010] The data posted by users may be text data, image data, audio data, or video data, or may be a combination of two or more of the text data, image data, audio data, and video data.

[0011] In one embodiment of the anomaly detection method of the present disclosure, the model is trained to, when data is input, output a score corresponding to the likelihood that the data is data related to an anomaly of the game application, and based on the score, determine whether the input data is data related to an anomaly.

[0012] In the anomaly detection method of the present disclosure, the stored data is associated with data on classifications of anomalies, such as "bug," "stop," and "crash," using a trained model.

[0013] In the anomaly detection method of the present disclosure, data may be collected periodically. At the end of each period, determination of whether the data is related to an anomaly and classification may be performed for each period. The period may be variable and may be set.

[0014] The anomaly detection method according to an embodiment of the present disclosure creates a report including a list of data relating to the stored anomalies, and notifies the outside of the creation of the report.

[0015] The anomaly detection method of the present disclosure creates a report and allows the report to be viewed on a computer operated by an operator. Being able to view the report as a report can speed up the process of taking action.

[0016] In the anomaly detection method according to one embodiment of the present disclosure, the report includes a link for accessing each piece of data.

[0017] In the anomaly detection method of the present disclosure, the operator can access data (primary information) posted by game users through the report that is created. [Effects of the Invention]

[0018] According to the present disclosure, it is possible to detect comments regarding anomalies in a specific game on a specific website 24 hours a day, 365 days a year. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a schematic diagram of an anomaly detection system of the present disclosure. [Figure 2] FIG. 2 is a block diagram showing the configuration of each device in the anomaly detection system. [Figure 3] 10 is a flowchart illustrating an example of a data collection and extraction process performed by a detection device. [Figure 4] 10 is a flowchart illustrating an example of a data classification processing procedure performed by the detection device. [Figure 5] 10 is a flowchart illustrating an example of a procedure for a report creation process performed by a detection device. [Figure 6] FIG. 10 is a diagram illustrating an example of a display on a client device. [Figure 7] FIG. 10 is a diagram showing an example of the contents of a report display screen. DETAILED DESCRIPTION OF THE INVENTION

[0020] The present invention will be specifically described with reference to the drawings showing embodiments thereof. The present invention extracts comments related to anomalies in a predetermined game from a large number of comments posted on a website (such as an electronic bulletin board or SNS) where users can post and view their desired comments. In this specification, an "anomaly" refers to an event that must be dealt with immediately in terms of the operation of a specific game, and relates to an abnormality in at least some elements of the game, such as an error, a malfunction, a bug, a game stopping, a game forced termination, etc. Therefore, for example, a comment regarding an anomaly in a specific game is a comment indicating that an error or bug has occurred in the specific game, or a comment calling attention to such an error or bug.

[0021] FIG. 1 is a schematic diagram of an anomaly detection system 100 according to the present disclosure. The anomaly detection system 100 according to this embodiment detects comments related to anomalies in applications known as mobile games, which use a smartphone or tablet device as hardware. The hardware for executing the game is not limited to a mobile device, but may also be a so-called game console such as a home game machine or an arcade game machine, or a personal computer. The detection target is not limited to mobile game applications or game software (hereinafter referred to as game apps), but also includes applications that use an information processing device such as a smartphone or tablet device as the main hardware.

[0022] The anomaly detection system 100 includes a detection device 1 and a client device 2 used by the creator of a game app to be detected. The detection device 1 is connected to a network that can be connected to the Internet. The detection device 1 transmits and receives data to and from the client device 2 via a server device that is connected to a network N inside or outside the system.

[0023] The detection device 1 collects data related to the game app to be detected via the Internet by crawling. The detection device 1 classifies the collected data and outputs a report for each classification. The detection device 1 continues crawling 24 hours a day without omission and stores the data in the memory unit 11. Note that crawling may be set to be performed only during specified time periods. The detection device 1 automatically performs classification processing on the data stored sequentially. The detection device 1 creates a report based on the results of the classification processing and notifies the user. The timing of this creation and notification can be set as desired (for example, every hour, etc.).

[0024] Previously, for game apps targeted for detection, staff would collect text posted on social media, blogs, and other platforms related to the app, compile it into a report, and share it with relevant parties. Because this work required manual effort, it was difficult to collect all posted text over a 24-hour period, given the cost and effort involved. Therefore, the collection and report creation were performed at specific time slots. In this case, text posted outside of the designated collection time slots could be overlooked and missed, or collected at the next designated collection time slot, resulting in a long time lag. Furthermore, the larger the game app's user base, the more difficult it becomes to extract anomalies from posted text. In other words, the larger the game app's user base, the greater the volume of illustrations, photos, and comments (whether simple criticism or comments such as "it's fun") posted by fans of the game app. These are not targeted for anomaly detection and must be excluded from extraction.

[0025] In the anomaly detection system 100 of the present disclosure shown in FIG. 1, the detection device 1 automatically continues crawling the search target website and stores the data. The detection device 1 sequentially classifies the data collected by crawling, dividing the data into time periods. This reduces the amount of posted data that is omitted from the collected data.

[0026] If the classified data includes, for example, text, audio, and video with the same content (purpose), it is advisable to select one of them as the representative data (name matching) and then compile the representative data by adding the number of data items with the same content.

[0027] The detection device 1 may extract data collected by crawling and classify the extracted data using a trained model. It is preferable to use a model that has undergone supervised learning using data that has been manually extracted and classified in advance.

[0028] The detection device 1 creates a report from the classified data, stores it, uploads it to the server device, and notifies the client device. The report may be a list of the classified data or may be presented as a chart. Notification may be via data sharing, message exchange, or email, which are functions implemented by the server device.

[0029] It is desirable that the report display on the client device 2 be intuitive and easy to understand, and that all information be displayed. The report may be compiled in a spreadsheet and saved in a data sharing application as it is compiled. It is desirable that the compiled report include a direct link to each posted text.

[0030] The following describes in detail the configuration and processing content of each device for realizing such an anomaly detection system 100. Figure 2 is a block diagram showing the configuration of each device of the anomaly detection system 100. The detection device 1 includes a processing unit 10, a storage unit 11, and a communication unit 12.

[0031] In the following description, the detection device 1 is described as being configured with one server computer, but it may also be configured as a configuration in which multiple server computers are communicatively connected via a network to perform distributed processing. The detection device 1 may be a cloud-type server that can be communicatively connected from an information processing device via a public network, or may be an on-premise server that communicatively connects with each client device 2 via a virtual or physical private network.

[0032] The processing unit 10 is a processor that uses a CPU (Central Processing Unit) and / or a GPU (Graphics Processing Unit). Based on a detection program 1P stored in the storage unit 11, the processing unit 10 collects data related to target game apps and performs extraction and classification.

[0033] The storage unit 11 uses a non-volatile memory such as a hard disk or an SSD (Solid State Drive). The storage unit 11 stores data referenced by the processing unit 10. The storage unit 11 stores a detection program 1P. The storage unit 11 stores a trained extraction model 11M used for extraction and a trained classification model 12M used for classification. The detection program 1P may be a detection program 9P stored in a recording medium 9 that is read by the processing unit 10 and copied to the storage unit 11. The extraction model 11M or the classification model 12M may also be an extraction model 91M or a classification model 92M stored in the recording medium 9 that is read by the processing unit 10 and copied to the storage unit 11.

[0034] The storage unit 11 stores data relating to anomalies extracted and classified by the process described below. The storage unit 11 storing the data relating to anomalies may be an external storage medium such as a flash memory, and may be removable from the detection device 1.

[0035] The communication unit 12 realizes data communication via the network N. Specifically, the communication unit 12 is, for example, a network card. The processing unit 10 accesses data on the Internet via the communication unit 12 and performs crawling.

[0036] The client device 2 is, for example, a laptop personal computer, but may also be a desktop personal computer, a tablet terminal, or a so-called smartphone.

[0037] The client device 2 includes a processing unit 20, a storage unit 21, a communication unit 22, a display unit 23, and an operation unit 24. The processing unit 20 is a processor using a CPU and / or a GPU. Based on a client program 2P stored in the storage unit 21, the processing unit 20 causes the display unit 23 to display a report obtained from the detection device 1 via the server device.

[0038] The storage unit 21 uses a nonvolatile memory such as a hard disk, a flash memory, or an SSD. The storage unit 21 stores data referenced by the processing unit 20. The storage unit 21 stores the client program 2P.

[0039] The communication unit 22 realizes data communication via the network N. Specifically, the communication unit 22 is, for example, a network card. The processing unit 20 can acquire data from the detection device 1 via the server device using the communication unit 22.

[0040] The display unit 23 is a display such as a liquid crystal display or an organic EL (Electro Luminescence) display. The display unit 23 displays a screen including information based on data stored in the storage unit 21 or data provided from the detection device 1. The display unit 23 may be a display with a built-in touch panel.

[0041] The operation unit 24 is a user interface such as a keyboard and a pointing device that can input and output data to and from the processing unit 20. The operation unit 24 may be a voice input unit. The operation unit 24 may be a touch panel built into the display unit 23. The operation unit 24 may be physical buttons.

[0042] 3 is a flowchart showing an example of a data collection and extraction process procedure by the detection device 1. The detection device 1 repeatedly executes the following process at a predetermined timing. The predetermined timing may arrive periodically or may arrive when a predetermined condition is satisfied.

[0043] The processing unit 10 accesses a website (such as a predetermined community site, bulletin board site, SNS, or posting site) set as a detection target via the network N, and acquires data (text, audio, image, or video) posted during a target period related to the game app to be detected (step S101). The target period is the period from the arrival of the current timing back to the arrival of the previous timing.

[0044] The target period may be set by an operator via the client device 2. This shortens the time between when a user posts and when it is communicated to the creator, realizing immediate anomaly detection. If the creator can predict an increase in the frequency of anomalies, such as during a limited-time event or immediately after a major update, it is possible to change the division of this period to increase the frequency of detection.

[0045] In step S101, the processing unit 10 accesses each "group" related to the game app if it is a predetermined community site. If it is a bulletin board site, the processing unit 10 accesses each so-called "board," "genre," "thread," or "topic." If it is an SNS or posting site, the processing unit 10 acquires data extracted using keywords such as a hashtag for the name of the game or the names of characters appearing in the game. The processing unit 10 may crawl all data related to the target game app, or may select and acquire data based on keywords specified in advance at the time of step S101.

[0046] The processing unit 10 selects data from the acquired data (step S102) and provides the data to the extraction model 11M (step S103).

[0047] In step S103, the processing unit 10 may convert the selected data into text, perform morphological analysis, and then provide the text as a word list (vector) to the extraction model 11M. If the data is audio, the processing unit 10 may perform speech recognition and then morphological analysis. If the data is an image, the processing unit 10 may perform character recognition on the characters included in the image, or convert the metadata into text and then perform morphological analysis. If the data is a video, the processing unit 10 may perform both character recognition on the characters included in the image and speech recognition on the audio.

[0048] The processing unit 10 determines whether the data is related to an anomaly such as a bug or a malfunction based on the score from the extraction model 11M (step S104).

[0049] In step S104, the processing unit 10 obtains from the extraction model 11M a score corresponding to the likelihood that each word in the input word list is an anomaly-related data for that word. The extraction model 11M uses a neural network. For example, it is a model using a TextCNN (Convolutional Neural Network). Alternatively, a recurrent neural network (RNN) or a recurrent neural network (LSTM: Long Short-Term Memory / GRU: Gated Recurrent Unit), which learns by taking time series factors into account, may be used. The extraction model 11M may be trained in advance based on past manual extraction results, or may be trained using newly created training data. For example, the extraction model 11M may be trained by assigning a high likelihood that words such as "bug," "glitch," "crash," "problem," "stop," and "strange" are anomalies. The processing unit 10 sums up the scores obtained from the extraction model 11M for each word and calculates a score for the entire data indicating whether the data is an anomaly-related data. The extraction model 11M for determining whether the data is an anomaly-related data is not limited to this, and any known method for text analysis may be used. In steps S103 and S104, the processing unit 10 may determine whether the data is negative or not using the extraction model 11M that outputs a score corresponding to the likelihood that the data is negative (or positive).

[0050] If the processing unit 10 determines that the data is related to an anomaly (S104: YES), it stores the data in the storage unit 11 (step S105) and proceeds to step S106. If the processing unit 10 determines that the data is not related to an anomaly (S104: NO), it proceeds to step S106.

[0051] In step S105, the processing unit 10 may store a word list of the data morphologically analyzed in step S103. In step S105, the processing unit 10 may store link data that allows direct access to the original data (site, post) of the acquired data in association with the original data.

[0052] The processing unit 10 determines whether the process of providing all the data acquired in step S101 to the extraction model has been completed (step S106), and if it is determined that the process has been completed (S106: NO), returns the process to step S102.

[0053] If it is determined that all of the acquired data has been given to the extraction model (S106: YES), the processing unit 10 ends the process.

[0054] The processing unit 10 may execute the processing procedure shown in the flowchart of Fig. 3 for each predetermined site. The processing procedure shown in the flowchart of Fig. 3 causes the results of extracting data related to anomalies of the target game app to be stored in the storage unit 11.

[0055] 4 is a flowchart showing an example of a data classification processing procedure performed by the detection device 1. The detection device 1 may execute the following processing immediately after completing the data collection and extraction processing of the detection device 1. The execution timing of the following processing and the execution timing of the data collection and extraction processing of FIG. 3 may be independent of each other, or may be executed in parallel.

[0056] The processing unit 10 reads extracted data relating to anomalies for a game to be detected from the storage unit 11 (step S201).

[0057] The processing unit 10 provides the read data to the classification model 12M (step S202). In step S202, the processing unit 10 may convert the read data into text, perform morphological analysis, and then provide the text to the classification model. If a morphologically analyzed word list is stored, the word list may be provided. The classification model 12M may be trained in advance based on past manual classification results. Examples of classifications include "bug," "glitch," "stop / crash," and "improvement request." The processing unit 10 obtains from the classification model 12M a score indicating the likelihood that each word in the input word list belongs to one of the preset classifications. The classification model 12M may employ a known method for text classification. The classification model 12M may be trained to output a score corresponding to the severity level along with the classification.

[0058] The processing unit 10 stores classification data indicating the classification of the given data based on the score from the classification model 12M in association with the read data in the storage unit 11 (step S203). The processing unit 10 determines to which classification the entire data belongs based on the acquired score.

[0059] The processing unit 10 determines whether or not unclassified data remains in the storage unit 11 (step S204). If the processing unit 10 determines that unclassified data remains (S204: YES), the processing unit 10 returns the process to step S201, reads out the unclassified data, and continues the process.

[0060] If it is determined in step 204 that there are no remaining items (S204: NO), the processing unit 10 ends the classification process.

[0061] According to the processing procedure shown in the flowchart of FIG. 4, data extracted from a predetermined site or SNS as having a high probability of being data related to an anomaly is classified. The classified data (text, audio, image, or video) is stored in association with the classified data in the storage unit 11. This data can be read and displayed from the client device 2 together with the classified data, allowing the creator to more quickly identify defects, bugs, etc. Each piece of classified data is stored in association with a link to access the original data of that data. This allows the client device 2 to read the data stored in the storage unit 11, making it easy for an operator to check the primary information posted by users of the game app to be detected.

[0062] 5 is a flowchart showing an example of a report creation process procedure by the detection device 1. The detection device 1 may execute the following process immediately after completing the data classification process of the detection device 1. The execution timing of the following process, the execution timing of the data collection and data extraction process of FIG. 3, and the execution timing of the data classification process of FIG. 4 may be independent of each other, or may be executed in parallel.

[0063] The processing unit 10 reads out, for each category, the classified data associated with the classification data from the storage unit 11 (step S301). In step S301, the processing unit 10 reads out a word list or original data (text, audio, image, or video). In the case of audio, image, or video, the processing unit 10 may read out a text version of the data.

[0064] The processing unit 10 counts the number of data for each category (step S302) and stores the number of data in association with the category data and time information (step S303). The processing unit 10 consolidates duplicate data for each category (step S304). Step S304 is not essential. In step S304, the processing unit 10 selects one piece of data for each category and determines, one by one, whether the content of the selected data overlaps with other data in the same category, i.e., whether the text overlaps. If the degree of overlap is equal to or greater than a predetermined level, the number of overlaps for the selected data is added. The selected data and the number of overlaps for that data are stored in association with the data's identification data. The processing unit 10 repeatedly selects data and consolidates duplicate data.

[0065] The processing unit 10 creates a report using document data that lists the number of data items for each category, time information, the text of the selected data, and the number of duplicates (step S305). In step S305, the processing unit 10 creates spreadsheet data, document data, email, or specific HTML data. The report creation step may be realized by automating steps such as a step of reading data for each category from the storage unit 11 and displaying the data in a list using, for example, spreadsheet software, and a step of sorting the displayed list using a display unit provided in the detection device 1, using RPA (Robotic Process Automation).

[0066] The processing unit 10 notifies the client device 2 that the report has been created (step S306), and ends the process.

[0067] In step S306, the processing unit 10 notifies the user by a message, email, or the like including a link to the report. The notification recipients may be set in advance. For example, operator identification data of the operators to be notified for each category may be stored. The operator identification data may be an email address, an account in a data sharing application, or a message application. The operator identification data may be a community ID or team ID in a data sharing application.

[0068] The notification of the report may be the delivery of an external storage medium on which the extracted and classified data is stored, for example, a portable flash memory.

[0069] The report creation process in Fig. 5 is not essential. If it is determined that classification into all data has been completed at the stage of the classification process procedure shown in the flowchart in Fig. 4 (S204: NO), the processing unit 10 of the detection device 1 may notify the client device 2 in step S306.

[0070] Anomaly detection using the processing procedures shown in the flowcharts of FIGS. 3 to 5 will be described using specific examples. The anomaly detection system 100 detects comments related to anomalies in a specific game app. The creator of the specific game app stores, for example, the address of a community site related to the game app in advance in the storage unit 11 of the detection device 1. The community site for the specific game app is a message posting site or bulletin board for each game on a game strategy site. These posting sites and bulletin boards may be categorized by events or strategy themes. The creator may specify a URL specifying a specific "board" or "genre" on the bulletin board site and store it in the storage unit 11. The specification of a "board" or "genre" is not limited to a search keyword. The creator may specify a search keyword for an SNS and store it in the storage unit 11. The detection device 1 has a server function and may accept and store these specifications from the client device 2. When specifying a search keyword, it is necessary to also specify abbreviations or other terms used by the user.

[0071] In the specific example described below, it is assumed that the abbreviation of the game app is "ABC" and that a message with attached image data saying "I think this is a bug" has been posted. It is also assumed that a message such as "It froze" or "The display is strange" has been posted to a community site.

[0072] The processing unit 10 of the detection device 1 acquires data using the stored address, URL, and search keyword (S101). For example, the processing unit 10 accesses a specified address within a site for attacking the community site "ABC" and acquires the posted text. If image data has been uploaded along with the text, the processing unit 10 acquires the image data as well. At this time, the processing unit 10 acquires only data whose posting time falls within the target period. For community sites, the processing unit 10 acquires data posted within the target period of the specified site without sifting through it. By acquiring data without searching using search keywords, the processing unit 10 can reduce data that is missed from the search as much as possible. For example, the processing unit 10 may acquire data posted by users from an SNS using "ABC" as a search keyword.

[0073] As described above, the period for acquiring data by the detection device 1 may be set by an operator from the client device 2. If the period is set shorter than a predetermined initial value, the time from user posting to the completion of classification can be shortened. In periods when the possibility of anomalies occurring is low, such as when no events or updates are scheduled for the game application or there are no OS updates for the hardware, the period can be set longer than the predetermined initial value.

[0074] The processing unit 10 provides the word list obtained by morphologically analyzing the acquired data to the extraction model 11M (S103). As described above, the extraction model 11M outputs a score for each word included in the word list indicating the likelihood that the data is a post or comment related to an anomaly. The processing unit 10 totals the scores for each word included in the word list to calculate a score indicating the likelihood that the data as a whole is related to an anomaly. At this time, it is preferable to assign weights to specific words such as "bug," "glitch," "crash," and "failure," so that posts and comments containing these words can be determined with a high probability to be data related to an anomaly. The extraction model 11M itself may be trained by assigning large weights to these words. The score may be calculated not as a total score but as a numerical value such as the occurrence probability of words that are likely to be determined to be data related to an anomaly, by dividing the score by the number of words included in the word list.

[0075] The processing unit 10 determines whether the data is related to an anomaly based on whether the score calculated for the entire data is equal to or greater than a predetermined value (S104). For example, if a post with attached image data saying "I think this is a bug" is posted on a community site, the morphological analysis will analyze it as "This / Bug / Is / To / I think." No scores are calculated for particle words such as "This," "Is," or "To," but a score indicating the likelihood of the noun word "bug" being related to an anomaly is calculated. By assigning a large weight to "bug," the processing unit 10 determines based on the score calculation result that the text of the post saying "I think this is a bug" and the attached image data are related to an anomaly.

[0076] The processing unit 10 may perform character recognition on the attached image data before providing it to the extraction model 11M. The image data may be a game screen or an illustration reproducing a game screen. If it is a game screen, the processing unit 10 may convert the results of the character recognition into text, such as the names of enemy and ally characters, lines, status values, status, skill names for special effects, and names of selected techniques included in the game screen, and use this text as related data. Even if the data attached to the note is not image data but moving image data or audio data, the processing unit 10 may similarly convert this text into text and use this as related data.

[0077] The processing unit 10 may also provide the associated data, which is a text version of the attached data, to the extraction model 11M in step S103. However, if it is treated at the same level as the written text, there is a possibility that it will be recognized as data related to an anomaly based on negative comments. It is advisable not to provide the text of the image recognition results to the extraction model 11M, and to use the associated data obtained by image recognition for later classification.

[0078] The processing unit 10 stores the target data determined to be data related to an anomaly (S105). As described above for step S105, the processing unit 10 stores the acquired text of the post "I think this is a bug," the image data attached to the text, and the word list. The processing unit 10 may store both the text "I think this is a bug" and a URL for accessing the post itself, including the attached image data. This may be stored as cache data rather than as a URL for direct access. If image data is attached, the processing unit 10 may also store the text recognized from the image data. This is for use during classification, as described above.

[0079] In the classification process, the processing unit 10 reads data including the text "I think this is a bug" stored in the storage unit 11. The processing unit 10 may read the original text or text obtained by image recognition or voice recognition. If a word list after morphological analysis is stored, the processing unit 10 may read this.

[0080] The processing unit 10 provides the classification model 12M with a word list obtained after morphological analysis of the read data (S202), and determines the classification based on the score for each classification for each word obtained from the classification model 12M. When the data "I think this is a bug" is provided to the classification model 12M, the processing unit 10 can obtain a score indicating a high probability that the word "bug" is classified as a "bug." For other words, "I think this is a bug," the scores may not be calculated because they are general-purpose words. The processing unit 10 cannot obtain scores for classifications other than "bug," and stores classification data (identification number) indicating the classification of "bug" in association with the data "I think this is a bug." Note that the classification process does not have to be limited to processing using the classification model 12M.

[0081] In step S202, the processing unit 10 may classify the large category of "bug" into more detailed categories based on associated data such as character names and lines obtained from the attached image data associated with the data.

[0082] As a result of the classification process, the storage unit 11 stores the extracted data and classification data indicating the classification of the data for each period.

[0083] In a specific example, the processing unit 10 creates a report. The processing unit 10 reads the classified data for each category (S301) and summarizes them (S304). The processing unit 10 may summarize posts such as "I think this is a bug" and posts such as "Isn't this a bug?", "It's a bug," "I found a bug," and "A bug has occurred." The processing unit 10 may determine that the degree of overlap is high when, among the data associated with the classification data of "bug," a specific word, for example, "bug," is common and the words other than the word "bug" are general-purpose words. The processing unit 10 may determine whether the words other than the specific word, "discovery" and "think," are similar to each other using a thesaurus, co-occurrence dictionary, etc., and, if they are similar, process them to increase the degree of overlap. The processing unit 10 may also determine the overlap of meanings using known language processing.

[0084] The processing unit 10 creates a report based on the counting and / or summarizing results of step S302 (S305). In a specific example, the processing unit 10 creates the report using both spreadsheet data and document data. The processing unit 10 creates a report using spreadsheet data that describes, by classification, the text of the classified data extracted in each row and link data for accessing that data. The spreadsheet data here refers to data in a format that can be viewed with a spreadsheet application. The processing unit 10 creates the report using document data that describes, for each classification, the text of the summarized data and the number of duplicates. The document data does not need to include link data, etc. The document data may include multiple written texts and image data of representative data of the summarized data.

[0085] The processing unit 10 uploads the created report to the file sharing application. The processing unit 10 notifies the developer by sending a message including link data to the report (spreadsheet data, document data, etc.) in the file sharing application to the developer's team ID in the message application. The client device 2 can receive the notification via the message application. The processing unit 10 may change the identification data identifying the operator to be notified for each classification. The processing unit 10 may send a message including a link to spreadsheet data including links to each data to the identification data of the operator who is the developer, and may send a message including a link to document data to the identification data of the operator who is the planner. The team ID and community ID may be separately notified.

[0086] A display example on the client device 2 in a specific example will be described with reference to the drawings. Fig. 6 is a diagram showing a display example on the client device 2. Fig. 6 shows a home screen 230 of a message application. Fig. 6 shows a display example of the home screen 230 when a developer operator among the creators logs in with his / her account. When the message application is launched with the account of the developer operator, as shown in Fig. 6, the home screen 230 includes a list area 231 of teams, themes, etc. for each project in which the account is participating, and a display area 232 for messages for a team or theme selected from the list.

[0087] An icon 233 indicates that one notification has been received for a team (theme) called "anomaly report" in a project called "ABC" included in a list area 231.

[0088] The display area 232 displays the contents of messages exchanged with other operators in the selected team or theme. In the display example of FIG. 6, a message is displayed indicating that an "anomaly report" has been uploaded from the detection device 1. The display area 232 displays an icon 234 indicating a link to the report. When the icon 234 is selected by the operation unit 24, a report display screen is displayed on the display unit 23 based on the address to the report in the data sharing application associated with the icon 234.

[0089] Fig. 7 is a diagram showing an example of the contents of the report display screen 235. The report display screen 235 displays a report in a table format using a spreadsheet application launched within the data sharing application. In the example of Fig. 7, a list of data text is shown for each classification. The number shown in the cell next to the text indicates the number of data items with the same content, i.e., the number of duplicates. When an operation is performed on the operation unit 24 for the number of duplicates, a list of all data items may be displayed.

[0090] In the example of Figure 7, the categories of "Bug" and "Stop" are displayed. Corresponding to the "Bug" category, data on posts such as "I think this is a bug" and the number of duplicates with the same meaning are displayed. In the example of Figure 7, text on posts with the same meaning as "I think this is a bug," such as "I found a bug" and "A bug has occurred," are displayed in separate lines. Each line corresponding to data with the same meaning contains a link to directly check each piece of data, and the text of related data stored in association with each piece of data.

[0091] The "Bug" category also displays data on posts that say "The display is strange" and the number of duplicates. Data on posts with the same meaning as "The display is bugged" is also displayed.

[0092] The "stopped" category displays data on posts that say "frozen" and the number of duplicates.

[0093] In the example of Figure 7, data is sorted and displayed for each category in descending order of the number of duplicates. The more posts with the same content there are, the sooner an operator can check them, shortening the time lag from the occurrence of an anomaly. Regarding categories, categories with a higher number of detections may be displayed at the top. Data may also be sorted and displayed by the level of severity determined.

[0094] In this way, operators can collectively check all data automatically and completely crawled by the detection device 1. By displaying data by classification, operators can collectively check the content that needs to be addressed. Responding to anomalies promptly is effective in preventing active users of game apps that use smartphones as hardware from abandoning them. Although identifying anomalies requires significant costs, checking all postings on websites related to the game apps being detected requires significant costs. However, by reviewing the data automatically collected, extracted, and classified by the detection device 1 as a report, manpower and time costs can be reduced. Automatic crawling is possible 24 hours a day, 365 days a year, reducing the risk of anomalies reducing appeal.

[0095] The report also displays the number of duplicates, as shown in Figure 7, allowing users to understand the number of recognized anomalies in comparison with others. Each piece of data is associated with a link to access the original post or posted text, allowing operators to check the primary information and more quickly deduce the cause of the anomaly from the preceding and following information, enabling them to take action and make arrangements more quickly.

[0096] 7, an operator may point out data that is not an anomaly and provide feedback to the detection device 1. The detection device 1 may retrain the extraction model 11M using data that is not an anomaly but is determined by the extraction model 11M to be data related to an anomaly.

[0097] 6 or the data sharing application shown in Fig. 7, it may be possible to specify specific community sites, boards, genres, search keywords, etc. to be crawled, and to specify a cycle, to the detection device 1. For example, the detection device 1 connects to the data sharing application using an account assigned to the detection device 1, references a stored setting file, reads search keywords, etc. included in the setting file, stores them in the storage unit 11, and uses them.

[0098] The embodiments disclosed above are illustrative in all respects and are not restrictive. The scope of the present invention is defined by the claims, and includes all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0099] 100 Anomaly Detection System 1. Detection device 10 Processing section 11 Storage section 1P detection program 2. Client Device 20 Processing section 23 Display section 24 Control section 2P client program

Claims

1. Collect data posted about the game application, inputting the collected data into an extraction model that has been trained to output data related to an anomaly of the game application when the data is input; storing data determined to be data relating to an anomaly among the collected data based on the data output from the extraction model; inputting the stored data relating to an anomaly into a classification model that has been trained to classify the data into a preset anomaly classification of bug, malfunction, or stop / crash when the data is input, and outputting classification data indicating the classification of the anomaly from the classification model; calculating the number of overlaps for each anomaly classification based on the classification data indicating the classification of the anomaly output; Each of the classification data indicating the classification of the anomaly is associated with the corresponding number of overlaps and stored. Anomaly detection methods.

2. periodically collecting posted data relating to said game application; The collected data is input into the extraction model for each cycle at the end of each cycle. The anomaly detection method according to claim 1 .

3. Generate a report that lists the data about the stored anomalies, Notifying the public of report creation The anomaly detection method according to claim 1 or 2.

4. a collection unit that collects data posted regarding the game application; a first storage unit that stores a trained extraction model that, when data is input, outputs data related to an anomaly of the game application for the input data, and a trained classification model that, when data is input, classifies the input data into a preset anomaly classification of bugs, malfunctions, and stoppages / crashes; a calculation unit that calculates the number of overlaps for each classification of the anomaly based on classification data indicating the classification of the anomaly output from the classification model; a second storage unit that inputs collected data into the extraction model, and stores data determined to be data relating to an anomaly among the collected data based on data output from the extraction model, and classification data indicating the classification of the anomaly, in association with the corresponding number of overlaps; A detection device comprising:

5. On the computer, Collect data posted about the game application, inputting the collected data into a trained extraction model such that, when the data is input, the model outputs data related to an anomaly of the game application; storing data determined to be data relating to an anomaly among the collected data based on the data output from the extraction model; inputting the stored data relating to an anomaly into a classification model that has been trained to classify the data into a preset anomaly classification of bug, malfunction, or stop / crash when the data is input, and outputting classification data indicating the classification of the anomaly from the classification model; calculating the number of overlaps for each anomaly classification based on the classification data indicating the classification of the anomaly output; Each of the classification data indicating the classification of the anomaly is associated with the corresponding number of overlaps and stored. A computer program that executes a process.

Images