system

Abstract

We provide the system. [Solution] A means for analyzing the screen interface and recognizing the screen configuration and each element, A means for automatically generating vulnerability test items based on each element on the screen that is recognized, A means of conducting an attack simulation against a screen interface using automatically generated test items, A means of analyzing test results and reporting discovered vulnerabilities, Means for scheduling continuous monitoring and retesting, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In conventional vulnerability diagnosis methods for software and websites, a lot of manpower is required for creating test items and preparing test data, and inspections with limited time and resources are restricted. Furthermore, frequent and continuous diagnosis is costly, and it is difficult to sufficiently guarantee its accuracy. Therefore, there is a demand for providing a system that can automatically detect vulnerabilities efficiently and with high accuracy and quickly take countermeasures.

Means for Solving the Problems

[0005] This invention provides a system that analyzes a screen interface to recognize the screen configuration and each element, and automatically generates vulnerability test items based on the recognized elements. Using the automatically generated test items, it performs attack simulations against the screen interface, analyzes the results, and reports the vulnerabilities found. Furthermore, it includes means for scheduling continuous monitoring and retesting, and achieves efficient diagnosis by automating input operations to each screen element. In addition, it generates a report including specific remediation measures for the discovered vulnerabilities and notifies the user, enabling a rapid response.

[0006] A "screen interface" is a visual structure for users to interact with software or websites, and it includes elements such as input fields and buttons.

[0007] "Analysis" is the process of investigating a specific object in detail to understand or grasp its structure and operation.

[0008] A "vulnerability" is a security flaw or weakness in a system or software that can be exploited by external attacks.

[0009] A "test item" is an element or condition that is verified in a test to achieve a specific purpose, and is used to evaluate the functionality and safety of a system in a particular situation.

[0010] "Automatic generation" refers to the process of mechanically creating information or content through a system or program without artificial intervention.

[0011] An "attack simulation" is a virtual test or experiment conducted to identify vulnerabilities in a system by mimicking potential threats and attack methods.

[0012] "Monitoring" is the activity of continuously observing a system or process, recording its state and changes, and analyzing them.

[0013] A "report" is a document that summarizes the results of a specific investigation or analysis in detail, and is used to provide information and aid understanding.

[0014] A "schedule" refers to a time plan or chart for carrying out a specific activity or task according to a plan. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] This invention provides a system for automatically detecting and efficiently reporting vulnerabilities in software and websites. The entire system mainly consists of a server and terminals, with the server handling overall control and data processing, and the terminals performing direct operations with the user interface.

[0037] Server-based analysis and test item generation

[0038] The server first analyzes the screen interface displayed on the terminal, identifying its structure and each element. Using the information obtained from this analysis, the server automatically generates vulnerability test items. For example, if a login page is detected, the server prepares test data for simulating SQL injection and brute-force attacks.

[0039] Vulnerability detection and testing

[0040] The terminal automatically performs operations based on the generated test items. The server receives these test results and analyzes the data responses returned by each operation. For example, if the terminal enters a specific string into an input form, the server determines whether or not there is a vulnerability from the resulting error or warning messages.

[0041] Reporting of results and suggestions for improvement

[0042] The server analyzes the detected vulnerabilities and creates a detailed report for the user based on the results. The report includes the issues found, a risk assessment, and specific improvement suggestions. For example, if an input field is determined to be vulnerable to SQL injection, the server will suggest to the user the implementation of prepared statements.

[0043] Continuous monitoring and scheduling

[0044] Users can schedule regular diagnostics for their servers. Based on this setting, the servers can automatically retest at specified times, constantly monitoring for the latest threats. For example, by scheduling weekly system-wide tests, it's possible to maintain security even for sites that require frequent updates.

[0045] Thus, this system provides detailed and automated vulnerability detection and reporting capabilities, contributing to the mitigation of security risks. User intervention is minimized, enabling efficient and effective security management.

[0046] The following describes the processing flow.

[0047] Step 1:

[0048] The server scans the screen interface displayed on the terminal to recognize the page structure and each element. Specifically, it parses the DOM (Document Object Model) and extracts attributes of input fields and buttons.

[0049] Step 2:

[0050] The server automatically generates vulnerability test items based on each element of the recognized interface. For example, it builds the test data and scenarios necessary to detect SQL injection and cross-site scripting (XSS).

[0051] Step 3:

[0052] The device uses the generated test items to perform automated actions on the user interface. Specifically, it enters attack strings into forms and executes the submit action. This is done using browser automation tools.

[0053] Step 4:

[0054] The server receives the operation results provided by the terminal and analyzes the response. Specifically, it evaluates response codes and error messages to identify security vulnerabilities.

[0055] Step 5:

[0056] The server generates a report for the user based on the analysis results, notifying them of the vulnerability details along with suggested improvements. Providing users with specific corrective measures enables a rapid response.

[0057] Step 6:

[0058] Users can set up regular diagnostic schedules on the server to perform continuous security monitoring. The server automatically repeats diagnostics based on the settings, performing security checks to address the latest threats.

[0059] (Example 1)

[0060] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0061] Current security management systems require significant human resources and time to identify and properly address vulnerabilities in websites and software. This leads to delays in vulnerability discovery and response, increasing security risks. Furthermore, continuous monitoring to prevent vulnerability recurrence is insufficient.

[0062] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0063] In this invention, the server includes computer executable means for identifying screen configurations and each element, calculation means for generating test items based on each identified element, and processing means for performing automated tests based on the generated test items. This makes it possible to quickly and efficiently detect vulnerabilities in an automated manner and take appropriate countermeasures.

[0064] "Screen layout" refers to the overall design and layout of a computer's user interface, including the information displayed and the elements that can be interacted with.

[0065] "Each element" refers to an individual component present on a user interface that provides a specific function or information, including buttons, input fields, labels, etc.

[0066] "Computer executable means" refers to a set of instructions or processes that a computer can execute through software or hardware, and includes logic for achieving a specific task.

[0067] "Computational means" refers to the functions of a computer used to process, analyze, and evaluate data, and the process of deriving specific results using algorithms and mathematical formulas.

[0068] A "processing unit" is a part of a system that performs specific functions based on received data or instructions, and is responsible for functions such as data input, data conversion, and output generation.

[0069] "Time management tools" refer to functions for executing tasks and processes based on a fixed schedule, and involve managing regular actions using timers and schedulers.

[0070] "Communication means" refers to functions for transmitting data and information to other systems or users, enabling two-way information exchange via networks and digital messaging systems.

[0071] This invention is a system that automatically detects and reports vulnerabilities in software and websites, primarily using a server and terminals. The server is responsible for analysis, generation of test items, analysis of test results, and report creation, while the terminals perform direct operations with the user interface.

[0072] The server uses state-of-the-art image processing and natural language processing technologies to analyze the screen interface information received from the terminal. This allows it to identify the screen configuration and its elements, and generate vulnerability test items based on this information. As part of this process, the server can use a generative AI model to prepare datasets necessary for simulating, for example, SQL injection or brute-force attacks.

[0073] The terminal follows instructions for test items provided by the server and uses automation tools to reproduce operations on the user interface. This makes it possible to discover vulnerabilities in a real operating environment. Specifically, the terminal uses scripts and browser automation tools to automate form input and button clicks.

[0074] The server evaluates data responses from terminals and identifies potential security issues. Based on the analysis, it provides the user with a detailed report and proposes specific corrective actions. This process includes analyzing normal error messages and abnormal behavior.

[0075] Furthermore, users can schedule regular diagnostics through the server. This feature allows for continuous system monitoring and the detection of the latest threats. For example, users can set up weekly tests to maintain system security.

[0076] An example of a prompt message is, "Please tell me how to perform an SQL injection vulnerability test on a company's website and suggest improvements." In this way, the present invention provides a comprehensive solution for efficient security management.

[0077] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0078] Step 1:

[0079] The server receives screen interface data from the terminal as input. It analyzes the screen's components using a combination of OCR and natural language processing technologies. From this analysis, it identifies buttons, input fields, text labels, and other elements on the screen, and extracts their attribute information. Through this process, the server outputs detailed information about each element that makes up the screen.

[0080] Step 2:

[0081] The server automatically generates vulnerability test items from the screen configuration information obtained in Step 1. In this process, an AI model is used to determine the test items based on historical data and known vulnerability patterns. Specifically, test items for SQL injection and cross-site scripting are generated. This allows the server to output test data corresponding to specific attack scenarios.

[0082] Step 3:

[0083] The terminal uses test items provided by the server as input and performs automated operations on the user interface. Using a browser automation tool, it replicates operations such as entering test data into a login form and clicking buttons. Through this process, the terminal outputs specific test results to the server.

[0084] Step 4:

[0085] The server analyzes the test results received from the terminal in step 3 as input. It analyzes error messages and abnormal behavior from the test result data response to identify vulnerabilities. This analysis evaluates the system's response to malicious input and outputs potential security risks.

[0086] Step 5:

[0087] The server generates a comprehensive report based on the analyzed vulnerability information and provides it to the user. The report includes details of the discovered vulnerabilities, their impact, and specific remediation measures. Users can receive this report and use it to improve their system security.

[0088] (Application Example 1)

[0089] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0090] In recent years, security vulnerabilities in information systems have become more diverse and sophisticated, requiring immediate and continuous monitoring and countermeasures. However, conventional methods have presented challenges, such as the need for manual operation and specialized knowledge for vulnerability detection and reporting, making real-time vulnerability scanning difficult. Therefore, there is a need for a system that can easily and effectively detect vulnerabilities in digital devices used daily by users and respond quickly.

[0091] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0092] In this invention, the server includes means for analyzing the screen interface and recognizing the screen configuration and each element; means for automatically generating vulnerability test items based on each element on the screen that has been recognized; means for performing attack simulations against the screen interface using the automatically generated test items; means for analyzing the test results and reporting the vulnerabilities discovered; means for scheduling continuous monitoring and retesting; means for performing real-time vulnerability scans and notifying the user of the results; and means for performing analysis using visual data from digital devices. As a result, users can instantly detect vulnerabilities on digital devices they use daily without requiring specialized knowledge and take rapid security measures.

[0093] A "screen interface" is a visual structure that allows users to directly interact with software and web services through a digital device.

[0094] A "vulnerability test item" is a set of test criteria designed to assess the security risks inherent in a system or application.

[0095] "Attack simulation" is a process that measures the security performance of a system by simulating actual cyberattacks.

[0096] "Test results" refer to data and information obtained after conducting vulnerability tests, and are indicators of the system's security status.

[0097] "Continuous monitoring" is a process aimed at constantly monitoring the security status of systems and networks and detecting signs of anomalies or vulnerabilities at an early stage.

[0098] "Means for scheduling retesting" refers to a function that sets the timing for automatically re-running vulnerability detection tests, either periodically or as needed.

[0099] "Real-time vulnerability scanning" is an immediate assessment method for instantly detecting security risks while a digital device is in operation.

[0100] "Visual data from digital devices" refers to image data that includes information displayed on the screen of devices such as smartphones and personal computers.

[0101] This invention aims to realize a system that detects and reports vulnerabilities in screen interfaces operating on digital devices in real time. The system mainly consists of a server and terminals, and easily provides security information to users.

[0102] The server analyzes the screen interface of the terminal being operated by the user, recognizing its structure and elements. Software such as OpenCV for image processing is used for this analysis. Based on this analysis, the server automatically generates vulnerability test items and performs attack simulations based on these items.

[0103] The device uses acquired visual data to scan for vulnerabilities in real time. The scan results are sent to a server, and recommended countermeasures are notified to the user as needed. This notification is delivered through the interface of a smartphone or smart glasses. This allows users to instantly check the security status of their device without requiring specialized knowledge.

[0104] The server features continuous monitoring capabilities and allows for scheduled periodic retesting. This ensures that the overall system security remains up-to-date at all times.

[0105] For example, when a user is browsing a website on a mobile device, the application immediately performs a security scan and issues a warning about potential vulnerabilities. Another example is when using a news app at night; the application might notify the user about a potential SQL injection attack in a test environment. In this way, users can quickly take the necessary actions.

[0106] An example of a prompt when using a generative AI model is: "How can we design a feature that detects vulnerabilities in real time while using a smartphone app and provides the user with specific suggestions for improvement?"

[0107] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0108] Step 1:

[0109] The device takes a screenshot of the screen interface based on user actions. This screenshot serves as input data, playing a role in collecting visual information. Specifically, the device's camera or screen capture function is automatically activated, and image data is collected after obtaining user consent.

[0110] Step 2:

[0111] The device sends the captured screenshot to the server. Here, the input data is the captured image, and the output data is transferred to the server. In this step, the device uses data communication to send the image data to the specified server address.

[0112] Step 3:

[0113] The server analyzes the received screenshot using an image processing algorithm to recognize the screen layout and each element. In this step, the screenshot is used as input data, and the recognition result is obtained as output data. Specifically, the server analyzes the image using libraries such as OpenCV to identify elements such as buttons, text fields, and links.

[0114] Step 4:

[0115] The server automatically generates vulnerability test items based on the screen elements it recognizes. Here, the input data is the recognition result, and the output data is the automatically generated test items. Specifically, the server uses the generated AI model to create vulnerability test cases for each screen element.

[0116] Step 5:

[0117] The server performs attack simulations using automatically generated test items. In this step, the test items become input data, and results for security evaluation are output. The server uses existing security tools to simulate attacks such as SQL injection and XSS attacks and detect vulnerabilities.

[0118] Step 6:

[0119] The server analyzes the results of attack simulations and generates a report summarizing the vulnerabilities it discovers. The input data is the simulation results, and the output data is the report. Specifically, the server creates a report based on the analysis results that details the discovered vulnerabilities and recommends appropriate improvements.

[0120] Step 7:

[0121] The server sends the report to the terminal and notifies the user. In this step, the generated report is the input data and the notification is the output. When the terminal receives the notification, it displays a pop-up on the screen, allowing the user to view the report.

[0122] Step 8:

[0123] The user receives a notification and reviews the report. Here, the input data is the report notification, and the user's understanding is the output. Specifically, the user taps the push notification on their device and can access the report details page.

[0124] Step 9:

[0125] The server performs continuous monitoring and periodic retesting based on a schedule specified by the user. The input data is the user-specified schedule, and the output data is the periodic test results. The server automatically executes retests according to the schedule and accumulates the results.

[0126] In this system, the generative AI model is used to generate test items and analyze simulation results. An example of a prompt is, "How can we design a feature that detects vulnerabilities in real time while using a smartphone app and provides the user with specific improvement suggestions?"

[0127] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0128] This invention provides a system that automatically detects vulnerabilities in software and websites and provides improvement suggestions and reports based on the user's emotional state. The system mainly consists of a server, terminals, and an emotion engine. The server is responsible for central control and data processing, while the terminals handle the user interface and direct interaction with the user. The emotion engine analyzes the user's emotions and dynamically adjusts the user interface and notification methods.

[0129] Server-based processes

[0130] The server recognizes the screen interface displayed on the terminal and analyzes its structure and each element. Based on this, the server automatically generates vulnerability test items and uses them to perform attack simulations. For example, it sets up data to test threats such as SQL injection and cross-site scripting (XSS). Upon receiving the test results, the server analyzes the response to identify vulnerabilities and records them.

[0131] Adjustment by the emotion engine

[0132] The emotion engine determines the user's emotions from factors such as facial expressions and tone of voice, and provides this information to the server. This emotion information is used to adjust the content of reports and how they are presented. For example, if a user is feeling stressed, the system can adjust to provide concise and positive feedback.

[0133] Automated operation and notifications for the device

[0134] The device performs tests based on test items by automatically inputting information into the user interface. During this process, the emotion engine ensures that the device acts in a way that considers the user's emotions when necessary. Furthermore, reports and improvement suggestions are provided to the user via the device, and their content is adjusted based on emotion data.

[0135] Continuous monitoring and optimization

[0136] Users can schedule diagnostics for the server and perform continuous vulnerability assessments. The server uses data from the sentiment engine to perform diagnostics and notifications at the optimal time, improving the user experience. This system enables flexible responses based on user emotions, achieving advanced security management and customized user notifications.

[0137] The following describes the processing flow.

[0138] Step 1:

[0139] The server analyzes the screen interface data received from the terminal to recognize the page structure and its elements. Specifically, it analyzes HTML and CSS and extracts attribute information for input fields and buttons.

[0140] Step 2:

[0141] The server automatically generates vulnerability test items based on the analyzed information. For example, it prepares test data for SQL injection and scripts for cross-site scripting.

[0142] Step 3:

[0143] The terminal automatically performs tests on the screen interface using test items provided by the server. Specifically, it uses a browser automation tool to input data into an input form and then performs a submit action.

[0144] Step 4:

[0145] The emotion engine analyzes the user's facial expressions and voice data to determine their current emotional state. For example, it collects emotional data in real time using a camera and microphone.

[0146] Step 5:

[0147] The server integrates test results from the terminals with sentiment information from the sentiment engine and performs an evaluation. Specifically, it analyzes the risk level of vulnerabilities detected in the tests and selects a reporting method that matches the user's sentiment.

[0148] Step 6:

[0149] Based on the analysis results, the server generates a report for the user and suggests improvement measures as needed. It takes emotional information into consideration and strives to provide feedback that reduces stress.

[0150] Step 7:

[0151] The user initiates continuous monitoring based on a regular diagnostic schedule set on the server. The server utilizes data from the emotion engine to select and execute the optimal diagnostic timing.

[0152] (Example 2)

[0153] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0154] Traditional security systems rely on static methods to detect vulnerabilities in websites and software, which means they cannot flexibly respond to user emotional states or usage patterns. Furthermore, the way in which improvement suggestions are not communicated and results are presented is not appropriately tailored to the user, hindering improvements in the user experience.

[0155] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0156] In this invention, the server includes means for analyzing the screen interface and recognizing the screen configuration and each element, means for automatically generating vulnerability test items, and means for analyzing the user's emotional state and dynamically adjusting the reporting and notification methods. This enables flexible responses in accordance with the user's emotional state, and improves the user experience through more personalized improvement suggestions and notifications.

[0157] "Screen interface" refers to the display and operation portion used for exchanging information between the user and the system.

[0158] "Screen layout" is a concept that refers to the overall structure of how elements within an interface are arranged and displayed.

[0159] An "element" refers to individual parts or components included within a screen interface, such as buttons or text fields.

[0160] "To recognize" means that a system has the ability to detect and understand the configuration and elements within a screen interface.

[0161] "Vulnerability test items" refer to specific tests set up to detect potential security holes or flaws in a screen interface.

[0162] "Automatic generation" refers to a process in which a system generates test items and results without human intervention.

[0163] "Attack simulation" refers to a testing method that simulates techniques for exploiting vulnerabilities in a virtual environment, thereby verifying the system's defensive capabilities.

[0164] "Test results" refer to the data and analysis obtained as a result of the attack simulations that were conducted.

[0165] "Reporting" refers to the process of notifying users of detected vulnerabilities and issues and providing information to help them understand them.

[0166] "Emotional state" refers to a user's current emotional response and feelings, and is used to develop approaches tailored to each user's individual situation.

[0167] "Dynamic adjustment" means automatically changing actions and displays to suit the situation based on the user's emotions and circumstances.

[0168] "Continuous monitoring" refers to a series of activities aimed at regularly and continuously monitoring vulnerabilities to detect new problems early.

[0169] "Schedule a retest" means making a plan to conduct security tests again, either periodically or at specific times.

[0170] This invention provides a system for effectively managing the security of software and websites. The system mainly consists of a server, terminals, and an emotion engine. The server is responsible for centrally analyzing data and performing tests. The terminals provide the user interface and receive user interaction and feedback. The emotion engine analyzes the user's emotional state and provides data for the entire system to be dynamically adjusted.

[0171] The server analyzes the website's screen interface, recognizing its structure and individual elements. This uses software libraries for parsing HTML documents. Based on the analyzed information, the server automatically generates vulnerability test items. These generated test items are used to perform attack simulations against specific security threats. This process allows for the efficient detection of vulnerabilities, such as those related to SQL injection and cross-site scripting (XSS).

[0172] The emotion engine analyzes the user's facial expressions and voice data to infer their emotional state. This allows the server to dynamically adjust the content and method of reports and notifications to the user. For example, if the user is stressed, the system will adjust to provide simpler, more positive feedback.

[0173] For example, if a user wants to perform regular security checks on their web application, the system would display the check results at a time convenient for the user and provide a report including solutions for any problems found. This allows the user to effectively implement security measures.

[0174] Examples of prompts for a generative AI model:

[0175] "A user has requested a security assessment of a specific web application. How does the server analyze the interface and generate vulnerability test items? Please provide specific processing steps."

[0176] This system flexibly responds to the latest security needs and provides more user-friendly security management.

[0177] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0178] Step 1:

[0179] The user requests a security scan of a website or software through the terminal and sets the desired diagnostic schedule. Input includes the target URL and diagnostic criteria. Based on this, the terminal sends the user's request to the server. The output is the diagnostic request data passed to the server.

[0180] Step 2:

[0181] The server parses the received diagnostic request data and obtains the HTML structure of the specified screen interface. This process uses a specific software library to analyze the screen interface and extract information about its components (e.g., input fields, buttons). The input is the received interface data, and the output is the structure data based on the analysis results.

[0182] Step 3:

[0183] The server automatically generates vulnerability test items based on the analyzed structural data. Specifically, it generates test data related to SQL injection and cross-site scripting (XSS) attacks. In this step, data calculations are performed to verify whether elements within the interface have potential vulnerabilities. The input is structural data, and the output is test item data.

[0184] Step 4:

[0185] The server uses the generated test item data to perform attack simulations against the target screen interface. For example, it sends malicious input data and observes the system's response. The input is the test item data, and the output is the simulation result data.

[0186] Step 5:

[0187] The server analyzes the simulation results data and identifies any vulnerabilities found. These results are then compiled into a report for the user. The input is the simulation results data, and the output is the vulnerability report data.

[0188] Step 6:

[0189] The emotion engine analyzes the user's facial expressions and voice using the device's camera and microphone to evaluate the user's emotional state. This information is sent to a server and used to adjust reporting and notification methods. The input is real-time audio and video data, and the output is emotional state data.

[0190] Step 7:

[0191] The device displays a personalized report to the user based on their emotional state data. For example, if the user is feeling anxious, it will display only the most important information concisely. The input consists of vulnerability report data and emotional state data, and the output is the content displayed to the user.

[0192] Step 8:

[0193] The user reviews the report and takes action based on the suggested improvements. Furthermore, the server plans the next test according to a recurring diagnostic schedule. The input is the final report, and the output is the schedule information for the next diagnostic.

[0194] (Application Example 2)

[0195] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0196] Traditional vulnerability detection systems can be stressful for users because they provide uniform notifications and reports without considering the user's emotional state. Furthermore, there is a lack of concrete support for non-technical users to fully understand and improve security issues. Solving these problems is essential.

[0197] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0198] In this invention, the server includes means for analyzing the screen interface and recognizing the screen configuration and each element, means for automatically generating vulnerability test items based on each recognized element on the screen, and means for analyzing the user's emotional state. This makes it possible to select the optimal notification method according to the user's emotions and to provide appropriate security improvement suggestions without causing stress.

[0199] A "screen interface" is the visual display portion that allows a user to interact with software or a website.

[0200] "To recognize" means to identify and understand the characteristics of an object or data.

[0201] "Vulnerability test items" are test items used to evaluate vulnerabilities hidden in software or websites.

[0202] An "attack simulation" is a method of evaluating the resistance to security vulnerabilities by intentionally performing operations that exploit them.

[0203] "Test results" refer to the data and findings obtained from vulnerability testing.

[0204] "Continuous monitoring" means monitoring the system at regular time intervals and tracking changes in its status.

[0205] "Schedule a retest" means determining a time for administering the test again after the initial examination.

[0206] "User emotional state" refers to the emotions a user experiences while using the product or service, and includes stress and a sense of security.

[0207] "Notification method" refers to the means or methods used to convey information to the user.

[0208] "Report content" refers to the contents of a document summarizing test results and analysis results.

[0209] The system that implements this application is a complex configuration including a server, terminal, and emotion engine. First, the server analyzes the screen interface and recognizes the screen configuration and each element. This automatically generates vulnerability test items based on each element. Using these automatically generated items, the server performs attack simulations against the screen interface and evaluates threats such as SQL injection and cross-site scripting (XSS).

[0210] The emotion engine analyzes the user's facial expressions and voice using the smartphone's built-in camera and microphone. The specific software used for analysis is Microsoft® Azure® Face API and Google® Cloud Speech-to-Text. Based on this emotion information, the server optimizes notifications and generates reports that reduce user stress.

[0211] The terminal is equipped with tools that perform automated input operations on screen elements and conducts inspections based on test items. This automated operation allows the system to perform continuous monitoring and optimization with minimal user intervention.

[0212] For example, small businesses can use this system to strengthen the security of their websites and reduce stress on employees during work hours. For instance, if an SQL injection vulnerability is found, they can receive a notification message such as, "To ensure the security of your site, please try the following steps:"

[0213] An example of an input prompt for the generating AI model is: "Write an example of positive and concise feedback that should be sent to a calm user after detecting an SQL injection vulnerability on their site."

[0214] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0215] Step 1:

[0216] The server scans the screen interface and recognizes the displayed screen configuration and each element. It receives the screen data as input and converts it into a format that stores the characteristics of the elements in a database. This prepares the foundational data necessary for generating subsequent vulnerability test items.

[0217] Step 2:

[0218] The server automatically generates vulnerability test items based on each element of the recognized screen. Using the element data obtained in the previous step as input, it outputs test scenarios for SQL injection and cross-site scripting. The generated test items are optimized based on a standard vulnerability checklist.

[0219] Step 3:

[0220] The server performs attack simulations using the generated test items. It obtains the test items as input and response data to the screen interface as output. This allows for the identification of actual security vulnerabilities. Detailed execution logs are recorded during this step.

[0221] Step 4:

[0222] The terminal automates input operations to screen elements using an automated input tool. Using the test item data generated in step 3 as input, it reproduces the simulated behavior on the user interface. As output, the system's response to each operation is collected.

[0223] Step 5:

[0224] The server uses an emotion engine to analyze the user's emotional state. It takes facial and audio data obtained from the camera and microphone as input and outputs emotional states such as stress and reassurance. Based on this, the optimal method for maximizing the effectiveness of notifications is devised.

[0225] Step 6:

[0226] The server generates vulnerability reports and improvement suggestions based on test results and the user's emotional state, and sends them to the terminal. It combines emotional information and test results as input, outputting concise feedback that is easy for the user to understand. A specific action plan is also provided at this point.

[0227] Step 7:

[0228] Users review the feedback provided through their devices and implement necessary improvements. At this stage, they follow specific instructions from the system and implement security enhancement procedures.

[0229] Step 8:

[0230] The server continuously monitors the system even after the remediation process is complete, checking for any new vulnerabilities. It receives the latest interface data as input and continuously generates new test items as output. This ensures that the system's security is always up-to-date.

[0231] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0232] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0233] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0234] [Second Embodiment]

[0235] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0236] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0237] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0238] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0239] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0240] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0241] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0242] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0243] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0244] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0245] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0246] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0247] This invention provides a system for automatically detecting and efficiently reporting vulnerabilities in software and websites. The entire system mainly consists of a server and terminals, with the server handling overall control and data processing, and the terminals performing direct operations with the user interface.

[0248] Server-based analysis and test item generation

[0249] The server first analyzes the screen interface displayed on the terminal, identifying its structure and each element. Using the information obtained from this analysis, the server automatically generates vulnerability test items. For example, if a login page is detected, the server prepares test data for simulating SQL injection and brute-force attacks.

[0250] Vulnerability detection and testing

[0251] The terminal automatically performs operations based on the generated test items. The server receives these test results and analyzes the data responses returned by each operation. For example, if the terminal enters a specific string into an input form, the server determines whether or not there is a vulnerability from the resulting error or warning messages.

[0252] Reporting of results and suggestions for improvement

[0253] The server analyzes the detected vulnerabilities and creates a detailed report for the user based on the results. The report includes the issues found, a risk assessment, and specific improvement suggestions. For example, if an input field is determined to be vulnerable to SQL injection, the server will suggest to the user the implementation of prepared statements.

[0254] Continuous monitoring and scheduling

[0255] Users can schedule regular diagnostics for their servers. Based on this setting, the servers can automatically retest at specified times, constantly monitoring for the latest threats. For example, by scheduling weekly system-wide tests, it's possible to maintain security even for sites that require frequent updates.

[0256] Thus, this system provides detailed and automated vulnerability detection and reporting capabilities, contributing to the mitigation of security risks. User intervention is minimized, enabling efficient and effective security management.

[0257] The following describes the processing flow.

[0258] Step 1:

[0259] The server scans the screen interface displayed on the terminal to recognize the page structure and each element. Specifically, it parses the DOM (Document Object Model) and extracts attributes of input fields and buttons.

[0260] Step 2:

[0261] The server automatically generates vulnerability test items based on each element of the recognized interface. For example, it builds the test data and scenarios necessary to detect SQL injection and cross-site scripting (XSS).

[0262] Step 3:

[0263] The device uses the generated test items to perform automated actions on the user interface. Specifically, it enters attack strings into forms and executes the submit action. This is done using browser automation tools.

[0264] Step 4:

[0265] The server receives the operation results provided by the terminal and analyzes the response. Specifically, it evaluates response codes and error messages to identify security vulnerabilities.

[0266] Step 5:

[0267] The server generates a report for the user based on the analysis results, notifying them of the vulnerability details along with suggested improvements. Providing users with specific corrective measures enables a rapid response.

[0268] Step 6:

[0269] Users can set up regular diagnostic schedules on the server to perform continuous security monitoring. The server automatically repeats diagnostics based on the settings, performing security checks to address the latest threats.

[0270] (Example 1)

[0271] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0272] Current security management systems require significant human resources and time to identify and properly address vulnerabilities in websites and software. This leads to delays in vulnerability discovery and response, increasing security risks. Furthermore, continuous monitoring to prevent vulnerability recurrence is insufficient.

[0273] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0274] In this invention, the server includes computer executable means for identifying screen configurations and each element, calculation means for generating test items based on each identified element, and processing means for performing automated tests based on the generated test items. This makes it possible to quickly and efficiently detect vulnerabilities in an automated manner and take appropriate countermeasures.

[0275] "Screen layout" refers to the overall design and layout of a computer's user interface, including the information displayed and the elements that can be interacted with.

[0276] "Each element" refers to an individual component present on a user interface that provides a specific function or information, including buttons, input fields, labels, etc.

[0277] "Computer executable means" refers to a set of instructions or processes that a computer can execute through software or hardware, and includes logic for achieving a specific task.

[0278] "Computational means" refers to the functions of a computer used to process, analyze, and evaluate data, and the process of deriving specific results using algorithms and mathematical formulas.

[0279] A "processing unit" is a part of a system that performs specific functions based on received data or instructions, and is responsible for functions such as data input, data conversion, and output generation.

[0280] "Time management tools" refer to functions for executing tasks and processes based on a fixed schedule, and involve managing regular actions using timers and schedulers.

[0281] "Communication means" refers to functions for transmitting data and information to other systems or users, enabling two-way information exchange via networks and digital messaging systems.

[0282] This invention is a system that automatically detects and reports vulnerabilities in software and websites, primarily using a server and terminals. The server is responsible for analysis, generation of test items, analysis of test results, and report creation, while the terminals perform direct operations with the user interface.

[0283] The server uses the latest image processing technology and natural language processing technology to analyze the information of the screen interface received from the terminal. Thereby, it identifies the screen configuration and each element, and generates vulnerability test items based on this. The server can use the generated AI model as part of this process, for example, to prepare the dataset required for simulating SQL injection and brute-force attacks.

[0284] The terminal follows the instructions of the test items provided by the server and uses an automation tool to reproduce the operations on the user interface. Thereby, it becomes possible to discover vulnerabilities in the actual operating environment. Specifically, the terminal uses scripts or browser automation tools to automate form input and button clicks.

[0285] The server evaluates the data response from the terminal and identifies potential security problems. Based on the analysis results, it reports a detailed report to the user and proposes specific improvement measures. This process includes analyzing normal error messages and abnormal operations.

[0286] Furthermore, the user can set the scheduling of regular diagnosis through the server. With this function, it becomes possible to continuously monitor the system and discover the latest threats. For example, the user can set a weekly test to maintain the security of the system.

[0287] An example of the prompt text is "Please teach me how to perform a vulnerability check for SQL injection on a company's website and propose improvement points." Thus, the present invention provides a comprehensive solution for efficient security management.

[0288] The flow of the specific process in Example 1 will be described using FIG. 11.

[0289] Step 1:

[0290] The server receives screen interface data from the terminal as input. It analyzes the screen's components using a combination of OCR and natural language processing technologies. From this analysis, it identifies buttons, input fields, text labels, and other elements on the screen, and extracts their attribute information. Through this process, the server outputs detailed information about each element that makes up the screen.

[0291] Step 2:

[0292] The server automatically generates vulnerability test items from the screen configuration information obtained in Step 1. In this process, an AI model is used to determine the test items based on historical data and known vulnerability patterns. Specifically, test items for SQL injection and cross-site scripting are generated. This allows the server to output test data corresponding to specific attack scenarios.

[0293] Step 3:

[0294] The terminal uses test items provided by the server as input and performs automated operations on the user interface. Using a browser automation tool, it replicates operations such as entering test data into a login form and clicking buttons. Through this process, the terminal outputs specific test results to the server.

[0295] Step 4:

[0296] The server analyzes the test results received from the terminal in step 3 as input. It analyzes error messages and abnormal behavior from the test result data response to identify vulnerabilities. This analysis evaluates the system's response to malicious input and outputs potential security risks.

[0297] Step 5:

[0298] The server generates a comprehensive report based on the analyzed vulnerability information and provides it to the user. The report includes details of the discovered vulnerabilities, their impact, and specific remediation measures. Users can receive this report and use it to improve their system security.

[0299] (Application Example 1)

[0300] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0301] In recent years, security vulnerabilities in information systems have become more diverse and sophisticated, requiring immediate and continuous monitoring and countermeasures. However, conventional methods have presented challenges, such as the need for manual operation and specialized knowledge for vulnerability detection and reporting, making real-time vulnerability scanning difficult. Therefore, there is a need for a system that can easily and effectively detect vulnerabilities in digital devices used daily by users and respond quickly.

[0302] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0303] In this invention, the server includes means for analyzing the screen interface and recognizing the screen configuration and each element; means for automatically generating vulnerability test items based on each element on the screen that has been recognized; means for performing attack simulations against the screen interface using the automatically generated test items; means for analyzing the test results and reporting the vulnerabilities discovered; means for scheduling continuous monitoring and retesting; means for performing real-time vulnerability scans and notifying the user of the results; and means for performing analysis using visual data from digital devices. As a result, users can instantly detect vulnerabilities on digital devices they use daily without requiring specialized knowledge and take rapid security measures.

[0304] The "screen interface" is a visual structure for users to directly interact with software and web services through digital devices.

[0305] The "vulnerability test items" are a series of test criteria designed to evaluate potential security risks in a system or application.

[0306] "Attack simulation" is a process of measuring the security performance of a system by mimicking actual cyberattacks.

[0307] The "test results" are the data and information obtained after conducting vulnerability tests and are indicators showing the security status of the system.

[0308] "Continuous monitoring" is a process aimed at constantly monitoring the security status of a system or network and detecting signs of anomalies and vulnerabilities at an early stage.

[0309] The "means of scheduling retesting" is a function for setting the timing to automatically re - execute tests for vulnerability detection regularly or as needed.

[0310] "Real - time vulnerability scan" is an immediate evaluation method for detecting security risks instantly while a digital device is operating.

[0311] The "visual data of digital devices" is image data including information displayed on the screens of devices such as smartphones and personal computers.

[0312] This invention is for realizing a system that detects and reports the vulnerabilities of a screen interface operating on a digital device in real - time. The system is mainly composed of a server and a terminal, and provides security information to users easily.

[0313] The server analyzes the screen interface of the terminal being operated by the user, recognizing its structure and elements. Software such as OpenCV for image processing is used for this analysis. Based on this analysis, the server automatically generates vulnerability test items and performs attack simulations based on these items.

[0314] The device uses acquired visual data to scan for vulnerabilities in real time. The scan results are sent to a server, and recommended countermeasures are notified to the user as needed. This notification is delivered through the interface of a smartphone or smart glasses. This allows users to instantly check the security status of their device without requiring specialized knowledge.

[0315] The server features continuous monitoring capabilities and allows for scheduled periodic retesting. This ensures that the overall system security remains up-to-date at all times.

[0316] For example, when a user is browsing a website on a mobile device, the application immediately performs a security scan and issues a warning about potential vulnerabilities. Another example is when using a news app at night; the application might notify the user about a potential SQL injection attack in a test environment. In this way, users can quickly take the necessary actions.

[0317] An example of a prompt when using a generative AI model is: "How can we design a feature that detects vulnerabilities in real time while using a smartphone app and provides the user with specific suggestions for improvement?"

[0318] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0319] Step 1:

[0320] The device takes a screenshot of the screen interface based on user actions. This screenshot serves as input data, playing a role in collecting visual information. Specifically, the device's camera or screen capture function is automatically activated, and image data is collected after obtaining user consent.

[0321] Step 2:

[0322] The device sends the captured screenshot to the server. Here, the input data is the captured image, and the output data is transferred to the server. In this step, the device uses data communication to send the image data to the specified server address.

[0323] Step 3:

[0324] The server analyzes the received screenshot using an image processing algorithm to recognize the screen layout and each element. In this step, the screenshot is used as input data, and the recognition result is obtained as output data. Specifically, the server analyzes the image using libraries such as OpenCV to identify elements such as buttons, text fields, and links.

[0325] Step 4:

[0326] The server automatically generates vulnerability test items based on the screen elements it recognizes. Here, the input data is the recognition result, and the output data is the automatically generated test items. Specifically, the server uses the generated AI model to create vulnerability test cases for each screen element.

[0327] Step 5:

[0328] The server performs attack simulations using automatically generated test items. In this step, the test items become input data, and results for security evaluation are output. The server uses existing security tools to simulate attacks such as SQL injection and XSS attacks and detect vulnerabilities.

[0329] Step 6:

[0330] The server analyzes the results of attack simulations and generates a report summarizing the vulnerabilities it discovers. The input data is the simulation results, and the output data is the report. Specifically, the server creates a report based on the analysis results that details the discovered vulnerabilities and recommends appropriate improvements.

[0331] Step 7:

[0332] The server sends the report to the terminal and notifies the user. In this step, the generated report is the input data and the notification is the output. When the terminal receives the notification, it displays a pop-up on the screen, allowing the user to view the report.

[0333] Step 8:

[0334] The user receives a notification and reviews the report. Here, the input data is the report notification, and the user's understanding is the output. Specifically, the user taps the push notification on their device and can access the report details page.

[0335] Step 9:

[0336] The server performs continuous monitoring and periodic retesting based on a schedule specified by the user. The input data is the user-specified schedule, and the output data is the periodic test results. The server automatically executes retests according to the schedule and accumulates the results.

[0337] In this system, the generative AI model is used to generate test items and analyze simulation results. An example of a prompt is, "How can we design a feature that detects vulnerabilities in real time while using a smartphone app and provides the user with specific improvement suggestions?"

[0338] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0339] This invention provides a system that automatically detects vulnerabilities in software and websites and provides improvement suggestions and reports based on the user's emotional state. The system mainly consists of a server, terminals, and an emotion engine. The server is responsible for central control and data processing, while the terminals handle the user interface and direct interaction with the user. The emotion engine analyzes the user's emotions and dynamically adjusts the user interface and notification methods.

[0340] Server-based processes

[0341] The server recognizes the screen interface displayed on the terminal and analyzes its structure and each element. Based on this, the server automatically generates vulnerability test items and uses them to perform attack simulations. For example, it sets up data to test threats such as SQL injection and cross-site scripting (XSS). Upon receiving the test results, the server analyzes the response to identify vulnerabilities and records them.

[0342] Adjustment by the emotion engine

[0343] The emotion engine determines the user's emotions from factors such as facial expressions and tone of voice, and provides this information to the server. This emotion information is used to adjust the content of reports and how they are presented. For example, if a user is feeling stressed, the system can adjust to provide concise and positive feedback.

[0344] Automated operation and notifications for the device

[0345] The device performs tests based on test items by automatically inputting information into the user interface. During this process, the emotion engine ensures that the device acts in a way that considers the user's emotions when necessary. Furthermore, reports and improvement suggestions are provided to the user via the device, and their content is adjusted based on emotion data.

[0346] Continuous monitoring and optimization

[0347] Users can schedule diagnostics for the server and perform continuous vulnerability assessments. The server uses data from the sentiment engine to perform diagnostics and notifications at the optimal time, improving the user experience. This system enables flexible responses based on user emotions, achieving advanced security management and customized user notifications.

[0348] The following describes the processing flow.

[0349] Step 1:

[0350] The server analyzes the screen interface data received from the terminal to recognize the page structure and its elements. Specifically, it analyzes HTML and CSS and extracts attribute information for input fields and buttons.

[0351] Step 2:

[0352] The server automatically generates vulnerability test items based on the analyzed information. For example, it prepares test data for SQL injection and scripts for cross-site scripting.

[0353] Step 3:

[0354] The terminal automatically performs tests on the screen interface using test items provided by the server. Specifically, it uses a browser automation tool to input data into an input form and then performs a submit action.

[0355] Step 4:

[0356] The emotion engine analyzes the user's facial expressions and voice data to determine their current emotional state. For example, it collects emotional data in real time using a camera and microphone.

[0357] Step 5:

[0358] The server integrates test results from the terminals with sentiment information from the sentiment engine and performs an evaluation. Specifically, it analyzes the risk level of vulnerabilities detected in the tests and selects a reporting method that matches the user's sentiment.

[0359] Step 6:

[0360] Based on the analysis results, the server generates a report for the user and suggests improvement measures as needed. It takes emotional information into consideration and strives to provide feedback that reduces stress.

[0361] Step 7:

[0362] The user initiates continuous monitoring based on a regular diagnostic schedule set on the server. The server utilizes data from the emotion engine to select and execute the optimal diagnostic timing.

[0363] (Example 2)

[0364] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0365] Traditional security systems rely on static methods to detect vulnerabilities in websites and software, which means they cannot flexibly respond to user emotional states or usage patterns. Furthermore, the way in which improvement suggestions are not communicated and results are presented is not appropriately tailored to the user, hindering improvements in the user experience.

[0366] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0367] In this invention, the server includes means for analyzing the screen interface and recognizing the screen configuration and each element, means for automatically generating vulnerability test items, and means for analyzing the user's emotional state and dynamically adjusting the reporting and notification methods. This enables flexible responses in accordance with the user's emotional state, and improves the user experience through more personalized improvement suggestions and notifications.

[0368] "Screen interface" refers to the display and operation portion used for exchanging information between the user and the system.

[0369] "Screen layout" is a concept that refers to the overall structure of how elements within an interface are arranged and displayed.

[0370] An "element" refers to individual parts or components included within a screen interface, such as buttons or text fields.

[0371] "To recognize" means that a system has the ability to detect and understand the configuration and elements within a screen interface.

[0372] "Vulnerability test items" refer to specific tests set up to detect potential security holes or flaws in a screen interface.

[0373] "Automatic generation" refers to a process in which a system generates test items and results without human intervention.

[0374] "Attack simulation" refers to a testing method that simulates techniques for exploiting vulnerabilities in a virtual environment, thereby verifying the system's defensive capabilities.

[0375] "Test results" refer to the data and analysis obtained as a result of the attack simulations that were conducted.

[0376] "Reporting" refers to the process of notifying users of detected vulnerabilities and issues and providing information to help them understand them.

[0377] "Emotional state" refers to a user's current emotional response and feelings, and is used to develop approaches tailored to each user's individual situation.

[0378] "Dynamic adjustment" means automatically changing actions and displays to suit the situation based on the user's emotions and circumstances.

[0379] "Continuous monitoring" refers to a series of activities aimed at regularly and continuously monitoring vulnerabilities to detect new problems early.

[0380] "Schedule a retest" means making a plan to conduct security tests again, either periodically or at specific times.

[0381] This invention provides a system for effectively managing the security of software and websites. The system mainly consists of a server, terminals, and an emotion engine. The server is responsible for centrally analyzing data and performing tests. The terminals provide the user interface and receive user interaction and feedback. The emotion engine analyzes the user's emotional state and provides data for the entire system to be dynamically adjusted.

[0382] The server analyzes the website's screen interface, recognizing its structure and individual elements. This uses software libraries for parsing HTML documents. Based on the analyzed information, the server automatically generates vulnerability test items. These generated test items are used to perform attack simulations against specific security threats. This process allows for the efficient detection of vulnerabilities, such as those related to SQL injection and cross-site scripting (XSS).

[0383] The emotion engine analyzes the user's facial expressions and voice data to infer their emotional state. This allows the server to dynamically adjust the content and method of reports and notifications to the user. For example, if the user is stressed, the system will adjust to provide simpler, more positive feedback.

[0384] For example, if a user wants to perform regular security checks on their web application, the system would display the check results at a time convenient for the user and provide a report including solutions for any problems found. This allows the user to effectively implement security measures.

[0385] Examples of prompts for a generative AI model:

[0386] "A user has requested a security assessment of a specific web application. How does the server analyze the interface and generate vulnerability test items? Please provide specific processing steps."

[0387] This system flexibly responds to the latest security needs and provides more user-friendly security management.

[0388] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0389] Step 1:

[0390] The user requests a security scan of a website or software through the terminal and sets the desired diagnostic schedule. Input includes the target URL and diagnostic criteria. Based on this, the terminal sends the user's request to the server. The output is the diagnostic request data passed to the server.

[0391] Step 2:

[0392] The server parses the received diagnostic request data and obtains the HTML structure of the specified screen interface. This process uses a specific software library to analyze the screen interface and extract information about its components (e.g., input fields, buttons). The input is the received interface data, and the output is the structure data based on the analysis results.

[0393] Step 3:

[0394] The server automatically generates vulnerability test items based on the analyzed structural data. Specifically, it generates test data related to SQL injection and cross-site scripting (XSS) attacks. In this step, data calculations are performed to verify whether elements within the interface have potential vulnerabilities. The input is structural data, and the output is test item data.

[0395] Step 4:

[0396] The server uses the generated test item data to perform attack simulations against the target screen interface. For example, it sends malicious input data and observes the system's response. The input is the test item data, and the output is the simulation result data.

[0397] Step 5:

[0398] The server analyzes the simulation results data and identifies any vulnerabilities found. These results are then compiled into a report for the user. The input is the simulation results data, and the output is the vulnerability report data.

[0399] Step 6:

[0400] The emotion engine analyzes the user's facial expressions and voice using the device's camera and microphone to evaluate the user's emotional state. This information is sent to a server and used to adjust reporting and notification methods. The input is real-time audio and video data, and the output is emotional state data.

[0401] Step 7:

[0402] The device displays a personalized report to the user based on their emotional state data. For example, if the user is feeling anxious, it will display only the most important information concisely. The input consists of vulnerability report data and emotional state data, and the output is the content displayed to the user.

[0403] Step 8:

[0404] The user reviews the report and takes action based on the suggested improvements. Furthermore, the server plans the next test according to a recurring diagnostic schedule. The input is the final report, and the output is the schedule information for the next diagnostic.

[0405] (Application Example 2)

[0406] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0407] Traditional vulnerability detection systems can be stressful for users because they provide uniform notifications and reports without considering the user's emotional state. Furthermore, there is a lack of concrete support for non-technical users to fully understand and improve security issues. Solving these problems is essential.

[0408] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0409] In this invention, the server includes means for analyzing the screen interface and recognizing the screen configuration and each element, means for automatically generating vulnerability test items based on each recognized element on the screen, and means for analyzing the user's emotional state. This makes it possible to select the optimal notification method according to the user's emotions and to provide appropriate security improvement suggestions without causing stress.

[0410] A "screen interface" is the visual display portion that allows a user to interact with software or a website.

[0411] "To recognize" means to identify and understand the characteristics of an object or data.

[0412] "Vulnerability test items" are test items used to evaluate vulnerabilities hidden in software or websites.

[0413] An "attack simulation" is a method of evaluating the resistance to security vulnerabilities by intentionally performing operations that exploit them.

[0414] "Test results" refer to the data and findings obtained from vulnerability testing.

[0415] "Continuous monitoring" means monitoring the system at regular time intervals and tracking changes in its status.

[0416] "Schedule a retest" means determining a time for administering the test again after the initial examination.

[0417] "User emotional state" refers to the emotions a user experiences while using the product or service, and includes stress and a sense of security.

[0418] "Notification method" refers to the means or methods used to convey information to the user.

[0419] "Report content" refers to the contents of a document summarizing test results and analysis results.

[0420] The system that implements this application is a complex configuration including a server, terminal, and emotion engine. First, the server analyzes the screen interface and recognizes the screen configuration and each element. This automatically generates vulnerability test items based on each element. Using these automatically generated items, the server performs attack simulations against the screen interface and evaluates threats such as SQL injection and cross-site scripting (XSS).

[0421] The emotion engine analyzes the user's facial expressions and voice using the smartphone's built-in camera and microphone. The specific software used for analysis includes Microsoft Azure Face API and Google Cloud Speech-to-Text. Based on this emotion information, the server optimizes notifications and generates reports designed to reduce user stress.

[0422] The terminal is equipped with tools that perform automated input operations on screen elements and conducts inspections based on test items. This automated operation allows the system to perform continuous monitoring and optimization with minimal user intervention.

[0423] For example, small businesses can use this system to strengthen the security of their websites and reduce stress on employees during work hours. For instance, if an SQL injection vulnerability is found, they can receive a notification message such as, "To ensure the security of your site, please try the following steps:"

[0424] An example of an input prompt for the generating AI model is: "Write an example of positive and concise feedback that should be sent to a calm user after detecting an SQL injection vulnerability on their site."

[0425] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0426] Step 1:

[0427] The server scans the screen interface and recognizes the displayed screen configuration and each element. It receives the screen data as input and converts it into a format that stores the characteristics of the elements in a database. This prepares the foundational data necessary for generating subsequent vulnerability test items.

[0428] Step 2:

[0429] The server automatically generates vulnerability test items based on each element of the recognized screen. Using the element data obtained in the previous step as input, it outputs test scenarios for SQL injection and cross-site scripting. The generated test items are optimized based on a standard vulnerability checklist.

[0430] Step 3:

[0431] The server performs attack simulations using the generated test items. It obtains the test items as input and response data to the screen interface as output. This allows for the identification of actual security vulnerabilities. Detailed execution logs are recorded during this step.

[0432] Step 4:

[0433] The terminal automates input operations to screen elements using an automated input tool. Using the test item data generated in step 3 as input, it reproduces the simulated behavior on the user interface. As output, the system's response to each operation is collected.

[0434] Step 5:

[0435] The server uses an emotion engine to analyze the user's emotional state. It takes facial and audio data obtained from the camera and microphone as input and outputs emotional states such as stress and reassurance. Based on this, the optimal method for maximizing the effectiveness of notifications is devised.

[0436] Step 6:

[0437] The server generates vulnerability reports and improvement suggestions based on test results and the user's emotional state, and sends them to the terminal. It combines emotional information and test results as input, outputting concise feedback that is easy for the user to understand. A specific action plan is also provided at this point.

[0438] Step 7:

[0439] Users review the feedback provided through their devices and implement necessary improvements. At this stage, they follow specific instructions from the system and implement security enhancement procedures.

[0440] Step 8:

[0441] The server continuously monitors the system even after the remediation process is complete, checking for any new vulnerabilities. It receives the latest interface data as input and continuously generates new test items as output. This ensures that the system's security is always up-to-date.

[0442] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0443] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0444] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0445] [Third Embodiment]

[0446] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0447] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0448] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0449] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0450] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0451] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0452] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0453] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0454] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0455] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0456] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0457] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0458] This invention provides a system for automatically detecting and efficiently reporting vulnerabilities in software and websites. The entire system mainly consists of a server and terminals, with the server handling overall control and data processing, and the terminals performing direct operations with the user interface.

[0459] Server-based analysis and test item generation

[0460] The server first analyzes the screen interface displayed on the terminal, identifying its structure and each element. Using the information obtained from this analysis, the server automatically generates vulnerability test items. For example, if a login page is detected, the server prepares test data for simulating SQL injection and brute-force attacks.

[0461] Vulnerability detection and testing

[0462] The terminal automatically performs operations based on the generated test items. The server receives these test results and analyzes the data responses returned by each operation. For example, if the terminal enters a specific string into an input form, the server determines whether or not there is a vulnerability from the resulting error or warning messages.

[0463] Reporting of results and suggestions for improvement

[0464] The server analyzes the detected vulnerabilities and creates a detailed report for the user based on the results. The report includes the issues found, a risk assessment, and specific improvement suggestions. For example, if an input field is determined to be vulnerable to SQL injection, the server will suggest to the user the implementation of prepared statements.

[0465] Continuous monitoring and scheduling

[0466] Users can schedule regular diagnostics for their servers. Based on this setting, the servers can automatically retest at specified times, constantly monitoring for the latest threats. For example, by scheduling weekly system-wide tests, it's possible to maintain security even for sites that require frequent updates.

[0467] Thus, this system provides detailed and automated vulnerability detection and reporting capabilities, contributing to the mitigation of security risks. User intervention is minimized, enabling efficient and effective security management.

[0468] The following describes the processing flow.

[0469] Step 1:

[0470] The server scans the screen interface displayed on the terminal to recognize the page structure and each element. Specifically, it parses the DOM (Document Object Model) and extracts attributes of input fields and buttons.

[0471] Step 2:

[0472] The server automatically generates vulnerability test items based on each element of the recognized interface. For example, it builds the test data and scenarios necessary to detect SQL injection and cross-site scripting (XSS).

[0473] Step 3:

[0474] The device uses the generated test items to perform automated actions on the user interface. Specifically, it enters attack strings into forms and executes the submit action. This is done using browser automation tools.

[0475] Step 4:

[0476] The server receives the operation results provided by the terminal and analyzes the response. Specifically, it evaluates response codes and error messages to identify security vulnerabilities.

[0477] Step 5:

[0478] The server generates a report for the user based on the analysis results, notifying them of the vulnerability details along with suggested improvements. Providing users with specific corrective measures enables a rapid response.

[0479] Step 6:

[0480] Users can set up regular diagnostic schedules on the server to perform continuous security monitoring. The server automatically repeats diagnostics based on the settings, performing security checks to address the latest threats.

[0481] (Example 1)

[0482] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0483] Current security management systems require significant human resources and time to identify and properly address vulnerabilities in websites and software. This leads to delays in vulnerability discovery and response, increasing security risks. Furthermore, continuous monitoring to prevent vulnerability recurrence is insufficient.

[0484] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0485] In this invention, the server includes computer executable means for identifying screen configurations and each element, calculation means for generating test items based on each identified element, and processing means for performing automated tests based on the generated test items. This makes it possible to quickly and efficiently detect vulnerabilities in an automated manner and take appropriate countermeasures.

[0486] "Screen layout" refers to the overall design and layout of a computer's user interface, including the information displayed and the elements that can be interacted with.

[0487] "Each element" refers to an individual component present on a user interface that provides a specific function or information, including buttons, input fields, labels, etc.

[0488] "Computer executable means" refers to a set of instructions or processes that a computer can execute through software or hardware, and includes logic for achieving a specific task.

[0489] "Computational means" refers to the functions of a computer used to process, analyze, and evaluate data, and the process of deriving specific results using algorithms and mathematical formulas.

[0490] A "processing unit" is a part of a system that performs specific functions based on received data or instructions, and is responsible for functions such as data input, data conversion, and output generation.

[0491] "Time management tools" refer to functions for executing tasks and processes based on a fixed schedule, and involve managing regular actions using timers and schedulers.

[0492] "Communication means" refers to functions for transmitting data and information to other systems or users, enabling two-way information exchange via networks and digital messaging systems.

[0493] This invention is a system that automatically detects and reports vulnerabilities in software and websites, primarily using a server and terminals. The server is responsible for analysis, generation of test items, analysis of test results, and report creation, while the terminals perform direct operations with the user interface.

[0494] The server uses state-of-the-art image processing and natural language processing technologies to analyze the screen interface information received from the terminal. This allows it to identify the screen configuration and its elements, and generate vulnerability test items based on this information. As part of this process, the server can use a generative AI model to prepare datasets necessary for simulating, for example, SQL injection or brute-force attacks.

[0495] The terminal follows instructions for test items provided by the server and uses automation tools to reproduce operations on the user interface. This makes it possible to discover vulnerabilities in a real operating environment. Specifically, the terminal uses scripts and browser automation tools to automate form input and button clicks.

[0496] The server evaluates data responses from terminals and identifies potential security issues. Based on the analysis, it provides the user with a detailed report and proposes specific corrective actions. This process includes analyzing normal error messages and abnormal behavior.

[0497] Furthermore, users can schedule regular diagnostics through the server. This feature allows for continuous system monitoring and the detection of the latest threats. For example, users can set up weekly tests to maintain system security.

[0498] An example of a prompt message is, "Please tell me how to perform an SQL injection vulnerability test on a company's website and suggest improvements." In this way, the present invention provides a comprehensive solution for efficient security management.

[0499] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0500] Step 1:

[0501] The server receives screen interface data from the terminal as input. It analyzes the screen's components using a combination of OCR and natural language processing technologies. From this analysis, it identifies buttons, input fields, text labels, and other elements on the screen, and extracts their attribute information. Through this process, the server outputs detailed information about each element that makes up the screen.

[0502] Step 2:

[0503] The server automatically generates vulnerability test items from the screen configuration information obtained in Step 1. In this process, an AI model is used to determine the test items based on historical data and known vulnerability patterns. Specifically, test items for SQL injection and cross-site scripting are generated. This allows the server to output test data corresponding to specific attack scenarios.

[0504] Step 3:

[0505] The terminal uses test items provided by the server as input and performs automated operations on the user interface. Using a browser automation tool, it replicates operations such as entering test data into a login form and clicking buttons. Through this process, the terminal outputs specific test results to the server.

[0506] Step 4:

[0507] The server analyzes the test results received from the terminal in step 3 as input. It analyzes error messages and abnormal behavior from the test result data response to identify vulnerabilities. This analysis evaluates the system's response to malicious input and outputs potential security risks.

[0508] Step 5:

[0509] The server generates a comprehensive report based on the analyzed vulnerability information and provides it to the user. The report includes details of the discovered vulnerabilities, their impact, and specific remediation measures. Users can receive this report and use it to improve their system security.

[0510] (Application Example 1)

[0511] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0512] In recent years, security vulnerabilities in information systems have become more diverse and sophisticated, requiring immediate and continuous monitoring and countermeasures. However, conventional methods have presented challenges, such as the need for manual operation and specialized knowledge for vulnerability detection and reporting, making real-time vulnerability scanning difficult. Therefore, there is a need for a system that can easily and effectively detect vulnerabilities in digital devices used daily by users and respond quickly.

[0513] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0514] In this invention, the server includes means for analyzing the screen interface and recognizing the screen configuration and each element; means for automatically generating vulnerability test items based on each element on the screen that has been recognized; means for performing attack simulations against the screen interface using the automatically generated test items; means for analyzing the test results and reporting the vulnerabilities discovered; means for scheduling continuous monitoring and retesting; means for performing real-time vulnerability scans and notifying the user of the results; and means for performing analysis using visual data from digital devices. As a result, users can instantly detect vulnerabilities on digital devices they use daily without requiring specialized knowledge and take rapid security measures.

[0515] A "screen interface" is a visual structure that allows users to directly interact with software and web services through a digital device.

[0516] A "vulnerability test item" is a set of test criteria designed to assess the security risks inherent in a system or application.

[0517] "Attack simulation" is a process that measures the security performance of a system by simulating actual cyberattacks.

[0518] "Test results" refer to data and information obtained after conducting vulnerability tests, and are indicators of the system's security status.

[0519] "Continuous monitoring" is a process aimed at constantly monitoring the security status of systems and networks and detecting signs of anomalies or vulnerabilities at an early stage.

[0520] "Means for scheduling retesting" refers to a function that sets the timing for automatically re-running vulnerability detection tests, either periodically or as needed.

[0521] "Real-time vulnerability scanning" is an immediate assessment method for instantly detecting security risks while a digital device is in operation.

[0522] "Visual data from digital devices" refers to image data that includes information displayed on the screen of devices such as smartphones and personal computers.

[0523] This invention aims to realize a system that detects and reports vulnerabilities in screen interfaces operating on digital devices in real time. The system mainly consists of a server and terminals, and easily provides security information to users.

[0524] The server analyzes the screen interface of the terminal being operated by the user, recognizing its structure and elements. Software such as OpenCV for image processing is used for this analysis. Based on this analysis, the server automatically generates vulnerability test items and performs attack simulations based on these items.

[0525] The device uses acquired visual data to scan for vulnerabilities in real time. The scan results are sent to a server, and recommended countermeasures are notified to the user as needed. This notification is delivered through the interface of a smartphone or smart glasses. This allows users to instantly check the security status of their device without requiring specialized knowledge.

[0526] The server features continuous monitoring capabilities and allows for scheduled periodic retesting. This ensures that the overall system security remains up-to-date at all times.

[0527] For example, when a user is browsing a website on a mobile device, the application immediately performs a security scan and issues a warning about potential vulnerabilities. Another example is when using a news app at night; the application might notify the user about a potential SQL injection attack in a test environment. In this way, users can quickly take the necessary actions.

[0528] An example of a prompt when using a generative AI model is: "How can we design a feature that detects vulnerabilities in real time while using a smartphone app and provides the user with specific suggestions for improvement?"

[0529] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0530] Step 1:

[0531] The device takes a screenshot of the screen interface based on user actions. This screenshot serves as input data, playing a role in collecting visual information. Specifically, the device's camera or screen capture function is automatically activated, and image data is collected after obtaining user consent.

[0532] Step 2:

[0533] The device sends the captured screenshot to the server. Here, the input data is the captured image, and the output data is transferred to the server. In this step, the device uses data communication to send the image data to the specified server address.

[0534] Step 3:

[0535] The server analyzes the received screenshot using an image processing algorithm to recognize the screen layout and each element. In this step, the screenshot is used as input data, and the recognition result is obtained as output data. Specifically, the server analyzes the image using libraries such as OpenCV to identify elements such as buttons, text fields, and links.

[0536] Step 4:

[0537] The server automatically generates vulnerability test items based on the screen elements it recognizes. Here, the input data is the recognition result, and the output data is the automatically generated test items. Specifically, the server uses the generated AI model to create vulnerability test cases for each screen element.

[0538] Step 5:

[0539] The server performs attack simulations using automatically generated test items. In this step, the test items become input data, and results for security evaluation are output. The server uses existing security tools to simulate attacks such as SQL injection and XSS attacks and detect vulnerabilities.

[0540] Step 6:

[0541] The server analyzes the results of attack simulations and generates a report summarizing the vulnerabilities it discovers. The input data is the simulation results, and the output data is the report. Specifically, the server creates a report based on the analysis results that details the discovered vulnerabilities and recommends appropriate improvements.

[0542] Step 7:

[0543] The server sends the report to the terminal and notifies the user. In this step, the generated report is the input data and the notification is the output. When the terminal receives the notification, it displays a pop-up on the screen, allowing the user to view the report.

[0544] Step 8:

[0545] The user receives a notification and reviews the report. Here, the input data is the report notification, and the user's understanding is the output. Specifically, the user taps the push notification on their device and can access the report details page.

[0546] Step 9:

[0547] The server performs continuous monitoring and periodic retesting based on a schedule specified by the user. The input data is the user-specified schedule, and the output data is the periodic test results. The server automatically executes retests according to the schedule and accumulates the results.

[0548] In this system, the generative AI model is used to generate test items and analyze simulation results. An example of a prompt is, "How can we design a feature that detects vulnerabilities in real time while using a smartphone app and provides the user with specific improvement suggestions?"

[0549] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0550] This invention provides a system that automatically detects vulnerabilities in software and websites and provides improvement suggestions and reports based on the user's emotional state. The system mainly consists of a server, terminals, and an emotion engine. The server is responsible for central control and data processing, while the terminals handle the user interface and direct interaction with the user. The emotion engine analyzes the user's emotions and dynamically adjusts the user interface and notification methods.

[0551] Server-based processes

[0552] The server recognizes the screen interface displayed on the terminal and analyzes its structure and each element. Based on this, the server automatically generates vulnerability test items and uses them to perform attack simulations. For example, it sets up data to test threats such as SQL injection and cross-site scripting (XSS). Upon receiving the test results, the server analyzes the response to identify vulnerabilities and records them.

[0553] Adjustment by the emotion engine

[0554] The emotion engine determines the user's emotions from factors such as facial expressions and tone of voice, and provides this information to the server. This emotion information is used to adjust the content of reports and how they are presented. For example, if a user is feeling stressed, the system can adjust to provide concise and positive feedback.

[0555] Automated operation and notifications for the device

[0556] The device performs tests based on test items by automatically inputting information into the user interface. During this process, the emotion engine ensures that the device acts in a way that considers the user's emotions when necessary. Furthermore, reports and improvement suggestions are provided to the user via the device, and their content is adjusted based on emotion data.

[0557] Continuous monitoring and optimization

[0558] Users can schedule diagnostics for the server and perform continuous vulnerability assessments. The server uses data from the sentiment engine to perform diagnostics and notifications at the optimal time, improving the user experience. This system enables flexible responses based on user emotions, achieving advanced security management and customized user notifications.

[0559] The following describes the processing flow.

[0560] Step 1:

[0561] The server analyzes the screen interface data received from the terminal to recognize the page structure and its elements. Specifically, it analyzes HTML and CSS and extracts attribute information for input fields and buttons.

[0562] Step 2:

[0563] The server automatically generates vulnerability test items based on the analyzed information. For example, it prepares test data for SQL injection and scripts for cross-site scripting.

[0564] Step 3:

[0565] The terminal automatically performs tests on the screen interface using test items provided by the server. Specifically, it uses a browser automation tool to input data into an input form and then performs a submit action.

[0566] Step 4:

[0567] The emotion engine analyzes the user's facial expressions and voice data to determine their current emotional state. For example, it collects emotional data in real time using a camera and microphone.

[0568] Step 5:

[0569] The server integrates test results from the terminals with sentiment information from the sentiment engine and performs an evaluation. Specifically, it analyzes the risk level of vulnerabilities detected in the tests and selects a reporting method that matches the user's sentiment.

[0570] Step 6:

[0571] Based on the analysis results, the server generates a report for the user and suggests improvement measures as needed. It takes emotional information into consideration and strives to provide feedback that reduces stress.

[0572] Step 7:

[0573] The user initiates continuous monitoring based on a regular diagnostic schedule set on the server. The server utilizes data from the emotion engine to select and execute the optimal diagnostic timing.

[0574] (Example 2)

[0575] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0576] Traditional security systems rely on static methods to detect vulnerabilities in websites and software, which means they cannot flexibly respond to user emotional states or usage patterns. Furthermore, the way in which improvement suggestions are not communicated and results are presented is not appropriately tailored to the user, hindering improvements in the user experience.

[0577] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0578] In this invention, the server includes means for analyzing the screen interface and recognizing the screen configuration and each element, means for automatically generating vulnerability test items, and means for analyzing the user's emotional state and dynamically adjusting the reporting and notification methods. This enables flexible responses in accordance with the user's emotional state, and improves the user experience through more personalized improvement suggestions and notifications.

[0579] "Screen interface" refers to the display and operation portion used for exchanging information between the user and the system.

[0580] "Screen layout" is a concept that refers to the overall structure of how elements within an interface are arranged and displayed.

[0581] An "element" refers to individual parts or components included within a screen interface, such as buttons or text fields.

[0582] "To recognize" means that a system has the ability to detect and understand the configuration and elements within a screen interface.

[0583] "Vulnerability test items" refer to specific tests set up to detect potential security holes or flaws in a screen interface.

[0584] "Automatic generation" refers to a process in which a system generates test items and results without human intervention.

[0585] "Attack simulation" refers to a testing method that simulates techniques for exploiting vulnerabilities in a virtual environment, thereby verifying the system's defensive capabilities.

[0586] "Test results" refer to the data and analysis obtained as a result of the attack simulations that were conducted.

[0587] "Reporting" refers to the process of notifying users of detected vulnerabilities and issues and providing information to help them understand them.

[0588] "Emotional state" refers to a user's current emotional response and feelings, and is used to develop approaches tailored to each user's individual situation.

[0589] "Dynamic adjustment" means automatically changing actions and displays to suit the situation based on the user's emotions and circumstances.

[0590] "Continuous monitoring" refers to a series of activities aimed at regularly and continuously monitoring vulnerabilities to detect new problems early.

[0591] "Schedule a retest" means making a plan to conduct security tests again, either periodically or at specific times.

[0592] This invention provides a system for effectively managing the security of software and websites. The system mainly consists of a server, terminals, and an emotion engine. The server is responsible for centrally analyzing data and performing tests. The terminals provide the user interface and receive user interaction and feedback. The emotion engine analyzes the user's emotional state and provides data for the entire system to be dynamically adjusted.

[0593] The server analyzes the website's screen interface, recognizing its structure and individual elements. This uses software libraries for parsing HTML documents. Based on the analyzed information, the server automatically generates vulnerability test items. These generated test items are used to perform attack simulations against specific security threats. This process allows for the efficient detection of vulnerabilities, such as those related to SQL injection and cross-site scripting (XSS).

[0594] The emotion engine analyzes the user's facial expressions and voice data to infer their emotional state. This allows the server to dynamically adjust the content and method of reports and notifications to the user. For example, if the user is stressed, the system will adjust to provide simpler, more positive feedback.

[0595] For example, if a user wants to perform regular security checks on their web application, the system would display the check results at a time convenient for the user and provide a report including solutions for any problems found. This allows the user to effectively implement security measures.

[0596] Examples of prompts for a generative AI model:

[0597] "A user has requested a security assessment of a specific web application. How does the server analyze the interface and generate vulnerability test items? Please provide specific processing steps."

[0598] This system flexibly responds to the latest security needs and provides more user-friendly security management.

[0599] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0600] Step 1:

[0601] The user requests a security scan of a website or software through the terminal and sets the desired diagnostic schedule. Input includes the target URL and diagnostic criteria. Based on this, the terminal sends the user's request to the server. The output is the diagnostic request data passed to the server.

[0602] Step 2:

[0603] The server parses the received diagnostic request data and obtains the HTML structure of the specified screen interface. This process uses a specific software library to analyze the screen interface and extract information about its components (e.g., input fields, buttons). The input is the received interface data, and the output is the structure data based on the analysis results.

[0604] Step 3:

[0605] The server automatically generates vulnerability test items based on the analyzed structural data. Specifically, it generates test data related to SQL injection and cross-site scripting (XSS) attacks. In this step, data calculations are performed to verify whether elements within the interface have potential vulnerabilities. The input is structural data, and the output is test item data.

[0606] Step 4:

[0607] The server uses the generated test item data to perform attack simulations against the target screen interface. For example, it sends malicious input data and observes the system's response. The input is the test item data, and the output is the simulation result data.

[0608] Step 5:

[0609] The server analyzes the simulation results data and identifies any vulnerabilities found. These results are then compiled into a report for the user. The input is the simulation results data, and the output is the vulnerability report data.

[0610] Step 6:

[0611] The emotion engine analyzes the user's facial expressions and voice using the device's camera and microphone to evaluate the user's emotional state. This information is sent to a server and used to adjust reporting and notification methods. The input is real-time audio and video data, and the output is emotional state data.

[0612] Step 7:

[0613] The device displays a personalized report to the user based on their emotional state data. For example, if the user is feeling anxious, it will display only the most important information concisely. The input consists of vulnerability report data and emotional state data, and the output is the content displayed to the user.

[0614] Step 8:

[0615] The user reviews the report and takes action based on the suggested improvements. Furthermore, the server plans the next test according to a recurring diagnostic schedule. The input is the final report, and the output is the schedule information for the next diagnostic.

[0616] (Application Example 2)

[0617] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0618] Traditional vulnerability detection systems can be stressful for users because they provide uniform notifications and reports without considering the user's emotional state. Furthermore, there is a lack of concrete support for non-technical users to fully understand and improve security issues. Solving these problems is essential.

[0619] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0620] In this invention, the server includes means for analyzing the screen interface and recognizing the screen configuration and each element, means for automatically generating vulnerability test items based on each recognized element on the screen, and means for analyzing the user's emotional state. This makes it possible to select the optimal notification method according to the user's emotions and to provide appropriate security improvement suggestions without causing stress.

[0621] A "screen interface" is the visual display portion that allows a user to interact with software or a website.

[0622] "To recognize" means to identify and understand the characteristics of an object or data.

[0623] "Vulnerability test items" are test items used to evaluate vulnerabilities hidden in software or websites.

[0624] An "attack simulation" is a method of evaluating the resistance to security vulnerabilities by intentionally performing operations that exploit them.

[0625] "Test results" refer to the data and findings obtained from vulnerability testing.

[0626] "Continuous monitoring" means monitoring the system at regular time intervals and tracking changes in its status.

[0627] "Schedule a retest" means determining a time for administering the test again after the initial examination.

[0628] "User emotional state" refers to the emotions a user experiences while using the product or service, and includes stress and a sense of security.

[0629] "Notification method" refers to the means or methods used to convey information to the user.

[0630] "Report content" refers to the contents of a document summarizing test results and analysis results.

[0631] The system that implements this application is a complex configuration including a server, terminal, and emotion engine. First, the server analyzes the screen interface and recognizes the screen configuration and each element. This automatically generates vulnerability test items based on each element. Using these automatically generated items, the server performs attack simulations against the screen interface and evaluates threats such as SQL injection and cross-site scripting (XSS).

[0632] The emotion engine analyzes the user's facial expressions and voice using the smartphone's built-in camera and microphone. The specific software used for analysis includes Microsoft Azure Face API and Google Cloud Speech-to-Text. Based on this emotion information, the server optimizes notifications and generates reports designed to reduce user stress.

[0633] The terminal is equipped with tools that perform automated input operations on screen elements and conducts inspections based on test items. This automated operation allows the system to perform continuous monitoring and optimization with minimal user intervention.

[0634] For example, small businesses can use this system to strengthen the security of their websites and reduce stress on employees during work hours. For instance, if an SQL injection vulnerability is found, they can receive a notification message such as, "To ensure the security of your site, please try the following steps:"

[0635] An example of an input prompt for the generating AI model is: "Write an example of positive and concise feedback that should be sent to a calm user after detecting an SQL injection vulnerability on their site."

[0636] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0637] Step 1:

[0638] The server scans the screen interface and recognizes the displayed screen configuration and each element. It receives the screen data as input and converts it into a format that stores the characteristics of the elements in a database. This prepares the foundational data necessary for generating subsequent vulnerability test items.

[0639] Step 2:

[0640] The server automatically generates vulnerability test items based on each element of the recognized screen. Using the element data obtained in the previous step as input, it outputs test scenarios for SQL injection and cross-site scripting. The generated test items are optimized based on a standard vulnerability checklist.

[0641] Step 3:

[0642] The server performs attack simulations using the generated test items. It obtains the test items as input and response data to the screen interface as output. This allows for the identification of actual security vulnerabilities. Detailed execution logs are recorded during this step.

[0643] Step 4:

[0644] The terminal automates input operations to screen elements using an automated input tool. Using the test item data generated in step 3 as input, it reproduces the simulated behavior on the user interface. As output, the system's response to each operation is collected.

[0645] Step 5:

[0646] The server uses an emotion engine to analyze the user's emotional state. It takes facial and audio data obtained from the camera and microphone as input and outputs emotional states such as stress and reassurance. Based on this, the optimal method for maximizing the effectiveness of notifications is devised.

[0647] Step 6:

[0648] The server generates vulnerability reports and improvement suggestions based on test results and the user's emotional state, and sends them to the terminal. It combines emotional information and test results as input, outputting concise feedback that is easy for the user to understand. A specific action plan is also provided at this point.

[0649] Step 7:

[0650] Users review the feedback provided through their devices and implement necessary improvements. At this stage, they follow specific instructions from the system and implement security enhancement procedures.

[0651] Step 8:

[0652] The server continuously monitors the system even after the remediation process is complete, checking for any new vulnerabilities. It receives the latest interface data as input and continuously generates new test items as output. This ensures that the system's security is always up-to-date.

[0653] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0654] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0655] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0656] [Fourth Embodiment]

[0657] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0658] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0659] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0660] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0661] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0662] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0663] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0664] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0665] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0666] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0667] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0668] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0669] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0670] This invention provides a system for automatically detecting and efficiently reporting vulnerabilities in software and websites. The entire system mainly consists of a server and terminals, with the server handling overall control and data processing, and the terminals performing direct operations with the user interface.

[0671] Server-based analysis and test item generation

[0672] The server first analyzes the screen interface displayed on the terminal, identifying its structure and each element. Using the information obtained from this analysis, the server automatically generates vulnerability test items. For example, if a login page is detected, the server prepares test data for simulating SQL injection and brute-force attacks.

[0673] Vulnerability detection and testing

[0674] The terminal automatically performs operations based on the generated test items. The server receives these test results and analyzes the data responses returned by each operation. For example, if the terminal enters a specific string into an input form, the server determines whether or not there is a vulnerability from the resulting error or warning messages.

[0675] Reporting of results and suggestions for improvement

[0676] The server analyzes the detected vulnerabilities and creates a detailed report for the user based on the results. The report includes the issues found, a risk assessment, and specific improvement suggestions. For example, if an input field is determined to be vulnerable to SQL injection, the server will suggest to the user the implementation of prepared statements.

[0677] Continuous monitoring and scheduling

[0678] Users can schedule regular diagnostics for their servers. Based on this setting, the servers can automatically retest at specified times, constantly monitoring for the latest threats. For example, by scheduling weekly system-wide tests, it's possible to maintain security even for sites that require frequent updates.

[0679] Thus, this system provides detailed and automated vulnerability detection and reporting capabilities, contributing to the mitigation of security risks. User intervention is minimized, enabling efficient and effective security management.

[0680] The following describes the processing flow.

[0681] Step 1:

[0682] The server scans the screen interface displayed on the terminal to recognize the page structure and each element. Specifically, it parses the DOM (Document Object Model) and extracts attributes of input fields and buttons.

[0683] Step 2:

[0684] The server automatically generates vulnerability test items based on each element of the recognized interface. For example, it builds the test data and scenarios necessary to detect SQL injection and cross-site scripting (XSS).

[0685] Step 3:

[0686] The device uses the generated test items to perform automated actions on the user interface. Specifically, it enters attack strings into forms and executes the submit action. This is done using browser automation tools.

[0687] Step 4:

[0688] The server receives the operation results provided by the terminal and analyzes the response. Specifically, it evaluates response codes and error messages to identify security vulnerabilities.

[0689] Step 5:

[0690] The server generates a report for the user based on the analysis results, notifying them of the vulnerability details along with suggested improvements. Providing users with specific corrective measures enables a rapid response.

[0691] Step 6:

[0692] Users can set up regular diagnostic schedules on the server to perform continuous security monitoring. The server automatically repeats diagnostics based on the settings, performing security checks to address the latest threats.

[0693] (Example 1)

[0694] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0695] Current security management systems require significant human resources and time to identify and properly address vulnerabilities in websites and software. This leads to delays in vulnerability discovery and response, increasing security risks. Furthermore, continuous monitoring to prevent vulnerability recurrence is insufficient.

[0696] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0697] In this invention, the server includes computer executable means for identifying screen configurations and each element, calculation means for generating test items based on each identified element, and processing means for performing automated tests based on the generated test items. This makes it possible to quickly and efficiently detect vulnerabilities in an automated manner and take appropriate countermeasures.

[0698] "Screen layout" refers to the overall design and layout of a computer's user interface, including the information displayed and the elements that can be interacted with.

[0699] "Each element" refers to an individual component present on a user interface that provides a specific function or information, including buttons, input fields, labels, etc.

[0700] "Computer executable means" refers to a set of instructions or processes that a computer can execute through software or hardware, and includes logic for achieving a specific task.

[0701] "Computational means" refers to the functions of a computer used to process, analyze, and evaluate data, and the process of deriving specific results using algorithms and mathematical formulas.

[0702] A "processing unit" is a part of a system that performs specific functions based on received data or instructions, and is responsible for functions such as data input, data conversion, and output generation.

[0703] "Time management tools" refer to functions for executing tasks and processes based on a fixed schedule, and involve managing regular actions using timers and schedulers.

[0704] "Communication means" refers to functions for transmitting data and information to other systems or users, enabling two-way information exchange via networks and digital messaging systems.

[0705] This invention is a system that automatically detects and reports vulnerabilities in software and websites, primarily using a server and terminals. The server is responsible for analysis, generation of test items, analysis of test results, and report creation, while the terminals perform direct operations with the user interface.

[0706] The server uses state-of-the-art image processing and natural language processing technologies to analyze the screen interface information received from the terminal. This allows it to identify the screen configuration and its elements, and generate vulnerability test items based on this information. As part of this process, the server can use a generative AI model to prepare datasets necessary for simulating, for example, SQL injection or brute-force attacks.

[0707] The terminal follows instructions for test items provided by the server and uses automation tools to reproduce operations on the user interface. This makes it possible to discover vulnerabilities in a real operating environment. Specifically, the terminal uses scripts and browser automation tools to automate form input and button clicks.

[0708] The server evaluates data responses from terminals and identifies potential security issues. Based on the analysis, it provides the user with a detailed report and proposes specific corrective actions. This process includes analyzing normal error messages and abnormal behavior.

[0709] Furthermore, users can schedule regular diagnostics through the server. This feature allows for continuous system monitoring and the detection of the latest threats. For example, users can set up weekly tests to maintain system security.

[0710] An example of a prompt message is, "Please tell me how to perform an SQL injection vulnerability test on a company's website and suggest improvements." In this way, the present invention provides a comprehensive solution for efficient security management.

[0711] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0712] Step 1:

[0713] The server receives screen interface data from the terminal as input. It analyzes the screen's components using a combination of OCR and natural language processing technologies. From this analysis, it identifies buttons, input fields, text labels, and other elements on the screen, and extracts their attribute information. Through this process, the server outputs detailed information about each element that makes up the screen.

[0714] Step 2:

[0715] The server automatically generates vulnerability test items from the screen configuration information obtained in Step 1. In this process, an AI model is used to determine the test items based on historical data and known vulnerability patterns. Specifically, test items for SQL injection and cross-site scripting are generated. This allows the server to output test data corresponding to specific attack scenarios.

[0716] Step 3:

[0717] The terminal uses test items provided by the server as input and performs automated operations on the user interface. Using a browser automation tool, it replicates operations such as entering test data into a login form and clicking buttons. Through this process, the terminal outputs specific test results to the server.

[0718] Step 4:

[0719] The server analyzes the test results received from the terminal in step 3 as input. It analyzes error messages and abnormal behavior from the test result data response to identify vulnerabilities. This analysis evaluates the system's response to malicious input and outputs potential security risks.

[0720] Step 5:

[0721] The server generates a comprehensive report based on the analyzed vulnerability information and provides it to the user. The report includes details of the discovered vulnerabilities, their impact, and specific remediation measures. Users can receive this report and use it to improve their system security.

[0722] (Application Example 1)

[0723] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0724] In recent years, security vulnerabilities in information systems have become more diverse and sophisticated, requiring immediate and continuous monitoring and countermeasures. However, conventional methods have presented challenges, such as the need for manual operation and specialized knowledge for vulnerability detection and reporting, making real-time vulnerability scanning difficult. Therefore, there is a need for a system that can easily and effectively detect vulnerabilities in digital devices used daily by users and respond quickly.

[0725] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0726] In this invention, the server includes means for analyzing the screen interface and recognizing the screen configuration and each element; means for automatically generating vulnerability test items based on each element on the screen that has been recognized; means for performing attack simulations against the screen interface using the automatically generated test items; means for analyzing the test results and reporting the vulnerabilities discovered; means for scheduling continuous monitoring and retesting; means for performing real-time vulnerability scans and notifying the user of the results; and means for performing analysis using visual data from digital devices. As a result, users can instantly detect vulnerabilities on digital devices they use daily without requiring specialized knowledge and take rapid security measures.

[0727] A "screen interface" is a visual structure that allows users to directly interact with software and web services through a digital device.

[0728] A "vulnerability test item" is a set of test criteria designed to assess the security risks inherent in a system or application.

[0729] "Attack simulation" is a process that measures the security performance of a system by simulating actual cyberattacks.

[0730] "Test results" refer to data and information obtained after conducting vulnerability tests, and are indicators of the system's security status.

[0731] "Continuous monitoring" is a process aimed at constantly monitoring the security status of systems and networks and detecting signs of anomalies or vulnerabilities at an early stage.

[0732] "Means for scheduling retesting" refers to a function that sets the timing for automatically re-running vulnerability detection tests, either periodically or as needed.

[0733] "Real-time vulnerability scanning" is an immediate assessment method for instantly detecting security risks while a digital device is in operation.

[0734] "Visual data from digital devices" refers to image data that includes information displayed on the screen of devices such as smartphones and personal computers.

[0735] This invention aims to realize a system that detects and reports vulnerabilities in screen interfaces operating on digital devices in real time. The system mainly consists of a server and terminals, and easily provides security information to users.

[0736] The server analyzes the screen interface of the terminal being operated by the user, recognizing its structure and elements. Software such as OpenCV for image processing is used for this analysis. Based on this analysis, the server automatically generates vulnerability test items and performs attack simulations based on these items.

[0737] The device uses acquired visual data to scan for vulnerabilities in real time. The scan results are sent to a server, and recommended countermeasures are notified to the user as needed. This notification is delivered through the interface of a smartphone or smart glasses. This allows users to instantly check the security status of their device without requiring specialized knowledge.

[0738] The server features continuous monitoring capabilities and allows for scheduled periodic retesting. This ensures that the overall system security remains up-to-date at all times.

[0739] For example, when a user is browsing a website on a mobile device, the application immediately performs a security scan and issues a warning about potential vulnerabilities. Another example is when using a news app at night; the application might notify the user about a potential SQL injection attack in a test environment. In this way, users can quickly take the necessary actions.

[0740] An example of a prompt when using a generative AI model is: "How can we design a feature that detects vulnerabilities in real time while using a smartphone app and provides the user with specific suggestions for improvement?"

[0741] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0742] Step 1:

[0743] The device takes a screenshot of the screen interface based on user actions. This screenshot serves as input data, playing a role in collecting visual information. Specifically, the device's camera or screen capture function is automatically activated, and image data is collected after obtaining user consent.

[0744] Step 2:

[0745] The device sends the captured screenshot to the server. Here, the input data is the captured image, and the output data is transferred to the server. In this step, the device uses data communication to send the image data to the specified server address.

[0746] Step 3:

[0747] The server analyzes the received screenshot using an image processing algorithm to recognize the screen layout and each element. In this step, the screenshot is used as input data, and the recognition result is obtained as output data. Specifically, the server analyzes the image using libraries such as OpenCV to identify elements such as buttons, text fields, and links.

[0748] Step 4:

[0749] The server automatically generates vulnerability test items based on the screen elements it recognizes. Here, the input data is the recognition result, and the output data is the automatically generated test items. Specifically, the server uses the generated AI model to create vulnerability test cases for each screen element.

[0750] Step 5:

[0751] The server performs attack simulations using automatically generated test items. In this step, the test items become input data, and results for security evaluation are output. The server uses existing security tools to simulate attacks such as SQL injection and XSS attacks and detect vulnerabilities.

[0752] Step 6:

[0753] The server analyzes the results of attack simulations and generates a report summarizing the vulnerabilities it discovers. The input data is the simulation results, and the output data is the report. Specifically, the server creates a report based on the analysis results that details the discovered vulnerabilities and recommends appropriate improvements.

[0754] Step 7:

[0755] The server sends the report to the terminal and notifies the user. In this step, the generated report is the input data and the notification is the output. When the terminal receives the notification, it displays a pop-up on the screen, allowing the user to view the report.

[0756] Step 8:

[0757] The user receives a notification and reviews the report. Here, the input data is the report notification, and the user's understanding is the output. Specifically, the user taps the push notification on their device and can access the report details page.

[0758] Step 9:

[0759] The server performs continuous monitoring and periodic retesting based on a schedule specified by the user. The input data is the user-specified schedule, and the output data is the periodic test results. The server automatically executes retests according to the schedule and accumulates the results.

[0760] In this system, the generative AI model is used to generate test items and analyze simulation results. An example of a prompt is, "How can we design a feature that detects vulnerabilities in real time while using a smartphone app and provides the user with specific improvement suggestions?"

[0761] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0762] This invention provides a system that automatically detects vulnerabilities in software and websites and provides improvement suggestions and reports based on the user's emotional state. The system mainly consists of a server, terminals, and an emotion engine. The server is responsible for central control and data processing, while the terminals handle the user interface and direct interaction with the user. The emotion engine analyzes the user's emotions and dynamically adjusts the user interface and notification methods.

[0763] Server-based processes

[0764] The server recognizes the screen interface displayed on the terminal and analyzes its structure and each element. Based on this, the server automatically generates vulnerability test items and uses them to perform attack simulations. For example, it sets up data to test threats such as SQL injection and cross-site scripting (XSS). Upon receiving the test results, the server analyzes the response to identify vulnerabilities and records them.

[0765] Adjustment by the emotion engine

[0766] The emotion engine determines the user's emotions from factors such as facial expressions and tone of voice, and provides this information to the server. This emotion information is used to adjust the content of reports and how they are presented. For example, if a user is feeling stressed, the system can adjust to provide concise and positive feedback.

[0767] Automated operation and notifications for the device

[0768] The device performs tests based on test items by automatically inputting information into the user interface. During this process, the emotion engine ensures that the device acts in a way that considers the user's emotions when necessary. Furthermore, reports and improvement suggestions are provided to the user via the device, and their content is adjusted based on emotion data.

[0769] Continuous monitoring and optimization

[0770] Users can schedule diagnostics for the server and perform continuous vulnerability assessments. The server uses data from the sentiment engine to perform diagnostics and notifications at the optimal time, improving the user experience. This system enables flexible responses based on user emotions, achieving advanced security management and customized user notifications.

[0771] The following describes the processing flow.

[0772] Step 1:

[0773] The server analyzes the screen interface data received from the terminal to recognize the page structure and its elements. Specifically, it analyzes HTML and CSS and extracts attribute information for input fields and buttons.

[0774] Step 2:

[0775] The server automatically generates vulnerability test items based on the analyzed information. For example, it prepares test data for SQL injection and scripts for cross-site scripting.

[0776] Step 3:

[0777] The terminal automatically performs tests on the screen interface using test items provided by the server. Specifically, it uses a browser automation tool to input data into an input form and then performs a submit action.

[0778] Step 4:

[0779] The emotion engine analyzes the user's facial expressions and voice data to determine their current emotional state. For example, it collects emotional data in real time using a camera and microphone.

[0780] Step 5:

[0781] The server integrates test results from the terminals with sentiment information from the sentiment engine and performs an evaluation. Specifically, it analyzes the risk level of vulnerabilities detected in the tests and selects a reporting method that matches the user's sentiment.

[0782] Step 6:

[0783] Based on the analysis results, the server generates a report for the user and suggests improvement measures as needed. It takes emotional information into consideration and strives to provide feedback that reduces stress.

[0784] Step 7:

[0785] The user initiates continuous monitoring based on a regular diagnostic schedule set on the server. The server utilizes data from the emotion engine to select and execute the optimal diagnostic timing.

[0786] (Example 2)

[0787] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0788] Traditional security systems rely on static methods to detect vulnerabilities in websites and software, which means they cannot flexibly respond to user emotional states or usage patterns. Furthermore, the way in which improvement suggestions are not communicated and results are presented is not appropriately tailored to the user, hindering improvements in the user experience.

[0789] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0790] In this invention, the server includes means for analyzing the screen interface and recognizing the screen configuration and each element, means for automatically generating vulnerability test items, and means for analyzing the user's emotional state and dynamically adjusting the reporting and notification methods. This enables flexible responses in accordance with the user's emotional state, and improves the user experience through more personalized improvement suggestions and notifications.

[0791] "Screen interface" refers to the display and operation portion used for exchanging information between the user and the system.

[0792] "Screen layout" is a concept that refers to the overall structure of how elements within an interface are arranged and displayed.

[0793] An "element" refers to individual parts or components included within a screen interface, such as buttons or text fields.

[0794] "To recognize" means that a system has the ability to detect and understand the configuration and elements within a screen interface.

[0795] "Vulnerability test items" refer to specific tests set up to detect potential security holes or flaws in a screen interface.

[0796] "Automatic generation" refers to a process in which a system generates test items and results without human intervention.

[0797] "Attack simulation" refers to a testing method that simulates techniques for exploiting vulnerabilities in a virtual environment, thereby verifying the system's defensive capabilities.

[0798] "Test results" refer to the data and analysis obtained as a result of the attack simulations that were conducted.

[0799] "Reporting" refers to the process of notifying users of detected vulnerabilities and issues and providing information to help them understand them.

[0800] "Emotional state" refers to a user's current emotional response and feelings, and is used to develop approaches tailored to each user's individual situation.

[0801] "Dynamic adjustment" means automatically changing actions and displays to suit the situation based on the user's emotions and circumstances.

[0802] "Continuous monitoring" refers to a series of activities aimed at regularly and continuously monitoring vulnerabilities to detect new problems early.

[0803] "Schedule a retest" means making a plan to conduct security tests again, either periodically or at specific times.

[0804] This invention provides a system for effectively managing the security of software and websites. The system mainly consists of a server, terminals, and an emotion engine. The server is responsible for centrally analyzing data and performing tests. The terminals provide the user interface and receive user interaction and feedback. The emotion engine analyzes the user's emotional state and provides data for the entire system to be dynamically adjusted.

[0805] The server analyzes the website's screen interface, recognizing its structure and individual elements. This uses software libraries for parsing HTML documents. Based on the analyzed information, the server automatically generates vulnerability test items. These generated test items are used to perform attack simulations against specific security threats. This process allows for the efficient detection of vulnerabilities, such as those related to SQL injection and cross-site scripting (XSS).

[0806] The emotion engine analyzes the user's facial expressions and voice data to infer their emotional state. This allows the server to dynamically adjust the content and method of reports and notifications to the user. For example, if the user is stressed, the system will adjust to provide simpler, more positive feedback.

[0807] For example, if a user wants to perform regular security checks on their web application, the system would display the check results at a time convenient for the user and provide a report including solutions for any problems found. This allows the user to effectively implement security measures.

[0808] Examples of prompts for a generative AI model:

[0809] "A user has requested a security assessment of a specific web application. How does the server analyze the interface and generate vulnerability test items? Please provide specific processing steps."

[0810] This system flexibly responds to the latest security needs and provides more user-friendly security management.

[0811] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0812] Step 1:

[0813] The user requests a security scan of a website or software through the terminal and sets the desired diagnostic schedule. Input includes the target URL and diagnostic criteria. Based on this, the terminal sends the user's request to the server. The output is the diagnostic request data passed to the server.

[0814] Step 2:

[0815] The server parses the received diagnostic request data and obtains the HTML structure of the specified screen interface. This process uses a specific software library to analyze the screen interface and extract information about its components (e.g., input fields, buttons). The input is the received interface data, and the output is the structure data based on the analysis results.

[0816] Step 3:

[0817] The server automatically generates vulnerability test items based on the analyzed structural data. Specifically, it generates test data related to SQL injection and cross-site scripting (XSS) attacks. In this step, data calculations are performed to verify whether elements within the interface have potential vulnerabilities. The input is structural data, and the output is test item data.

[0818] Step 4:

[0819] The server uses the generated test item data to perform attack simulations against the target screen interface. For example, it sends malicious input data and observes the system's response. The input is the test item data, and the output is the simulation result data.

[0820] Step 5:

[0821] The server analyzes the simulation results data and identifies any vulnerabilities found. These results are then compiled into a report for the user. The input is the simulation results data, and the output is the vulnerability report data.

[0822] Step 6:

[0823] The emotion engine analyzes the user's facial expressions and voice using the device's camera and microphone to evaluate the user's emotional state. This information is sent to a server and used to adjust reporting and notification methods. The input is real-time audio and video data, and the output is emotional state data.

[0824] Step 7:

[0825] The device displays a personalized report to the user based on their emotional state data. For example, if the user is feeling anxious, it will display only the most important information concisely. The input consists of vulnerability report data and emotional state data, and the output is the content displayed to the user.

[0826] Step 8:

[0827] The user reviews the report and takes action based on the suggested improvements. Furthermore, the server plans the next test according to a recurring diagnostic schedule. The input is the final report, and the output is the schedule information for the next diagnostic.

[0828] (Application Example 2)

[0829] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0830] Traditional vulnerability detection systems can be stressful for users because they provide uniform notifications and reports without considering the user's emotional state. Furthermore, there is a lack of concrete support for non-technical users to fully understand and improve security issues. Solving these problems is essential.

[0831] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0832] In this invention, the server includes means for analyzing the screen interface and recognizing the screen configuration and each element, means for automatically generating vulnerability test items based on each recognized element on the screen, and means for analyzing the user's emotional state. This makes it possible to select the optimal notification method according to the user's emotions and to provide appropriate security improvement suggestions without causing stress.

[0833] A "screen interface" is the visual display portion that allows a user to interact with software or a website.

[0834] "To recognize" means to identify and understand the characteristics of an object or data.

[0835] "Vulnerability test items" are test items used to evaluate vulnerabilities hidden in software or websites.

[0836] An "attack simulation" is a method of evaluating the resistance to security vulnerabilities by intentionally performing operations that exploit them.

[0837] "Test results" refer to the data and findings obtained from vulnerability testing.

[0838] "Continuous monitoring" means monitoring the system at regular time intervals and tracking changes in its status.

[0839] "Schedule a retest" means determining a time for administering the test again after the initial examination.

[0840] "User emotional state" refers to the emotions a user experiences while using the product or service, and includes stress and a sense of security.

[0841] "Notification method" refers to the means or methods used to convey information to the user.

[0842] "Report content" refers to the contents of a document summarizing test results and analysis results.

[0843] The system that implements this application is a complex configuration including a server, terminal, and emotion engine. First, the server analyzes the screen interface and recognizes the screen configuration and each element. This automatically generates vulnerability test items based on each element. Using these automatically generated items, the server performs attack simulations against the screen interface and evaluates threats such as SQL injection and cross-site scripting (XSS).

[0844] The emotion engine analyzes the user's facial expressions and voice using the smartphone's built-in camera and microphone. The specific software used for analysis includes Microsoft Azure Face API and Google Cloud Speech-to-Text. Based on this emotion information, the server optimizes notifications and generates reports designed to reduce user stress.

[0845] The terminal is equipped with tools that perform automated input operations on screen elements and conducts inspections based on test items. This automated operation allows the system to perform continuous monitoring and optimization with minimal user intervention.

[0846] For example, small businesses can use this system to strengthen the security of their websites and reduce stress on employees during work hours. For instance, if an SQL injection vulnerability is found, they can receive a notification message such as, "To ensure the security of your site, please try the following steps:"

[0847] An example of an input prompt for the generating AI model is: "Write an example of positive and concise feedback that should be sent to a calm user after detecting an SQL injection vulnerability on their site."

[0848] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0849] Step 1:

[0850] The server scans the screen interface and recognizes the displayed screen configuration and each element. It receives the screen data as input and converts it into a format that stores the characteristics of the elements in a database. This prepares the foundational data necessary for generating subsequent vulnerability test items.

[0851] Step 2:

[0852] The server automatically generates vulnerability test items based on each element of the recognized screen. Using the element data obtained in the previous step as input, it outputs test scenarios for SQL injection and cross-site scripting. The generated test items are optimized based on a standard vulnerability checklist.

[0853] Step 3:

[0854] The server performs attack simulations using the generated test items. It obtains the test items as input and response data to the screen interface as output. This allows for the identification of actual security vulnerabilities. Detailed execution logs are recorded during this step.

[0855] Step 4:

[0856] The terminal automates input operations to screen elements using an automated input tool. Using the test item data generated in step 3 as input, it reproduces the simulated behavior on the user interface. As output, the system's response to each operation is collected.

[0857] Step 5:

[0858] The server uses an emotion engine to analyze the user's emotional state. It takes facial and audio data obtained from the camera and microphone as input and outputs emotional states such as stress and reassurance. Based on this, the optimal method for maximizing the effectiveness of notifications is devised.

[0859] Step 6:

[0860] The server generates vulnerability reports and improvement suggestions based on test results and the user's emotional state, and sends them to the terminal. It combines emotional information and test results as input, outputting concise feedback that is easy for the user to understand. A specific action plan is also provided at this point.

[0861] Step 7:

[0862] Users review the feedback provided through their devices and implement necessary improvements. At this stage, they follow specific instructions from the system and implement security enhancement procedures.

[0863] Step 8:

[0864] The server continuously monitors the system even after the remediation process is complete, checking for any new vulnerabilities. It receives the latest interface data as input and continuously generates new test items as output. This ensures that the system's security is always up-to-date.

[0865] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0866] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0867] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0868] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0869] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0870] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0871] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0872] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0873] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0874] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0875] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0876] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0877] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0878] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0879] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0880] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0881] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0882] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0883] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0884] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0885] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0886] The following is further disclosed regarding the embodiments described above.

[0887] (Claim 1)

[0888] A means for analyzing the screen interface and recognizing the screen configuration and each element,

[0889] A means for automatically generating vulnerability test items based on each element on the screen that is recognized,

[0890] A means of conducting an attack simulation against a screen interface using automatically generated test items,

[0891] A means of analyzing test results and reporting discovered vulnerabilities,

[0892] Means for scheduling continuous monitoring and retesting,

[0893] A system that includes this.

[0894] (Claim 2)

[0895] The system according to claim 1, further comprising means for using an operation tool to automate input operations to each screen element.

[0896] (Claim 3)

[0897] The system according to claim 1, further comprising means for generating a report containing specific remediation measures for discovered vulnerabilities and notifying the user.

[0898] "Example 1"

[0899] (Claim 1)

[0900] Computer executable means for identifying screen configuration and each element,

[0901] A calculation means for generating test items based on each identified element,

[0902] Processing means for conducting automated tests based on generated test items,

[0903] Analytical means for reporting potential problems discovered based on test results,

[0904] A time management system for setting diagnostic timings and conducting continuous testing,

[0905] A system that includes this.

[0906] (Claim 2)

[0907] The system according to claim 1, which provides a mechanism for automating input operations for each screen element.

[0908] (Claim 3)

[0909] The system according to claim 1, comprising communication means for generating and communicating to a user a document containing detailed solutions to the discovered problems.

[0910] "Application Example 1"

[0911] (Claim 1)

[0912] A means for analyzing the screen interface and recognizing the screen configuration and each element,

[0913] A means for automatically generating vulnerability test items based on each element on the screen that is recognized,

[0914] A means of conducting an attack simulation against a screen interface using automatically generated test items,

[0915] A means of analyzing test results and reporting discovered vulnerabilities,

[0916] Means for scheduling continuous monitoring and retesting,

[0917] A means of performing real-time vulnerability scans and notifying users of the results,

[0918] A means of performing analysis using visual data from digital devices,

[0919] A system that includes this.

[0920] (Claim 2)

[0921] The system according to claim 1, further comprising means for using an operation tool to automate input operations to each screen element.

[0922] (Claim 3)

[0923] The system according to claim 1, further comprising means for generating a report containing specific remediation measures for discovered vulnerabilities and notifying the user.

[0924] "Example 2 of combining an emotion engine"

[0925] (Claim 1)

[0926] A means for analyzing the screen interface and recognizing the screen configuration and each element,

[0927] A means for automatically generating vulnerability test items based on each element on the screen that is recognized,

[0928] A means of conducting an attack simulation against a screen interface using automatically generated test items,

[0929] A means of analyzing test results and reporting discovered vulnerabilities,

[0930] A means for analyzing the user's emotional state and dynamically adjusting the reporting and notification methods,

[0931] Means for scheduling continuous monitoring and retesting,

[0932] A system that includes this.

[0933] (Claim 2)

[0934] The system according to claim 1, further comprising means for using an operating tool to automate input operations to each screen element.

[0935] (Claim 3)

[0936] The system according to claim 1, comprising means for generating a report containing specific corrective measures for discovered vulnerabilities and notifying the user.

[0937] "Application example 2 when combining with an emotional engine"

[0938] (Claim 1)

[0939] A means for analyzing the screen interface and recognizing the screen configuration and each element,

[0940] A means for automatically generating vulnerability test items based on each element on the screen that is recognized,

[0941] A means of conducting an attack simulation against a screen interface using automatically generated test items,

[0942] A means of analyzing test results and reporting discovered vulnerabilities,

[0943] Means for scheduling continuous monitoring and retesting,

[0944] A means of analyzing the user's emotional state,

[0945] A means to adjust notification methods and report content based on the results of user sentiment analysis,

[0946] A system that includes this.

[0947] (Claim 2)

[0948] The system according to claim 1, further comprising means for using an operation tool to automate input operations to each screen element.

[0949] (Claim 3)

[0950] The system according to claim 1, comprising means for generating a report that includes specific remediation measures for discovered vulnerabilities, adjusting the report content according to the user's emotional state, and notifying the user. [Explanation of symbols]

[0951] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for analyzing the screen interface and recognizing the screen configuration and each element, A means for automatically generating vulnerability test items based on each element on the screen that is recognized, A means of conducting an attack simulation against a screen interface using automatically generated test items, A means of analyzing test results and reporting discovered vulnerabilities, Means for scheduling continuous monitoring and retesting, A system that includes this.

2. The system according to claim 1, further comprising means for using an operation tool to automate input operations to each screen element.

3. The system according to claim 1, further comprising means for generating a report containing specific remediation measures for discovered vulnerabilities and notifying the user.

Images