Information processing device and control method

Abstract

Make better use of operations performed on the screen. [Solution] The information processing device comprises a drawing detection unit that detects drawing strokes corresponding to hand-drawn trajectories within the screen area, a drawing analysis unit that analyzes the drawing content based on the drawing strokes detected by the drawing detection unit, and an execution control unit that performs processing based on the positional relationship between the drawing strokes detected by the drawing detection unit and content displayed in the screen area other than the drawing strokes, and the drawing content analyzed by the drawing analysis unit.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and a control method.

Background Art

[0002] In recent years, there is a search function that allows users to search for a part of an image, illustration, etc. displayed on a screen by surrounding it with a touch pen or the like (see, for example, Non-Patent Document 1). This can be said to be a function that effectively utilizes the advantage of a pen that can select a free location within the screen.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, as described above, it is convenient for the user to be able to search for what is displayed in a part displayed on the screen by simply operating it with a touch pen or the like. However, it is expected to be utilized for more other functions, not limited to the search function.

[0005] In view of the above circumstances, the present invention is made, and one of its objects is to provide an information processing apparatus and a control method that can make better use of operations performed on a screen.

Means for Solving the Problems

[0006] The present invention has been made to solve the above problems, and an information processing device according to a first aspect of the present invention comprises: a drawing detection unit that detects drawing strokes corresponding to a trajectory entered by hand within a screen area; a drawing analysis unit that analyzes the drawing content by the drawing stroke detected by the drawing detection unit; and an execution control unit that performs processing based on the positional relationship between the drawing stroke detected by the drawing detection unit and content displayed in the screen area other than the drawing stroke, and the drawing content analyzed by the drawing analysis unit.

[0007] In the above-described information processing device, the execution control unit may cause the execution control unit to perform processing based on the drawing content analyzed by the drawing analysis unit for content displayed within the screen area in which the drawing strokes detected by the drawing detection unit overlap.

[0008] In the above-described information processing apparatus, if the drawing stroke detected by the drawing detection unit overlaps with the area of ​​a string of characters displayed within the screen area, the execution control unit may cause the string of characters to perform processing based on the drawing content analyzed by the drawing analysis unit.

[0009] In the above-described information processing device, if the drawing stroke detected by the drawing detection unit overlaps with the area of ​​the application window displayed within the screen area, the execution control unit may execute the processing of the function of the application that corresponds to the drawing content analyzed by the drawing analysis unit.

[0010] In the above-described information processing apparatus, if the drawing stroke detected by the drawing detection unit overlaps with an area of ​​an image displayed within the screen area, the execution control unit may cause the image to perform image processing based on the drawing content analyzed by the drawing analysis unit.

[0011] In the above-described information processing apparatus, the drawing analysis unit may analyze the drawing strokes detected by the drawing detection unit to extract drawing strokes representing characters and drawing strokes representing ranges as the drawing content, and the execution control unit may perform processing based on the drawing strokes representing characters for content that overlaps with the drawing strokes representing ranges among the content displayed within the screen area.

[0012] In the above-described information processing device, the execution control unit may execute a pre-set process if the drawing stroke detected by the drawing detection unit overlaps with a blank area within the screen area.

[0013] In the above-described information processing device, the execution control unit may cause the execution control unit to perform processing based on the drawing content analyzed by the drawing analysis unit for content displayed within the screen area that is adjacent to the drawing stroke detected by the drawing detection unit.

[0014] In the above-described information processing device, the execution control unit may, if the drawing stroke detected by the drawing detection unit includes a mathematical formula and is adjacent to a region of a number displayed within the screen area, cause the execution control unit to perform a calculation process based on the mathematical formula on the number.

[0015] In the above-described information processing device, the execution control unit may generate and output a command to execute processing based on the drawing content analyzed by the drawing analysis unit.

[0016] In the above-described information processing device, the execution control unit may generate and output a prompt as an input statement to the generating AI that performs processing based on the drawing content analyzed by the drawing analysis unit.

[0017] In addition, the control method in the information processing apparatus according to the second aspect of the present invention includes: a step in which a drawing detection unit detects a drawing stroke corresponding to a trajectory hand-drawn in a screen area; a step in which a drawing analysis unit analyzes the drawing content based on the drawing stroke detected by the drawing detection unit; and a step in which an execution control unit executes processing based on the positional relationship between the drawing stroke detected by the drawing detection unit and the content displayed in the screen area other than the drawing stroke, and the drawing content analyzed by the drawing analysis unit.

Effect of the Invention

[0018] According to the above aspect of the present invention, operations performed on the screen can be more effectively utilized.

Brief Description of the Drawings

[0019] [Figure 1] A diagram showing an example of the configuration of an information processing system according to an embodiment. [Figure 2] A block diagram showing an example of the hardware configuration of an information processing apparatus according to an embodiment. [Figure 3] A diagram showing an example of a UI for hand-drawn input according to an embodiment. [Figure 4] A diagram showing a first example of hand-drawn input execution processing according to an embodiment. <00_{000}085>A diagram showing a second example of hand-drawn input execution processing according to an embodiment. [Figure 6] A diagram showing a third example of hand-drawn input execution processing according to an embodiment. [Figure 7] A block diagram showing an example of a functional configuration related to hand-drawn input execution processing according to an embodiment. [Figure 8] A flowchart showing an example of hand-drawn input execution processing according to an embodiment. [Figure 9] A flowchart showing a specific example of hand-drawn input execution processing according to an embodiment. [Figure 10] A diagram showing a fourth example of hand-drawn input execution processing according to an embodiment. [Figure 11] A diagram showing a fifth example of hand-drawn input execution processing according to an embodiment. [Figure 12] This figure shows a sixth example of the handwriting input execution process according to the embodiment. [Modes for carrying out the invention]

[0020] Embodiments of the present invention will be described below with reference to the drawings. [System Configuration] Figure 1 shows an example of the configuration of the information processing system according to this embodiment. The information processing system SYS comprises an information processing device 10 and a server 30.

[0021] The information processing device 10 is, for example, a clamshell-type (notebook-type) PC (personal computer). The information processing device 10 consists of a roughly rectangular, plate-shaped (for example, flat) first enclosure 10A and a second enclosure 10B, which are connected (linked) via a hinge mechanism so that they can be opened and closed. The information processing device 10 also has a display 150 that spans from the first enclosure 10A to the second enclosure 10B.

[0022] The display 150 is a flexible display that can be bent when the first housing 10A and the second housing 10B are opened and closed. As a flexible display, for example, an organic EL display can be used. For example, the display 150 can be used not only in a single-screen mode in which the entire screen area is a single screen, but also in a two-screen mode in which the screen is divided into two screens: a first screen area 150A on the first housing 10A side and a second screen area 150B on the second housing 10B side. For example, when the information processing device 10 is used on a desk, the second screen area 150B on the second housing 10B side becomes a nearly horizontal screen parallel to the surface of the desk.

[0023] Furthermore, a touch sensor is provided on the top (surface) of the display 150, allowing the user to perform touch operations using their finger or a stylus. The information processing device 10 is capable of detecting touch operations on the screen area of ​​the display 150. By opening the information processing device 10, the user can view the displays on the displays 150 located on the inner surfaces of the first housing 10A and the second housing 10B, and perform touch operations on the displays 150, thereby enabling the use of the information processing device 10.

[0024] Server 30 is equipped with a Generative AI 31. Generative AI 31 can be equipped with various types of Generative AI (Artificial Intelligence). For example, Generative AI 31 is a Generative AI with Large Language Models (LLM), such as Vision LLM. As will be described in more detail later, for example, Generative AI 31 outputs text information (sentences) that causes the Generative AI 31 to perform processing based on the drawing content (such as characters) that is hand-drawn by the user in any part of the screen of the display 150 of the information processing device 10.

[0025] This text information may be used as a command to cause an application to perform a specific function, or as a prompt as input text to various generative AIs. Here, various generative AIs may be, for example, a generative AI having the large-scale language model (LLM) described above, or a generative AI having an image generation model trained by a diffusion model, and any generative AI can be applied depending on the application.

[0026] Furthermore, server 30 may be configured as a single server, or it may be configured as being distributed across multiple servers. For example, it may be configured as a server 30 equipped with multiple generating AIs, or multiple servers 30 may be provided for each generating AI.

[0027] In this embodiment, the server 30 is equipped with a generation AI 31, and the information processing device 10 communicates with the server 30 to utilize the functions of the generation AI 31. However, the functions of the generation AI 31 may also be downloaded to the information processing device 10 and used locally.

[0028] [Hardware configuration of the information processing device 10] The following describes the specific configuration of the information processing device 10. Figure 2 is a block diagram showing an example of the hardware configuration of the information processing device 10 according to this embodiment. The information processing device 10 includes a communication unit 11, a RAM (Random Access Memory) 12, a storage unit 13, a speaker 14, a display unit 15, a camera 16, and a control unit 18. Each of these units is connected to communicate via a bus or the like.

[0029] The communication unit 11 is comprised of, for example, multiple Ethernet® ports, multiple USB (Universal Serial Bus) or other digital input / output ports, and communication devices that perform wireless communication such as Bluetooth® or Wi-Fi®.

[0030] For example, the communication unit 11 can communicate with a stylus, mouse, touchpad, etc., using Bluetooth®. Furthermore, the communication unit 11 can communicate with the server 30 shown in Figure 1 by connecting to the internet via wireless communication such as Wi-Fi® or wired communication such as Ethernet®.

[0031] RAM12 is a volatile memory where programs and data for processing executed by the control unit 18 are stored, and various data are saved or erased as needed. Since RAM12 is a volatile memory, it will not retain data when power to RAM12 is cut off. Data that needs to be retained when power to RAM12 is cut off is transferred to the storage unit 13.

[0032] The memory unit 13 is a storage device that includes one or more of the following: SSD (Solid State Drive), HDD (Hard Disk Drive), ROM (Residual Only Memory), Flash-ROM, etc. For example, the memory unit 13 stores BIOS (Basic Input Output System) programs and configuration data, OS (Operating System) and application programs that run on the OS, and various data used by applications. Speaker 14 outputs electronic sounds, voices, etc.

[0033] The display unit 15 includes a display 150 and a touch sensor 155. As mentioned above, the display 150 is a flexible display that can be bent in accordance with the opening and closing of the first housing 10A and the second housing 10B. The display 150 displays the OS desktop screen, running application windows, etc., in accordance with the control of the control unit 18.

[0034] The touch sensor 155 is located on the screen of the display 150 and detects touch operations on the screen. Touch operations include, for example, tapping, sliding, flicking, swiping, and pinching. Touch operations can be performed using a finger or a stylus.

[0035] The camera 16 is composed of a lens, an image sensor, and the like. The camera 16 captures images (still images and videos) and outputs the data of the captured images in accordance with the control unit 18.

[0036] The control unit 18 is composed of processors such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a microcomputer, and these processors execute programs (such as the BIOS, OS, and applications that run on the OS) stored in the memory unit 13, etc., to realize various functions. For example, the control unit 18 controls the screen mode of the display 150, controls the display on the screen area, and detects touch operations on the touch sensor 155.

[0037] Next, we will describe the handwriting input execution process in the information processing device 10, which executes processing based on the handwritten input content (such as characters) entered by the user on the screen of the display 150. First, we will explain the UI (User Interface) with reference to Figure 3.

[0038] Figure 3 shows an example of a handwriting input UI according to this embodiment. In the illustrated example, the information processing device 10 controls the display 150 to a two-screen mode consisting of a first screen area 150A and a second screen area 150B. The first screen area 150A and the second screen area 150B display content such as the OS desktop screen and the windows of running applications.

[0039] In the handwriting input execution process, a transparent layer for handwriting input is added to the screen of the display 150 when the user performs a predetermined operation, allowing handwriting input using a stylus (or finger) at any location on the current screen display. The predetermined operation may be, for example, the simultaneous operation of several predetermined keys, or the operation of "launching" a specific application for handwriting input.

[0040] In the example shown in this figure, the user is inputting characters by hand using a stylus in the second screen area 150B, but input in the first screen area 150A is also acceptable. When the information processing device 10 detects a drawing stroke corresponding to the hand-drawn input trajectory, it sets a bounding box B1 indicating the range of the drawing stroke and analyzes its positional relationship with other displayed content. In the illustrated example, the bounding box B1 (drawing stroke) overlaps with content R1 displayed in the second screen area 150B. Therefore, the information processing device 10 determines that the drawing content (characters, etc.) input by the user by hand is an action for content R1 and executes an action on content R1 based on the drawing content (characters, etc.).

[0041] Content R1 is not particularly limited and could be, for example, an image displayed on the screen, text displayed on the screen, or an application window displayed on the screen. While images and text are commonly displayed in the windows of some application, this means that the scope of Content R1 is not limited to the entire window, but may be limited to only a portion of the window, such as an image or text.

[0042] The following examples of the handwriting input execution process will be explained with reference to Figures 4-6. Figure 4 shows a first example of the handwritten input execution process according to this embodiment. The example shown in Figure 4 is an example of a process that is executed when characters (drawing strokes) entered by the user by hand overlap an area of ​​an image. In this example, as shown in (A), the word "Simple" is entered by hand on top of the image. The information processing device 10 analyzes the user's desired process based on the fact that the entered characters are "Simple" and that the characters (drawing strokes) overlap an area of ​​an image, and generates a prompt instructing the generation of a simplified image. The information processing device 10 then inputs the prompt and the image to the image generation AI (for example, by sending it to the server 30) to obtain the simplified generated image shown in (B).

[0043] Figure 5 shows a second example of the handwritten input execution process according to this embodiment. The example shown in Figure 5 is an example of a process that is executed when characters (drawing strokes) entered by the user by hand overlap the area of ​​the application window. The application has at least mail and calendar functions. In this example, as shown in (A), "Monthly Report 3:30~4:30" is entered by hand on the mail screen window. The information processing device 10 analyzes that the entered characters (drawing strokes) relate to time and items (monthly report), and that they overlap the area of ​​the application window which has mail and calendar functions, so that adding a schedule is the process the user wants to do, and outputs a command to the application to execute the calendar function. As a result, as shown in (B), the calendar screen window is displayed. For example, if the information processing device 10 analyzes that adding a schedule is the process the user wants to do, a calendar entry is generated and displayed that adds the entered time and items (monthly report) to the calendar. If only the date and time are entered by hand, the calendar registration status for the entered date and time may be displayed.

[0044] Figure 6 shows a third example of the handwriting input execution process according to this embodiment. The example shown in Figure 6 is an example of a process that is executed when a drawing stroke entered by the user by hand overlaps with the area of ​​a string (text). Similar to the examples in Figures 4 and 5, the user may also enter characters indicating the desired process by hand, overlapping with the area of ​​the string, but when the target of processing is a string, it is difficult to determine the range to be processed.

[0045] Therefore, in the example shown in Figure 6, the drawing strokes entered by the user by hand include drawing strokes that indicate the range to be processed and drawing strokes of characters that indicate the processing content. In all of the examples shown in (A), (B), and (C), the user selects the range of strings to be processed by drawing a circle around it by hand. The range of strings to be processed is selected as a single block of strings when an approximate range is enclosed.

[0046] In addition, users handwrite the processing content next to the area they have selected by hand. In the example shown in (A), "to CN" is handwritten as the processing content, which means translation into Chinese. In the example shown in (B), "Casual" is handwritten as the processing content, which means conversion to casual language. In the example shown in (C), "Summarize" is handwritten as the processing content, which means generation of a summary.

[0047] The information processing device 10 executes processing on a string if a drawing stroke entered by the user by hand overlaps with the area of ​​a string. For example, if a drawing stroke indicating the range to be processed overlaps with the area of ​​a string, the information processing device 10 executes processing on the string that overlaps with the drawing stroke indicating the range, based on the drawing stroke of the character indicating the processing content.

[0048] Specifically, in the example shown in (A), the information processing device 10 executes a process to translate the string selected as the target of processing into Chinese. For example, the information processing device 10 executes a command to launch an application that has a translation function, inputs the string selected as the target of processing into the launched application as the target of translation, and obtains the translation result.

[0049] In the example shown in (B), the information processing device 10 executes a process to convert the selected string into a casual expression. For example, the information processing device 10 generates a prompt indicating an instruction to convert the selected string into a casual expression. The information processing device 10 then inputs the generated prompt to the generation AI 31 and obtains the string converted into a casual expression generated by the generation AI 31.

[0050] In the example shown in (C), the information processing device 10 executes a process to generate a summary of the string selected as the processing target. For example, the information processing device 10 generates a prompt indicating an instruction to generate a summary of the selected string. The information processing device 10 then inputs the generated prompt to the generation AI 31 and obtains the summary generated by the generation AI 31.

[0051] In the example shown in Figure 6, the user manually draws a box around the range of the string to be processed. However, if the range of the string is clear without drawing a box, the system may be configured so that processing is executed on the string by manually drawing characters indicating the processing content over the string.

[0052] [Functional Configuration of Handwriting Input Execution Process] Next, we will explain in detail the handwriting input execution process, which performs processing based on the handwritten input (such as characters) that the user enters by hand on the screen.

[0053] Figure 7 is a block diagram showing an example of a functional configuration related to the handwriting input execution process according to this embodiment. The illustrated processing unit 110 has a functional configuration in which the control unit 18 executes an application program to perform the handwriting input execution process. Note that the functions of each part of the processing unit 110 may be realized by one application or by multiple applications.

[0054] The processing unit 110 includes a drawing stroke acquisition unit 111, a screen capture unit 112, an OS event acquisition unit 113, a drawing analysis unit 114, a drawing position analysis unit 115, and an execution control unit 116.

[0055] The drawing stroke acquisition unit 111 (an example of a drawing detection unit) detects drawing strokes that have been hand-drawn within the screen area of ​​the display 150. The drawing stroke acquisition unit 111 also sets and acquires a bounding box (for example, bounding box B1 shown in Figure 2) that indicates the range of the hand-drawn drawing stroke within the screen area of ​​the display 150. For example, drawing strokes include drawing strokes of characters as exemplified in Figures 4 to 6, and drawing strokes indicating a range as exemplified in Figure 6.

[0056] The screen capture unit 112 captures the screen area of ​​the display 150 as one piece of information for analyzing the positional relationship between the hand-drawn strokes and the content displayed on the screen. For example, when a drawing stroke is detected by the drawing stroke acquisition unit 111, the screen capture unit 112 captures the screen area including the range (bounding box) of the detected drawing stroke and saves the captured image. For example, by analyzing the captured image of the screen area, information such as whether the content displayed within the screen area is an image or text can be obtained.

[0057] The OS event acquisition unit 113 acquires the OS event log as one piece of information for analyzing the positional relationship between hand-drawn strokes and the content displayed on the screen. For example, by analyzing the OS event log, information such as the windows displayed within the screen area and which window is the active window can be obtained.

[0058] The drawing analysis unit 114 analyzes the drawing content created by the drawing strokes detected by the drawing stroke acquisition unit 111. For example, the drawing analysis unit 114 detects characters or lines indicating a range (outline) as drawing content by analyzing the shape of the drawing strokes. If characters are detected, the drawing analysis unit 114 also detects character clusters consisting of one or more words within the characters.

[0059] The drawing position analysis unit 115 analyzes the positional relationship between the drawing stroke detected by the drawing stroke acquisition unit 111 and content displayed in the screen area of ​​the display 150 other than the drawing stroke. For example, the drawing position analysis unit 115 analyzes the positional relationship between the bounding box of the drawing stroke and the content displayed in the screen area based on a screen capture image obtained by the screen capture unit 112 or an OS event log obtained by the OS event acquisition unit 113.

[0060] As an example, the drawing position analysis unit 115 performs screen analysis to determine which content the user has hand-drawn on, based on the bounding box of the drawing stroke and a captured image of the screen. As another example, the drawing position analysis unit 115 performs UI configuration analysis of the OS to determine which content the user has hand-drawn on, based on the bounding box of the drawing stroke and the OS event log.

[0061] The execution control unit 116 executes processing based on the analysis results from the drawing position analysis unit 115 (positional relationship between the drawing stroke and the content) and the drawing content analyzed by the drawing analysis unit 114. For example, the execution control unit 116 executes processing based on the drawing content analyzed by the drawing analysis unit 114 for content displayed within the screen area whose drawing strokes (bounding boxes) overlap.

[0062] Specifically, if a drawing stroke overlaps with an area of ​​an image displayed within the screen area, the execution control unit 116 may perform image processing on that image based on the drawing content analyzed by the drawing analysis unit 114 (see Figure 4).

[0063] Furthermore, if the drawing stroke overlaps with the area of ​​the application window displayed within the screen area, the execution control unit 116 may execute the processing of a function among the functions of the application that corresponds to the drawing content analyzed by the drawing analysis unit 114 (see Figure 5).

[0064] Furthermore, if a drawing stroke overlaps with the area of ​​a string of characters displayed within the screen area, the execution control unit 116 may perform processing based on the drawing content analyzed by the drawing analysis unit 114 for that string of characters (see Figure 6). Here, if the drawing analysis unit 114 extracts a drawing stroke representing a character and a drawing stroke representing a range, the execution control unit 116 may perform processing based on the drawing stroke representing the character for content among the content displayed within the screen area that overlaps with the range of the drawing stroke representing the range (see Figure 6).

[0065] Furthermore, the execution control unit 116 includes a user command analysis unit 117, a prompt generation unit 118, and a command processing unit 119.

[0066] The user command analysis unit 117 analyzes whether or not a process based on instructions (user commands) handwritten by the user requires generation AI. For example, a command list may be pre-configured consisting of commands that directly execute a process on the application and characters (words, etc.) that a user who wants to execute that process is expected to input by hand. The user command analysis unit 117 may then refer to this command list to determine whether or not a process requires generation AI.

[0067] For example, the user command analysis unit 117 determines that if the instruction (drawing content of a drawing stroke) entered by the user by hand is present in the command list, then the generation AI is not required for that process. On the other hand, the user command analysis unit 117 determines that if the instruction (drawing content of a drawing stroke) entered by the user by hand is not present in the command list, then the generation AI is required for that process.

[0068] The prompt generation unit 118 generates prompts to input to the generation AI based on the analysis results from the drawing position analysis unit 115 (positional relationship between drawing stroke and content) and the drawing content analyzed by the drawing analysis unit 114, for processes that the user command analysis unit 117 has determined require generation AI. For example, the prompt generation unit 118 generates prompts to input to the generation AI when executing the process of generating the simplified image shown in Figure 4. The prompt generation unit 118 also generates prompts to input to the generation AI when executing the process of converting the string (text) shown in Figure 6(B) into a casual expression, and the process of generating a summary of the string (text) shown in Figure 6(C).

[0069] Furthermore, the prompt generation unit 118 may generate prompts using generation AI such as LLM or Vision LLM, or it may generate them using predefined phrases. For example, keywords (expected words) related to the process that the user instructs with handwriting input and predefined phrases for prompts using those keywords may be set in advance, and the prompt generation unit 118 may generate prompts using these predefined phrases.

[0070] For processes that the user command analysis unit 117 has determined do not require AI generation, the command processing unit 119 generates commands to directly execute those processes based on the analysis results from the drawing position analysis unit 115 (positional relationship between the drawing stroke and the content) and the drawing content analyzed by the drawing analysis unit 114. For example, when the command processing unit 119 executes the process of generating calendar entries for the application shown in Figure 5, it generates commands using the application's API (Application Programming Interface). Also, when the command processing unit 119 executes the process of translating the string (text) shown in Figure 6(A) into Chinese, it generates commands to launch an application with translation functionality, etc.

[0071] [Operation of handwritten input execution process] Next, the operation of the handwriting input execution process performed by the processing unit 110 will be described. Figure 8 is a flowchart showing an example of the handwriting input execution process according to this embodiment.

[0072] (Step S101) The processing unit 110 detects a hand-drawn stroke within the screen area of ​​the display 150 and obtains a bounding box indicating the range of the drawing stroke. Then, the process proceeds to steps S103, S105, and S109.

[0073] (Step S103) The processing unit 110 analyzes the drawing content created by the drawing stroke detected in step S101. For example, the processing unit 110 analyzes the shape of the drawing stroke to detect characters or lines indicating a range (outline) as drawing content. If characters are detected, the processing unit 110 also detects character clusters consisting of one or more words within the characters. Then, it proceeds to the processing in step S113.

[0074] (Step S105) The processing unit 110 captures the screen area of ​​the display 150 and saves the captured image. At this time, the processing unit 110 captures all layers except those with handwritten input. When the processing unit 110 captures the screen area of ​​the display 150, it may capture the entire screen area, or it may capture at least a portion of the screen area that includes the bounding box obtained in step S101. Then, the process proceeds to step S107.

[0075] (Step S107) The processing unit 110 performs screen analysis on the positional relationship between the drawing stroke and the content displayed within the screen area, based on the bounding box of the drawing stroke detected in step S101 and the captured image of the screen captured in step S105. For example, the processing unit 110 performs screen analysis on which content the user has handwritten input on. Then, it proceeds to the process in step S113.

[0076] (Step S109) The processing unit 110 obtains the OS event log. Then, it proceeds to the process in step S111.

[0077] (Step S111) The processing unit 110 performs an OS UI configuration analysis on the positional relationship between the drawing stroke and the content displayed within the screen area, based on the bounding box of the drawing stroke detected in step S101 and the OS event log acquired in step S109. For example, the processing unit 110 performs an OS UI configuration analysis on which content the user has entered by hand. Then, it proceeds to the process in step S113.

[0078] (Step S113) The processing unit 110 processes the drawing content of the drawing stroke analyzed in step S103 and the analysis results (positional relationship between the drawing stroke and the content) analyzed in step S107 or step S111, that is, it analyzes the instructions (user commands) that the user has hand-drawn. Then it proceeds to the processing in step S115.

[0079] (Step S115) The processing unit 110 determines whether the process (user command) analyzed in step S113 is a process that requires generation AI. For example, the processing unit 110 refers to the command list set in advance as described above and determines that if the process can be implemented with a command that can directly execute the process on the application, it does not require generation AI, and determines that any other process requires generation AI. If the processing unit 110 determines that the process does not require generation AI (NO), it proceeds to the process in step S117. On the other hand, if the processing unit 110 determines that the process requires generation AI (YES), it proceeds to the process in step S119.

[0080] (Step S117) The processing unit 110 generates and outputs a command to directly instruct the application to execute processing based on the drawing content of the drawing stroke analyzed in step S103 and the analysis results (positional relationship between the drawing stroke and the content) analyzed in step S107 or step S111.

[0081] (Step S119) The processing unit 110 generates and outputs a prompt to be input to the generating AI in order to execute processing based on the drawing content of the drawing stroke analyzed in step S103 and the analysis result (positional relationship between the drawing stroke and the content) analyzed in step S107 or step S111.

[0082] Next, with reference to Figure 9, a specific example of the operation of generating a prompt by the handwriting input execution process will be described. Figure 9 is a flowchart of a specific example of the handwriting input execution process according to this embodiment. Here, the example of generating a summary of the string (text) shown in Figure 6(C) will be explained. When the user completes drawing strokes indicating the range to be processed and drawing strokes for characters indicating the processing content using handwriting input, the process starts.

[0083] (Step S201) The processing unit 110 detects handwritten drawing strokes within the screen area of ​​the display 150 and obtains a bounding box indicating the range of the drawing strokes. Then, the processing unit 110 analyzes the drawing content of the detected drawing strokes. Here, drawing strokes indicating the range to be processed (enclosing lines) and drawing strokes of characters indicating the processing content (Summarize) are detected. The process in step S201 corresponds to specific examples of the processes in steps S101 and S103 shown in Figure 8. Then, the process proceeds to step S203.

[0084] (Step S203) The processing unit 110 analyzes the position of the drawing stroke (its positional relationship with the content displayed within the screen area) based on the screen capture image and the OS event log. Here, the drawing stroke (outline) indicating the area to be processed overlaps with the area of ​​the string displayed on the screen ("A keyboard is one of..."). The processing in step S203 corresponds to specific examples of the processing in steps S107 and S111 shown in Figure 8. Then, the process proceeds to step S205.

[0085] (Step S205) The processing unit 110 obtains a list of keywords (expected words) related to the process instructed by the user via handwriting input. Then, it proceeds to the process in step S205.

[0086] (Step S207) The processing unit 110 determines that the process is not one that can be implemented with a command that can directly execute the process on the application, but rather one that requires the generation AI, and generates and outputs a prompt to be input to the generation AI. For example, the processing unit 110 may generate a prompt using a fixed phrase because the word in the drawing stroke of the character (Summarize) detected in step S201 is present in the keywords obtained in step S205.

[0087] An example of generating a prompt using a predefined template is shown below. “Summarize following sentence:“A keyboard is one of...”” In this example, the instructions are given to the generation AI to summarize the string (text) to be processed.

[0088] Furthermore, the processing unit 110 may generate prompts using a generated Ai such as Vision LLM.

[0089] The following is an example of prompt generation using a generative AI. "On screen, user wrote commands in red lines. Please estimate user's intention and execute actions" In this example, the specific processing steps are not mentioned, but the instructions are given to the generating AI to infer the processing from the drawing strokes and execute it accordingly.

[0090] [Another example of handwritten input execution process] Next, we will explain another example of the handwriting input execution process with reference to Figures 10 to 12. Figure 10 shows a fourth example of the handwriting input execution process according to this embodiment. The example shown in Figure 10 is an example of a process that is executed when a drawing stroke entered by the user by hand overlaps with a blank area in the screen area. Note that this blank area may be a blank area within the application window. When the information processing device 10 detects a drawing stroke on a blank area, it executes a pre-configured process.

[0091] In the illustrated example, when the information processing device 10 detects a rectangular drawing stroke (a1, b1) in a blank area, as shown in (A), it executes a process to generate an image (a2, b2) that matches the aspect ratio of that rectangle, as shown in (B).

[0092] Figure 11 shows a fifth example of the handwritten input execution process according to this embodiment. The example shown in Figure 11 is an example in which content displayed within the screen area that is adjacent to a drawing stroke entered by the user by hand is the target of processing. For example, if the drawing stroke entered by the user by hand includes a mathematical formula (F1) and is adjacent to the area of ​​a number (N1) displayed within the screen area, the information processing device 10 will perform a calculation process based on the mathematical formula (F1) entered by the user by hand on the number (N1) and display the calculation result (C1).

[0093] Figure 12 shows a sixth example of the handwritten input execution process according to this embodiment. The example shown in Figure 12 is an example of a process that is executed when a drawing stroke indicating a range entered by the user by hand overlaps with the area of ​​a folder displayed within the screen area. When the information processing device 10 detects a drawing stroke indicating a range overlapping with the area of ​​a folder, it treats the overlapping folder as selected. When the information processing device 10 detects a drawing stroke of characters (zip) indicating the processing content next to the drawing stroke indicating the range, it executes a process to compress the selected folder in zip format. Note that this can be applied not only to folder selection but also to file selection. Furthermore, the processing is not limited to compression in zip format.

[0094] As described above, the information processing device 10 according to this embodiment detects drawing strokes corresponding to handwritten input within the screen area of ​​the display 150 and analyzes the content drawn by the detected drawing strokes. The information processing device 10 then executes processing based on the positional relationship between the detected drawing strokes and the content displayed in the screen area other than the drawing strokes, and the content drawn by the detected drawing strokes.

[0095] As a result, the information processing device 10 can perform the input on the content displayed on the screen simply by handwriting the content to be processed using a stylus or the like. Therefore, it can be applied to various functions, not just search functions, making it convenient for the user. Thus, the information processing device 10 can make better use of operations performed on the screen.

[0096] For example, the information processing device 10 causes the content displayed within the screen area to perform processing based on the drawing content created by the detected drawing strokes, for content where the detected drawing strokes overlap.

[0097] As a result, the information processing device 10 is convenient for the user because the user can simply input the content they want to process by handwriting it on top of the content displayed within the screen area using a stylus or the like, and the input will be processed on that content.

[0098] For example, if the detected drawing stroke overlaps with the area of ​​an application window displayed within the screen area, the information processing device 10 will execute the processing of the function of the application that corresponds to the drawing content.

[0099] As a result, the information processing device 10 is convenient for the user because the user can simply input the content they want to process by handwriting on the application window displayed within the screen area using a stylus or the like, and the input will be processed for that application. For example, as shown in Figure 5, the information processing device 10 is convenient for the user because the user can simply handwrite the time and items of the appointment they want to register on the window area of ​​an application that has email and calendar functions within the screen area, and the calendar registration screen will open, allowing the appointment to be registered.

[0100] For example, if the detected drawing stroke overlaps with an area of ​​an image displayed within the screen area, the information processing device 10 will perform image processing on the image based on the drawing content.

[0101] As a result, the information processing device 10 is convenient for the user because the user can simply input the content they want to process by handwriting on the image displayed within the screen area using a stylus or the like, and the input will be processed on that image. For example, as shown in Figure 4, the information processing device 10 is convenient for the user because a simplified image is generated simply by the user handwriting "Simple" on the image area within the screen area.

[0102] For example, if the detected drawing stroke overlaps with the area of ​​a string of characters displayed within the screen area, the information processing device 10 will perform processing on the string based on the drawing content.

[0103] As a result, the information processing device 10 is convenient for the user because the user can simply input the content they want to process by handwriting it on top of the string displayed within the screen area using a stylus or the like, and the input will be processed on that string.

[0104] Furthermore, the information processing device 10 analyzes the detected drawing strokes to extract drawing strokes that represent characters and drawing strokes that represent areas as drawing content. The information processing device 10 then performs processing based on the drawing strokes that represent characters for content that overlaps with the drawing strokes that represent areas among the content displayed within the screen area.

[0105] As a result, when a user wants to process a string of text displayed within the screen area, the information processing device 10 can perform the input by simply drawing the range of the string to be processed and the content to be processed using a stylus or the like, and the input will be processed on that string, making it convenient for the user. For example, as shown in Figure 6, the information processing device 10 can perform translation into Chinese, conversion to casual expressions, and generation of a summary simply by drawing a circle around the range of the string to be processed and writing "to CN," "Casual," or "Summarize" next to it as the content to be processed, making it convenient for the user.

[0106] Furthermore, instead of inputting the content to be processed (such as "to CN", "Casual", or "Summarize") by handwriting, it may be possible to input it by voice. For example, the information processing device 10 may be equipped with a voice input unit (not shown), such as a microphone, and when it extracts a drawing stroke indicating the range described above, it may accept voice input indicating the content to be processed on the content overlapping the drawing stroke indicating the range. This makes it convenient for the user, as when the user wants to process a string of characters displayed within the screen area, they can simply input the range of the string to be processed by handwriting using a stylus, and then input the content to be processed by voice, and the voice-inputted processing will be performed on that string. Furthermore, as shown in the example in Figure 12, if the drawing stroke indicating the range overlaps with the area of ​​a folder, the information processing device 10 may treat the overlapping folder as selected, and when the processing content is input by voice (zip), it may execute a process to compress the selected folder in zip format.

[0107] Furthermore, if the detected drawing stroke overlaps with a blank area within the screen area, the information processing device 10 may execute a pre-configured process.

[0108] As a result, the information processing device 10 is convenient for the user because it performs pre-configured processing simply by inputting by hand using a stylus or the like in a blank area of ​​the screen. For example, as shown in Figure 10, the information processing device 10 is convenient for the user because it generates an image that matches the aspect ratio of a rectangle simply by drawing a rectangle by hand in a blank area of ​​the screen.

[0109] Furthermore, the information processing device 10 may perform processing based on the drawing content for content adjacent to the detected drawing stroke among the content displayed within the screen area.

[0110] As a result, the information processing device 10 can input content to be processed not only on top of but also to the side of the content displayed within the screen area using a stylus or the like, allowing for flexible adaptation to the screen display situation, making it highly versatile and convenient for the user.

[0111] Furthermore, if the detected drawing stroke includes a mathematical formula and is adjacent to a region of a number displayed within the screen area, the information processing device 10 may cause the number to perform a calculation process based on the mathematical formula.

[0112] As a result, the information processing device 10 is convenient for the user because the calculation result is displayed simply by the user handwriting a mathematical formula next to the numbers displayed within the screen area. For example, the information processing device 10 is convenient for the user because, in cases where the price of a product is displayed excluding tax and the user wants to check the price including tax, or where the price is displayed including tax and the user wants to check the price excluding tax, or where the price is displayed before a discount and the user wants to check the price after a discount, the user can check the calculation result simply by handwriting a mathematical formula next to the price number without having to use a calculator.

[0113] Furthermore, the information processing device 10 generates and outputs a command to execute processing based on the drawing content.

[0114] As a result, the information processing device 10 can quickly generate and output a command to directly execute the process when generation AI is not required.

[0115] Furthermore, the information processing device 10 generates and outputs a prompt as an input sentence to the generating AI, which performs processing based on the drawing content.

[0116] As a result, the information processing device 10 can automatically generate prompts for input to the generation AI from the handwritten input when a generation AI is required for a particular process.

[0117] Furthermore, the control method in the information processing device 10 according to this embodiment includes the steps of: the processing unit 110 detecting a drawing stroke corresponding to a hand-drawn trajectory within the screen area; analyzing the drawing content of the detected drawing stroke; and causing the processing unit 110 to execute processing based on the positional relationship between the detected drawing stroke and content displayed in the screen area other than the drawing stroke, and the analyzed drawing content.

[0118] As a result, the control method in the information processing device 10 allows users to simply input the content they want to process by handwriting it on the screen using a stylus or the like, and the input will be processed on the content displayed in that area. Therefore, it can be applied not only to the search function but also to various other functions, making it convenient for the user. Thus, the control method in the information processing device 10 makes better use of operations performed on the screen.

[0119] Although embodiments of this invention have been described in detail above with reference to the drawings, the specific configurations are not limited to the embodiments described above, and include designs and the like that do not depart from the spirit of this invention. For example, the configurations described in the embodiments described above can be combined in any way.

[0120] Furthermore, although the above-described embodiment described an example in which the display 150 is a single display provided across the first housing 10A and the second housing 10B, the invention is not limited to this. For example, the information processing device 10 may have two displays, one in the first housing 10A and one in the second housing 10B. Alternatively, the information processing device 10 may have a display provided in only one of the first housing 10A and the second housing 10B (for example, only in the first housing 10A).

[0121] Furthermore, the information processing device 10 is not limited to a clamshell-type (notebook-type) PC, but may also be a tablet-type PC or a desktop PC, for example.

[0122] Furthermore, although the above-described embodiment described an example of a touch panel type display in which the input unit (touch sensor) and the display unit (display) are integrated, a non-touch panel type display without an input unit (touch sensor) may also be used. In that case, instead of an input unit (touch sensor), an input device that accepts operation may be used, such as a touchpad or a mouse.

[0123] The information processing device 10 described above has a computer system inside. The processing in each configuration of the information processing device 10 may be performed by recording a program for realizing the functions of each configuration of the information processing device 10 onto a computer-readable recording medium, loading the program recorded on this recording medium into the computer system, and executing it. Here, "loading the program recorded on the recording medium into the computer system and executing it" includes installing the program into the computer system. Here, "computer system" includes hardware such as the OS and peripheral devices. Furthermore, "computer system" may include multiple computer devices connected via a network including communication lines such as the Internet, WAN, LAN, and dedicated lines. Also, "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and storage devices such as hard disks built into the computer system. Thus, the recording medium storing the program may be a non-transient recording medium such as a CD-ROM.

[0124] Furthermore, the recording medium also includes internal or external recording media accessible from the distribution server for distributing the program. The program may be divided into multiple parts, downloaded at different times, and then combined in each configuration of the information processing device 10. The distribution servers for each of the divided programs may also be different. Moreover, "computer-readable recording media" includes volatile memory (RAM) within computer systems that act as servers or clients when a program is transmitted over a network, which retains the program for a certain period of time. The program itself may also be intended to implement some of the functions described above. Furthermore, the program may be a so-called differential file (differential program) that can implement the functions described above in combination with a program already recorded in the computer system.

[0125] Furthermore, some or all of the functions of the information processing device 10 in the above-described embodiment may be implemented as an integrated circuit such as an LSI (Large Scale Integration). Each function may be individually processorized, or some or all of them may be integrated into a single processor. In addition, the method of implementing the integrated circuit is not limited to LSIs; it may also be implemented using dedicated circuits or general-purpose processors. Furthermore, if an integrated circuit technology that can replace LSIs emerges due to advances in semiconductor technology, an integrated circuit using that technology may be used. [Explanation of Symbols]

[0126] 10 Information Processing Device, 10A First Enclosure, 10B Second Enclosure, 30 Server, 31 Generation AI, 11 Communication Unit, 12 RAM, 13 Storage Unit, 14 Speaker, 15 Display Unit, 16 Camera, 150 Display, 150A First Screen Area, 150B Second Screen Area, 155 Touch Sensor, 18 Control Unit, 110 Processing Unit, 111 Drawing Stroke Acquisition Unit, 112 Screen Capture Unit, 113 OS Event Acquisition Unit, 114 Drawing Analysis Unit, 115 Drawing Position Analysis Unit, 116 Execution Control Unit, 117 User Command Analysis Unit, 118 Prompt Generation Unit, 119 Command Processing Unit, SYS Information Processing System

Claims

1. A drawing detection unit that detects drawing strokes corresponding to hand-drawn paths within the screen area, A drawing analysis unit analyzes the drawing content of character drawing strokes that indicate processing content among the drawing strokes detected by the drawing detection unit, An execution control unit, which causes the execution control unit to execute processing of the processing content indicated by the drawing content based on the positional relationship between the drawing stroke detected by the drawing detection unit and the content displayed in the screen area other than the drawing stroke, and the drawing content analyzed by the drawing analysis unit, An information processing device equipped with the following features.

2. The execution control unit, For content displayed within the aforementioned screen area, the drawing analysis unit performs processing based on the drawing content analyzed for content where the drawing strokes detected by the drawing detection unit overlap. The information processing apparatus according to claim 1.

3. The execution control unit, If the drawing stroke detected by the drawing detection unit overlaps with the area of ​​a string of characters displayed within the screen area, the drawing analysis unit will perform processing on the string of characters based on the analyzed drawing content. The information processing apparatus according to claim 2.

4. The execution control unit, If the drawing stroke detected by the drawing detection unit overlaps with the area of ​​an image displayed within the screen area, the drawing analysis unit performs image processing on the image based on the analyzed drawing content. The information processing apparatus according to claim 2.

5. The execution control unit, Among the content displayed within the aforementioned screen area, the content adjacent to the drawing stroke detected by the drawing detection unit is processed based on the drawing content analyzed by the drawing analysis unit. The information processing apparatus according to claim 1.

6. A drawing detection unit that detects drawing strokes corresponding to hand-drawn paths within the screen area, A drawing analysis unit analyzes the drawing content based on the drawing stroke detected by the drawing detection unit, An execution control unit that performs processing based on the positional relationship between the drawing stroke detected by the drawing detection unit and the content displayed in the screen area other than the drawing stroke, and the drawing content analyzed by the drawing analysis unit, Equipped with, The aforementioned drawing analysis unit, By analyzing the drawing strokes detected by the drawing detection unit, the drawing strokes representing characters and the drawing strokes representing the selected area enclosed by handwriting are extracted as the drawing content. The execution control unit, For content displayed within the aforementioned screen area that overlaps with the drawing stroke indicating the range detected by the drawing detection unit, the drawing detection unit performs processing based on the drawing stroke indicating the character detected. An information processing device.

7. A drawing detection unit that detects drawing strokes corresponding to hand-drawn paths within the screen area, A drawing analysis unit analyzes the drawing content based on the drawing stroke detected by the drawing detection unit, An execution control unit that performs processing based on the positional relationship between the drawing stroke detected by the drawing detection unit and the content displayed in the screen area other than the drawing stroke, and the drawing content analyzed by the drawing analysis unit, Equipped with, The execution control unit, If the drawing stroke detected by the drawing detection unit overlaps with a blank area within the screen area, and the drawing stroke represents a frame, then content is generated to match the frame and displayed in accordance with the position of the frame. Information processing device.

8. A drawing detection unit that detects drawing strokes corresponding to hand-drawn paths within the screen area, A drawing analysis unit analyzes the drawing content based on the drawing stroke detected by the drawing detection unit, An execution control unit that performs processing based on the positional relationship between the drawing stroke detected by the drawing detection unit and the content displayed in the screen area other than the drawing stroke, and the drawing content analyzed by the drawing analysis unit, Equipped with, The execution control unit, If the drawing stroke detected by the drawing detection unit includes a mathematical formula and is adjacent to a region of a number displayed within the screen area, the unit will perform a calculation process based on the mathematical formula on the number. Information processing device.

9. The execution control unit, The drawing analysis unit generates and outputs a command to execute processing based on the drawing content analyzed by the drawing analysis unit. An information processing apparatus according to any one of claims 1, 6, 7, and 8.

10. The execution control unit, The drawing analysis unit generates and outputs a prompt as an input sentence to the generating AI that performs processing based on the drawing content analyzed by the drawing analysis unit. An information processing apparatus according to any one of claims 1, 6, 7, and 8.

11. A control method in an information processing device, The drawing detection unit detects drawing strokes corresponding to hand-drawn paths within the screen area, The drawing analysis unit performs the step of analyzing the drawing content of the character drawing strokes that indicate the processing content among the drawing strokes detected by the drawing detection unit, The execution control unit performs the processing of the processing content indicated by the drawing content based on the positional relationship between the drawing stroke detected by the drawing detection unit and the content displayed in the screen area other than the drawing stroke, and the drawing content analyzed by the drawing analysis unit. A control method including

Images