Method and system for electronic document editing based on multi-sensory interface and artificial intelligence

Abstract

A multi-sensory interface and an artificial intelligence-based electronic document editing method according to the present invention may include the steps of: executing an electronic document editing program on a first device; providing a user interface including a function selection area and an editing area to a second device in conjunction with the execution of the electronic document editing program on the first device; recognizing a pointing motion of a user pointing to one area among the function selection areas of the second device through a camera; identifying an editing function corresponding to one area among the function selection areas of the second device based on the pointing motion; applying the identified editing function to the electronic document editing program running on the first device to edit the electronic document being output on the display of the second device; and providing information corresponding to the result of editing the electronic document output on the display of the first device to the editing area of ​​the second device.

Description

Technology Field

[0001] The present invention relates to a multi-sensory interface, an artificial intelligence-based electronic document editing method, and a system. Background Technology

[0002] With the advancement of technology, the widespread adoption of electronic devices (e.g., smartphones, tablet PCs, automated devices, etc.) has become commonplace, and consequently, dependence on electronic devices is gradually increasing in many aspects of daily life.

[0003] As technology advances, the utilization of digital devices is increasing. In particular, electronic devices (such as smartphones and tablet PCs) are used for a wide range of purposes, including not only communication functions like phone calls and text messaging but also content consumption, information retrieval, and document creation and editing, establishing themselves as essential tools across daily life and work.

[0004] Amidst this trend, as various types of electronic documents such as electronic contracts, reports, personal statements, meeting minutes, and study materials are created and utilized in the digital environment, the need for systems that provide users with an easier and more intuitive editing environment is gradually increasing.

[0005] In particular, traditional visual-based editing methods have limitations in input means or information recognition methods, making it difficult for all users to experience the same level of convenience. Accordingly, there is a need for an electronic document editing system that can utilize various sensory information, such as touch and hearing, to improve interaction between the user and the system. The problem to be solved

[0006] The present invention relates to a multi-sensory interface, an artificial intelligence-based electronic document editing method, and a system that provides an environment for editing electronic documents based on various senses such as sight, touch, and hearing.

[0007] In particular, this invention relates to a multi-sensory interface and an AI-based electronic document editing method and system that support various users, including those with limited visual information, in efficiently and intuitively creating and editing electronic documents.

[0008] Furthermore, the present invention relates to a multi-sensory interface that supports a user's electronic document editing using artificial intelligence technology, and an artificial intelligence-based electronic document editing method and system. means of solving the problem

[0009] A multi-sensory interface and an artificial intelligence-based electronic document editing method according to the present invention may include the steps of: executing an electronic document editing program on a first device; providing a user interface including a function selection area and an editing area to a second device in conjunction with the execution of the electronic document editing program on the first device; recognizing a pointing motion of a user pointing to one area among the function selection areas of the second device through a camera; identifying an editing function corresponding to one area among the function selection areas of the second device based on the pointing motion; applying the identified editing function to the electronic document editing program running on the first device to edit the electronic document being output on the display of the second device; and providing information corresponding to the result of editing the electronic document output on the display of the first device to the editing area of ​​the second device.

[0010] In this case, a plurality of tactile stimulus information corresponding to each of a plurality of editing functions is provided in the function selection area of ​​the second device, and among the plurality of tactile stimulus information, the tactile stimulus information corresponding to the specified editing function is continuously provided in the function selection area of ​​the second device, and the tactile stimulus information corresponding to an editing function different from the specified editing function may disappear from the function selection area of ​​the second device.

[0011] Furthermore, when the specified editing function is selected by the user's pointing motion in the function selection area of ​​the second device, the electronic document editing program running on the first device provides a function graphic object associated with the specified editing function, and the user interface provided on the second device further includes a work area, and tactile function information corresponding to the function graphic object may be provided in the work area of ​​the second device.

[0012] Furthermore, the method may further include a step of providing guide information related to the specified editing function through a speaker, wherein the step of providing the guide information may include: generating a prompt containing information about the function graphic object and processing it as input to an artificial intelligence model; obtaining the guide information describing the function graphic object from the artificial intelligence model; and converting the guide information into voice information and providing it through the speaker.

[0013] Furthermore, when a pointing motion of the user pointing at the work area of ​​the second device is recognized using the camera, the electronic document is edited through the electronic document editing program running on the first device in conjunction with the pointing motion for the work area, and the edited electronic document can be provided to the editing area of ​​the second device.

[0014] Furthermore, the user interface provided by the second device includes a tactile interface, and in the step of providing it to the editing area, the electronic document edited by the first device is converted into tactile stimulus information that can be provided to the tactile interface, and the tactile stimulus information can be provided on the editing area of ​​the second device as the result of editing the electronic document.

[0015] Furthermore, in the step of providing to the editing area, the electronic document before editing and the edited electronic document are compared to identify information changed according to the specified editing function, and based on the identified information, an update target area among the editing areas of the second device is specified, and tactile stimulation information corresponding to the changed information can be provided to the update target area.

[0016] Furthermore, in the step of recognizing the user's pointing motion, the function selection area of ​​the second device and a preset marker are identified from the image, and in the step of specifying the editing function, the specified editing function can be determined based on the relative positional relationship between the function selection area of ​​the second device and the marker.

[0017] Meanwhile, the multi-sensory interface and artificial intelligence-based electronic document editing system according to the present invention comprises a first device on which an electronic document editing program is executed; a second device different from the first device; and a control unit that provides a user interface including a function selection area and an editing area to the second device in conjunction with the execution of the electronic document editing program on the first device. The control unit recognizes a user’s pointing motion pointing to one area among the function selection areas of the second device through a camera, and based on the pointing motion, identifies an editing function corresponding to one area among the function selection areas of the second device, applies the identified editing function to the electronic document editing program running on the first device, edits the electronic document being output on the display of the second device, and provides information corresponding to the result of editing the electronic document output on the display of the first device to the editing area of ​​the second device.

[0018] Meanwhile, the program according to the present invention is executed by one or more processes in an electronic device and is a program stored on a computer-readable medium, and may include instructions for performing the steps of: executing an electronic document editing program on a first device; providing a user interface including a function selection area and an editing area to a second device in conjunction with the execution of the electronic document editing program on the first device; recognizing a pointing motion of a user pointing to one area among the function selection areas of the second device through a camera; specifying an editing function corresponding to one area among the function selection areas of the second device based on the pointing motion; applying the specified editing function to the electronic document editing program running on the first device to edit the electronic document being output on the display of the second device; and providing information corresponding to the result of editing the electronic document output on the display of the first device to the editing area of ​​the second device. Effects of the invention

[0019] The multi-sensory interface and artificial intelligence-based electronic document editing method and system according to the present invention provide a user interface including a function selection area and an editing area to a second device in conjunction with the execution of the electronic document editing program on a first device, and can smoothly perform the creation, modification, and editing of electronic documents under various user conditions (e.g., visually impaired) or various device configuration environments.

[0020] Furthermore, the multi-sensory interface and artificial intelligence-based electronic document editing method and system according to the present invention recognize a user’s pointing motion pointing to one area among the function selection areas of the second device through a camera, specify an editing function corresponding to one area among the function selection areas of the second device based on the pointing motion, and apply the specified editing function to the electronic document editing program running on the first device to edit the electronic document being displayed on the display of the second device. Through this, the present invention enables even visually impaired individuals with limitations in recognizing visual information to intuitively select editing functions using only hand movements and to perform editing of the content of an electronic document without relying on sight, thereby providing enhanced accessibility and convenience to users with sensory limitations, including the visually impaired.

[0021] Furthermore, the multi-sensory interface and artificial intelligence-based electronic document editing method and system according to the present invention provide information corresponding to the electronic document editing result output on the display of the first device to the editing area of ​​the second device, and even a visually impaired person who has difficulty directly perceiving visual information can recognize the editing result in real time through the editing area of ​​the second device and can easily edit the electronic document independently without the help of another person. Brief explanation of the drawing

[0022] FIGS. 1 and 2 are conceptual diagrams for explaining a multi-sensory interface and an artificial intelligence-based electronic document editing system according to the present invention. FIG. 3 is a flowchart illustrating a multi-sensory interface and an artificial intelligence-based electronic document editing method according to the present invention. FIGS. 4a, FIGS. 4b, and FIGS. 4c are conceptual diagrams for illustrating the interface provided in the present invention. FIGS. 5A, FIGS. 5B, and FIGS. 5C are conceptual diagrams illustrating a method for editing an interface-based electronic document in the present invention. FIGS. 6 and FIGS. 7 are conceptual diagrams illustrating a method for editing an artificial intelligence-based electronic document in the present invention. Specific details for implementing the invention

[0023] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components are assigned the same reference number regardless of the drawing symbols, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not have distinct meanings or roles in themselves. Furthermore, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.

[0024] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0025] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0026] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0027] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0028] The present invention relates to a multi-sensory interface and an artificial intelligence-based electronic document editing method and system, and in particular, can provide an environment in which visually impaired individuals with limited visual information can efficiently and intuitively create and edit electronic documents.

[0029] In the present invention, “electronic document” refers to document data in a digital form that is created or stored on a computer or digital device and may include text, images, shapes, tables, or multimedia elements. The electronic document may be in a document format that is editable in an electronic document application. For example, the electronic document may be composed of pptx, .docx, .pdf, .hwp, etc., and may include data objects that contain the structure, layout, and content composition of the document along with a visual representation on the screen.

[0030] Hereinafter, a multi-sensory interface and an artificial intelligence-based electronic document editing method will be described together with the attached drawings. Figures 1 and 2 are conceptual diagrams for explaining a multi-sensory interface and an artificial intelligence-based electronic document editing system according to the present invention. Figure 3 is a flowchart for explaining a multi-sensory interface and an artificial intelligence-based electronic document editing method according to the present invention, Figures 4a, 4b, and 4c are conceptual diagrams for explaining an interface provided in the present invention, Figures 5a, 5b, and 5c are conceptual diagrams for explaining a method of editing an interface-based electronic document in the present invention, and Figures 6 and 7 are conceptual diagrams for explaining a method of editing an artificial intelligence-based electronic document in the present invention.

[0031] As illustrated in FIG. 1, the multisensory interface and artificial intelligence-based electronic document editing system according to the present invention (hereinafter referred to as the “multisensory interface-based electronic document editing system,” 1000) may include at least one of a first device (100), a second device (200), a camera (300), a marker (400), a keyboard (500), a speaker (600), an artificial intelligence model (700), and a control unit (800).

[0032] The first device (100) refers to an electronic device (or user terminal) on which an electronic document editing program (100b) is executed, and there are no restrictions on the type thereof, and it may be a notebook computer, a laptop computer, a slate PC, a tablet PC, an ultrabook, a desktop computer, a mobile phone, a smartphone, etc.

[0033] An electronic document editing program (100b) may be installed on the first device (100).

[0034] In the present invention, “electronic document editing program” refers to software that supports electronic document editing (or creation) operations, and may be various, such as a word processing-based electronic document editing program (e.g., Microsoft Word), a slide-based electronic document editing program (e.g., Microsoft PowerPoint), or a web-based collaborative electronic document editing tool (e.g., Google Docs).

[0035] In the present invention, “editing an electronic document” can be understood as creating (or writing) an electronic document using an electronic document editing program, or changing (modifying, adding, or deleting) the content of an already created electronic document. For example, using a slide-based electronic document editing program (e.g., Microsoft PowerPoint), inserting text, images, or shapes into a slide, changing the position of an existing inserted image, or changing the color or size of shapes can be described as editing an electronic document.

[0036] As illustrated in FIG. 2, the second device (200) is an independent electronic device physically separated from the first device (100), and can be understood as a device that performs the role of supporting a user to edit an electronic document by linking with an electronic document editing program (100b) running on the first device (100).

[0037] The second device (200) may include a tactile interface. For example, the second device (200) may include at least one of a tactile display, a haptic display, a refreshable Braille display, or a tactile feedback interface.

[0038] The present invention is described primarily in the case where the second device (200) is implemented as a type of tactile display. The second device (200) includes a tactile interface, and through the interface, can provide the user with tactile stimulation information regarding information or editing functions included in an electronic document.

[0039] Here, tactile stimulus information may refer to physical stimulus information, such as vibrations, protrusions, and changes in surface texture, configured to be perceived through the user's skin senses.

[0040] In the present invention, “tactile stimulation information” is generated by the control unit (800) according to the activation of functions of the electronic document editing program (100b), editing results, selection of editing functions, etc., and can be provided to the user in real time through the tactile interface of the second device (200).

[0041] For example, the second device (200) may include a Braille-based tactile display having a plurality of protruding cells arranged therein, and cells corresponding to specific information or functions of an electronic document are electronically controlled to protrude, and the user can perceive the information through the tactile sense of the fingertips.

[0042] The second device (200) may be provided with a user interface (200a) corresponding to an electronic document editing program (100b).

[0043] Here, the “user interface (200a)” can be understood as a tactile-based UI dedicated to editing electronic documents, provided to the user through a tactile interface included in the second device (200). The tactile interface is a tactile display device composed of a Braille-based protrusion cell array or other tactile feedback elements, and the user interface (200a) refers to actual interface content implemented through a pattern of tactile stimulus information on the tactile interface. That is, the tactile interface is a hardware display means for implementing tactile stimulus information, and the user interface (200a) can be understood as a soft interface including tactile-based UI components such as a function selection area and an editing area physically expressed on the display. In the present invention, the user interface (200a) of the second device may also be referred to as a “multisensory interface.”

[0044] In the present invention, the editing function (e.g., insertion function (text insertion, shape insertion, image insertion, shape insertion, etc.)) of an electronic document editing program running on a first device (100) and the information contained in the electronic document are converted into tactile haptic stimulation information and provided through a second device (200), thereby supporting users who have difficulty perceiving visual information to intuitively understand the flow and results of document editing.

[0045] Furthermore, the second device (200) may further include an input interface. Such an input interface may include mechanical or electrical input means. More specifically, the input interface may include a button to which a specific function can be selected upon direct contact or pressing action by a user. In the present invention, the input interface may be referred to as a “button,” “input means,” “physical input means,” “mechanical input means,” etc.

[0046] The camera (300) can capture the user interface (200a) provided on the second device (200). The type of camera (300) capable of capturing user actions (or gestures, for example, pointing actions pointing at the user interface) on the user interface (200a) of the second device (200) is not limited. For example, the camera (300) may be a webcam equipped on the first device (100) or connected to the first device (or other electronic device).

[0047] As illustrated in FIG. 2, the marker (400) may refer to a specific visual mark attached to or displayed on the user's finger or hand. The marker (400) may be used to accurately recognize (specifically) the user's pointing motion and pointing position.

[0048] A keyboard (500) may refer to an input device configured to allow a user to input characters or execute specific functions. In the present invention, the keyboard (500) may be provided in or connected to the first device (100).

[0049] A speaker (600) may refer to an output device that converts an electrical signal into an acoustic signal to provide auditory information to a user. In the present invention, the speaker (600) may be provided in at least one of the first device (100) and the second device (200). Additionally, the speaker (600) may be connected to at least one of the first device (100) and the second device (200).

[0050] Furthermore, the artificial intelligence model (700) is based on a pre-trained language model or a conversational artificial intelligence algorithm and can support the user's editing of electronic documents through conversation or question-and-answer with the user.

[0051] The artificial intelligence model (700) can provide an explanation of the editing status, composition content, or specific functions provided by the electronic document editing program.

[0052] The artificial intelligence model (700) can explain related visual information or file information, etc., to the user based on the result of executing the editing function when the user executes a specific editing function (e.g., insertion function (text insertion, shape insertion, image insertion, shape insertion)) in an electronic document editing program.

[0053] For example, if the "Insert Image" editing function is selected (or activated) by the user, the electronic document editing program (100b) can output an image file stored in the first device (100) to the display (100a) of the first device (100). A user (U) with limited visual information cannot view the image files output to the display (100a) of the first device. In this case, the artificial intelligence model (700) can provide the filename, extension, creation date, or description of the image file content (e.g., 'photo of a person smiling', 'image of a shape on a gray background', etc.).

[0054] The artificial intelligence model (700) can analyze not only the formal information (filename, extension, storage location, etc.) of the image file to be inserted, but also the visual content of the image, and generate natural language-based image file content description information (or image description information) based on objects, scenes, backgrounds, people, color compositions, etc. included in the image. For example, it can provide the user with a language-based description of images that are difficult for the user to visually access, such as “a scene of a person sitting at a desk,” “a landscape with a blue sky and green fields,” or “a smartphone on a white background in a product photo.”

[0055] Furthermore, when an editing function is selected in an electronic document editing program (100b), the artificial intelligence model (700) can explain the basic operation, purpose of use, and selectable sub-options of the editing function in natural language. For example, if the user selects the “Insert” menu, the artificial intelligence model (700) can generate guide information (or guidance information or explanatory information) such as “This menu includes insertion functions for images, shapes, tables, charts, etc.” The artificial intelligence model (700) can also continuously guide users on sub-functions (e.g., types such as rectangles, circles, and lines when ‘Shapes’ is selected). In other words, the artificial intelligence model (700) can assist the user in their editing work by interacting with the user’s electronic document editing work. Through this, the user can perceive the structure and usage method of the editing tool based on hearing or text without visual navigation.

[0056] The control unit (130) may be configured to control the overall operation of the electronic document editing system (1000) related to the present invention. The control unit (800) may control the electronic document editing system (1000) according to the present invention by utilizing at least one CPU among the first device (100) and the second device (200).

[0057] For example, the control unit (800) is installed in the first device (100) and can receive an image from the camera (300) using the CPU of the first device (100) and recognize a user's pointing motion in the image. The control unit (800) can also use the CPU of the first device (100) to specify an editing function corresponding to the user's pointing motion and control the editing of the electronic document in the electronic document editing program (100b) running on the first device (100).

[0058] The control unit (800) can provide information corresponding to the edited electronic document to the second device (200) based on the fact that editing of the electronic document is performed in the electronic document editing program (100b) of the first device (100).

[0059] The control unit (800) is connected to information via wired or wireless connection with at least one of the first device (100), second device (200), camera (300), keyboard (500), and speaker (600), and can transmit and receive data with them.

[0060] Hereinafter, a method for editing an electronic document will be described based on the configurations of the electronic document editing system (1000) according to the present invention. In particular, the present invention can provide an intuitive and efficient editing interface that allows a user who has difficulty perceiving visual information (e.g., a visually impaired person) to edit an electronic document by themselves without the help of others.

[0061] In the present invention, a process of executing an electronic document editing program on a first device may be carried out (S310, see FIG. 3).

[0062] As previously explained, the first device (100) may refer to an electronic device capable of running an electronic document editing program (100b). For example, the first device (100) may be a notebook computer, a laptop computer, a slate PC, a tablet PC, an ultrabook, a desktop computer, a mobile phone, a smartphone, etc.

[0063] An electronic document editing program (100b) is installed in the first device (100), and the electronic document editing program can be executed based on a user's execution command.

[0064] In the present invention, a process of providing a user interface including a function selection area and an editing area to a second device in conjunction with the execution of an electronic document editing program on a first device may be carried out (S320, see FIG. 3).

[0065] The control unit (800) may provide a user interface (200a) to the second device (200). The user interface (200a) may be configured to operate an electronic document editing program (100b) running on the first device (100) or to perform work using the electronic document editing program (100b).

[0066] As illustrated in FIGS. 4a and 4b, the user interface (200a) may include at least one of a function selection area (210, 220), an editing area (230), and a work area (240). The user interface (200a) may correspond to an electronic document editing program (100b) installed on the display (100a) of the first device. For example, as illustrated in FIG. 4c, the electronic document editing program (100b) may include a plurality of areas (110 to 180). These plurality of areas may include a menu area, a work area, a slide area, a format area, etc.

[0067] The function selection area (210, 220) may be provided with editing functions provided by an electronic document editing program. In the function selection area (210, 220), some of the multiple editing functions provided by the electronic document editing program (100b) may be provided in the form of tactile stimulus information so that the user can select them.

[0068] The “editing function” described in the present invention refers to an operation command or execution item of a functional unit provided to enable a user to edit an electronic document through an electronic document editing program (100b) executed on a first device (100).

[0069] For example, editing functions may include items such as insertion functions (e.g., inserting text, inserting shapes, inserting images, inserting forms), creating tables, aligning, grouping, ungrouping, undoing, redoing, deleting, copying, pasting, setting links, changing backgrounds, adding new slides, saving, and exiting.

[0070] The function selection area may include at least one of the first function selection area (210) and the second function selection area (220).

[0071] Tactile stimulation information corresponding to the main editing function may be provided in the main function selection area (210). At least some of the multiple editing functions provided in the electronic document editing program (100b) may be set as the main editing function.

[0072] Tactile stimulation information corresponding to each main editing function may be provided in the main function selection area (210). For example, in the first function selection area (210), the editing function names “Insert,” “Group,” “Sort,” “Revert,” and “Redo” may be provided as tactile stimulation information.

[0073] The control unit (800) can convert each of the main editing function information into tactile stimulation information and implement the tactile stimulation information on the tactile interface of the second device (200). For example, the control unit (800) can provide the main editing function information by controlling a plurality of protrusion cell arrays corresponding to the main function selection area (210).

[0074] The sub-function selection area (220) can be understood as an area that provides function items associated with the editing function specified in the main function selection area (210) as tactile stimulus information. For example, when the editing function “Insert” is selected, the sub-function selection area (220) may provide detailed function items associated with “Insert,” such as “Shape,” “Image,” “Confirm,” and “Cancel,” in the form of tactile stimulus information.

[0075] The editing area (230) is an area that converts information corresponding to the visual content of an electronic document into tactile stimulus information based on the result of editing through the electronic document editing program (100b). For example, when components such as shapes, images, text, and tables included in a slide within an electronic document are edited in the first device (100), the edited electronic document can be reflected in the editing area (230) of the second device (200).

[0076] The work area (240) is an area where function items or information linked to an editing function specified through function selection are provided as tactile stimulus information. For example, if the “Insert Image” function is selected, the work area (240) may provide information corresponding to a saved image insertion function item (e.g., “My picture”) or a new image insertion function item (e.g., “My picture”). As another example, the work area (240) may provide information such as an “image file list” or an “image file filename”.

[0077] In the present invention, the function selection area (210, 220), the editing area (230), and the work area (240) can be interconnected. The control unit (800) can change the information provided to the function selection area (210, 220), the editing area (230), and the work area (240) based on the recognition of a user pointing action for at least one of the function selection area (210, 220), the editing area (230), and the work area (240).

[0078] In the present invention, a process of recognizing a user's pointing motion pointing to one area of ​​a function selection area of ​​a second device through a camera can be performed (S330, see FIG. 3).

[0079] In the present invention, the camera (300) can photograph the second device (200). As illustrated in (a) of FIG. 5a, with the user interface (200a) provided on the second device (200), the user can point to at least one area of ​​the user interface (200a). The control unit (800) can recognize the pointing motion of the user pointing to the function selection area from the image captured by the camera (300).

[0080] The control unit (800) can recognize a user's pointing motion in an image obtained from a camera (300) based on the selection of a button (250) provided in the second device (200).

[0081] The control unit (800) can control the camera (300) to capture an image from the camera (300) based on the selection of the button (250). Additionally, the control unit (800) can specify one of the images captured by the camera (300) (e.g., an image frame if a video is being captured) based on the selection of the button (250).

[0082] The control unit (800) can recognize the user's pointing motion from the image. In this case, the control unit (800) can recognize the user's pointing motion based on a predefined identification mark (marker, 400).

[0083] Here, “predefined identification markers” may refer to visual markers specifically configured to recognize a user’s pointing motion from an image. Predefined identification markers can be defined in various forms, such as color, shape, pattern, QR marker, label, LED display, or dedicated marker sticker. Such identification markers may be referred to by various names, such as “identification marker,” “tracking marker,” “reference marker,” or “visual recognition tag.” For the sake of convenience of explanation, predefined identification markers will be referred to as “markers” in the following description.

[0084] The control unit (800) can detect (identify) a marker (or an object corresponding to the marker) in the image. Based on the marker detection result, the control unit (800) can specify an area in the user interface (200a) that is indicated by a user action.

[0085] In the present invention, based on a pointing operation, a process of specifying an editing function corresponding to one area of ​​the function selection area of ​​the second device may be performed (S340, see FIG. 3).

[0086] The control unit (800) can identify a pointing area in the function selection area (210) and specify an editing function corresponding to the pointing area.

[0087] In the function selection area (210), multiple tactile stimulation information corresponding to each of the multiple editing functions may be provided.

[0088] For example, as illustrated in (a) of FIG. 5a, tactile stimulus information corresponding to “insert,” “group,” “align,” “revert,” and “redo” can be provided in the function selection area (210). The control unit (800) can determine which editing function among “insert,” “group,” “align,” “redo,” and “redo” corresponds to the pointing area recognized from the image. If the pointing area corresponds to “insert,” the control unit (800) can specify the editing function of “insert.”

[0089] In another example, as illustrated in (a) of FIG. 5b, if the user pointing area for the function selection area (210) corresponds to “text insertion,” the control unit (800) can specify the editing function of “text insertion.”

[0090] Meanwhile, in the function selection area (210) of the second device (200), multiple tactile stimulation information corresponding to each of the multiple editing functions may be provided.

[0091] In the present invention, tactile stimulus information may refer to physical stimulus information such as vibration, protrusion, or change in surface texture configured to be perceived through the user's skin sense.

[0092] The second device (200) may include a Braille-based tactile display having a plurality of protruding cells arranged therein. In the function selection area (210), at least some of the plurality of protruding cells may protrude to provide information (e.g., names of editing functions) corresponding to each of the plurality of editing functions as tactile stimulation information. For example, in the function selection area (210), protruding cells may protrude to provide Braille codes corresponding to each of “Insert,” “Group,” “Align,” “Undo,” and “Redo” as tactile stimulation information.

[0093] The control unit (800) can control the function selection area (210) so that when a user’s pointing motion is recognized, only tactile stimulation information of a specific editing function is provided.

[0094] The control unit (800) can control the tactile stimulus information corresponding to a specific editing function to continue being provided in the pointing area, and control the tactile stimulus information corresponding to an editing function different from the specific editing function to disappear from the function selection area (210) of the second device.

[0095] For example, when the “insert” editing function is specified, the control unit (800) can ensure that the protrusion cell corresponding to “insert” continues to be provided in the function selection area (210). And, the control unit (800) can ensure that the protrusion cells corresponding to the remaining editing functions are no longer derived in the function selection area (210). The user can intuitively recognize whether the editing function they have pointed to has been accurately specified.

[0096] In the present invention, a specific editing function is applied to an electronic document editing program running on a first device, and a process of editing an electronic document being output on the display of a second device may be performed (S350. See FIG. 3).

[0097] The control unit (800) can control the electronic document editing program running on the first device (100) so that editing according to a specific editing function is performed.

[0098] In the storage unit (120), there may be matching information in which control information (or commands) are matched for each of the multiple editing functions. This matching information may include commands that command electronic document editing according to a specific editing function in the electronic document editing program (100b).

[0099] The control unit (800) can control the editing of an electronic document in an electronic document editing program by referring to matching information and using a command corresponding to a specified editing function.

[0100] The control unit (800) can edit an electronic document according to a specified editing function in the first device (100) while only tactile stimulation information corresponding to a specified editing function is provided in the function selection area (210) of the second device (200).

[0101] For example, if the specified editing function is “insert,” the control unit (800) can control the “insert” function to be performed in the electronic document editing program (100b). As illustrated in (b) of FIG. 5a, an electronic document (510) being output on the display (100a) of the first device (100) can be edited as the specified editing function, “insert (e.g., image insertion),” is performed. Based on the electronic document editing being performed, the edited electronic document (510a) can be displayed (output) on the display (100a) of the first device (100). The edited electronic document (510a) may include a new image (511) based on the specified editing function “insert (e.g., image insertion).”

[0102] In another example, if the specified editing function is “insert (e.g., text insertion),” the control unit (800) can control the “insert (e.g., text insertion)” function to be performed in the electronic document editing program (100b). As illustrated in (b) of FIG. 5b, new text (e.g., “meeting materials”, 521) may be inserted into the electronic document (520) being output on the display (100a) of the first device (100) based on the “text insertion” editing being performed. An edited electronic document (520a) containing the new text (521) may be output on the display (100a) of the first device (100).

[0103] Meanwhile, when a specific editing function is selected by a user's pointing action in the function selection area (210) of the second device, an electronic document editing program (100b) running on the first device may provide a function graphic object associated with the specific editing function.

[0104] In the present invention, an electronic document editing program (100b) executed on a first device (100) may visually provide a subsequent action item or user input item for performing a specific editing function selected by a user. Such subsequent function items may be implemented in various forms depending on the selected editing function, and

[0105] For example, when the “insert” function is selected by the user, the electronic document editing program (100b) may provide UI items for image selection, such as a file open dialog box, a recent image thumbnail, and a save location navigation menu.

[0106] In the present invention, subsequent function items provided based on the selection of an editing function in an electronic document editing program (100b) are described as “functional graphic objects.” That is, function items provided on the display (100a) of the first device (100), such as a file open dialog box and a recent image thumbnail, are named as functional graphic objects in the present invention.

[0107] The control unit (800) can provide tactile function information corresponding to a function graphic object to the work area (240) of the second device (200).

[0108] For example, let us assume that the “shape insertion” editing function is selected by a user pointing action on the function selection area (210) of the second device (200). Then, in the electronic document editing program (100b), a shape function graphic object (e.g., a rectangle, a circle, an arrow, etc.) may be provided based on the selection of the “shape insertion” editing function. As illustrated in FIG. 5c (a), the control unit (800) may provide the function graphic object corresponding to each shape to the work area (240) as tactile stimulation information. The control unit (800) may provide tactile stimulation information by controlling the protrusion cell corresponding to the shape, such as a rectangle, a circle, or an arrow, among a plurality of protrusion cells placed in the work area (240) to protrude.

[0109] The control unit (800) can recognize a user pointing to a work area (240) of the second device (200) using a camera (300). The control unit (800) can recognize a user pointing to a work area (240) from an image captured of the user interface (200a) of the second device (200). In this case, the control unit (800) can recognize a user pointing to a work area (240) from an image captured by the camera (300) based on the selection of a button (250) provided on the second device (200).

[0110] The control unit (800) can identify a pointing area that is pointed to by a user in the work area (240) and identify a function graphic object corresponding to the pointing area. The control unit (800) can control the execution of a function item corresponding to the specific function graphic object in the electronic document editing program (100b) of the first device (100).

[0111] For example, when an area corresponding to a “circle shape” in the work area (240) is pointed, the control unit (800) can control the insertion of the “circle shape” in the electronic document editing program (100b). As illustrated in (b) of FIG. 5c, editing can be performed on the electronic document (530) being output on the display (100a) of the first device (100) as the “circle shape” is inserted. Based on the electronic document editing being performed, the edited electronic document (530a) can be displayed (output) on the display (100a) of the first device (100). The edited electronic document (510a) may include a new object circle shape (531).

[0112] In the present invention, a process may be performed to provide information corresponding to the result of editing an electronic document output on the display of the first device to the editing area of ​​the second device (S360. See FIG. 3).

[0113] The user interface (200a) provided by the second device may include a tactile interface. The control unit (800) can convert an electronic document edited in the first device into tactile stimulus information that can be provided to the tactile interface. And the control unit (800) can provide the tactile stimulus information as an editing result of the electronic document on the editing area (230) of the second device.

[0114] The control unit (800) can provide tactile stimulation information corresponding to the edited electronic document in the editing area (230) of the second device (200) in conjunction with the editing of the electronic document in the electronic document editing program (100b) and the output of the edited electronic document on the display (100a) of the first device (100).

[0115] The control unit (800) can separate and analyze visual objects such as text, images, shapes, and charts that constitute the editing results of the electronic document editing program (100b). The control unit (800) can analyze location information, size information, and attribute information of each object included in the electronic document. And the control unit (800) can convert (map) the objects included in the electronic document into tactile stimulus information.

[0116] For example, the control unit (800) can convert text information into a Braille code corresponding to the text information. The control unit (800) can map a shape object, such as a square, circle, or line, to at least some of the multiple protrusion cells arranged in the editing area (230) based on the outline information of the shape object.

[0117] The control unit (800) can perform a conversion process so that the relative placement information of each object in the electronic document is arranged in the same layout as the tactile stimulation information on the editing area (230) of the second device.

[0118] The control unit (800) can extract location information for each of a plurality of objects, such as text objects, shape objects, and image objects, included in the electronic document. Based on the relative placement information that each object in the electronic document has with respect to one another, the control unit (800) can generate tactile stimulation information so that each object maintains the same layout structure on the editing area (230) of the second device (200).

[0119] The control unit (800) determines which cell area among a plurality of protruding cell arrays corresponds to tactile stimulation information based on layout information including the position coordinates, size, and placement order of each object in the electronic document, and can perform protrusion control for the cells on the editing area (230) according to the determination.

[0120] For example, as illustrated in (c) of FIG. 5a, in conjunction with the output of an edited electronic document (510a) on the display (100a) of the first device (100), the control unit (800) can generate tactile stimulation information (512) corresponding to an image (or image object, 511) inserted in the edited electronic document (510a). The control unit (800) can convert the inserted image (511) into tactile stimulation information (512) by selecting at least some of the plurality of protrusion cells arranged on the editing area (230) of the second device (200) based on the size, position, boundary information, etc. of the inserted image (511). The control unit (800) can provide tactile stimulation information (512) corresponding to the inserted image (511) by controlling the editing area (230) so that at least some of the plurality of protrusion cells are ejected.

[0121] As another example, as illustrated in (c) of FIG. 5b, when an edited electronic document (520a) containing new text (521) is output to the display (100a) of the first device (100), the control unit (800) can generate tactile stimulation information (522) corresponding to the new text (521) inserted into the edited electronic document (520a). The control unit (800) can perform a process of converting the inserted text (521) into a Braille code corresponding to each of the texts constituting the inserted text (521). The control unit (800) can map the converted Braille code to correspond to a plurality of protrusion cells arranged on the editing area (230) of the second device (200). Then, the control unit (800) can provide tactile stimulation information (522) by controlling the mapped protrusion cells to protrude in a shape that corresponds to the Braille pattern of the inserted text (521).

[0122] As another example, as illustrated in (c) of FIG. 5c, when an edited electronic document (530a) containing a circular shape (531) is displayed on the display (100a) of the first device (100), the control unit (800) can generate tactile stimulation information (532) corresponding to the circular shape (531), which is an object included in the edited electronic document (530a). The control unit (800) can extract contour information of the circular shape based on visual attributes such as the position, size, and shape of the circular shape (531), and convert the contour information so that it is mapped onto a plurality of protrusion cells arranged in the editing area (230) of the second device (200). The control unit (800) can generate tactile stimulation information (532) for the plurality of protrusion cells to form a non-linear curved arrangement of protrusions according to the curve information constituting the outline of the circular shape (531). The control unit (800) can provide tactile stimulation information (532) to the editing area (230) of the second device (200) so that the shape of the circle shape (531) can be perceived tactilely.

[0123] Meanwhile, when an electronic document is edited, the control unit (800) can compare the electronic document before editing with the edited electronic document and update the editing area (230) of the second device (200) so that the changed information is reflected.

[0124] The control unit (800) can compare the electronic document before editing and the edited electronic document to identify information changed according to a specific editing function.

[0125] In this case, the control unit (800) can compare the electronic document before editing and the edited electronic document based on the electronic document displayed on the display (100a) of the first device (100). For example, as shown in (b) of FIG. 5a, the control unit (800) can compare the electronic document (510) before the image (511) is inserted and the electronic document (510a) after the image (511) is inserted.

[0126] Furthermore, the control unit (800) can compare tactile stimulus information corresponding to the electronic document before editing with tactile stimulus information corresponding to the electronic document after editing. Based on the fact that the edited electronic document is output to the display (100a) of the first device (100), the control unit (800) can convert the edited electronic document into tactile stimulus information corresponding to the editing area (230) of the second device (200). The control unit (800) can identify changed information by comparing the tactile stimulus information corresponding to the electronic document before editing with the tactile stimulus information corresponding to the electronic document after editing.

[0127] The control unit (800) maintains electronic document object information (or tactile stimulus information) in a state prior to editing and electronic document object information (or tactile stimulus information) after editing in memory, respectively, and can identify changed objects or attributes by performing a comparison between them.

[0128] More specifically, the control unit (800) can compare the electronic document before editing (tactile stimulus information corresponding to the electronic document before editing) and the edited electronic document (tactile stimulus information corresponding to the edited electronic document). The control unit (800) can compare attribute information such as object type (e.g., text, image, shape, etc.), coordinate position, size, and content included in the first electronic document (tactile stimulus information corresponding to the electronic document before editing) and the second electronic document (tactile stimulus information corresponding to the edited electronic document).

[0129] The control unit (800) can specify an update target area among the editing areas of the second device based on the identified information. The control unit (800) can update the editing area (230) by providing tactile stimulation information corresponding to the changed information to the update target area.

[0130] The control unit (800) can identify the area where editing has been performed and the content of the editing in the electronic document based on the comparison result of the first electronic document and the second electronic document. And the control unit (800) can identify the area to be updated within the editing area (230) of the second device (200) based on the comparison result.

[0131] The control unit (800) can generate tactile stimulus information corresponding to the edited information and provide the tactile stimulus information to the update target area among the edit area (230). That is, the control unit (800) can update the update target area with new tactile stimulus information.

[0132] For example, if new text is inserted at a specific location in an electronic document or the location of an existing shape is changed, the control unit (800) may specify only the area where the change was made as the area to be updated. Subsequently, the control unit (800) may create new tactile stimulus information corresponding to the changed object only for the specified area to be updated, and perform an update by reflecting the tactile stimulus information only at the corresponding locations among the protrusion cells arranged on the editing area (230) of the second device (200).

[0133] Meanwhile, the present invention can support the user's electronic document editing work by utilizing an artificial intelligence model (700).

[0134] In the present invention, the artificial intelligence model (700) is based on a pre-trained language model or a conversational artificial intelligence algorithm and can support the user's editing of electronic documents through conversation or question-and-answer with the user.

[0135] The control unit (800) can generate guide information that guides information output to the display (100a) of the first device (100) based on the artificial intelligence model (700).

[0136] The control unit (800) can collect visual information and contextual information of the current editing status of the electronic document editing program, the active editing function, and user interface components placed on the screen (e.g., toolbar, icon, text block, image, etc.).

[0137] The control unit (800) can recognize the location, name, status (e.g., selected, disabled), and related actions of each object within the display (100a) of the first device. For example, when the user is waiting to input at a specific cursor position while the "Insert Text" function is selected, the control unit (800) can generate guide information such as "Currently in insert mode, and the cursor is located in the middle of the third paragraph of the text."

[0138] Furthermore, the control unit (800) can generate guide information that visually explains the type, location, role, status, etc. of individual objects (e.g., text, image, icon, button, etc.) placed on the display (100a) of the first device.

[0139] The control unit (800) can generate guide information describing the existence and status (active / inactive) of an edit function activated on an electronic document editing screen (e.g., insert function (insert text, insert image, insert shape, etc.)), a selected edit target (e.g., cursor position, selected image, etc.), a button waiting to operate (e.g., “Save”, “Close”, “Insert”, etc.), and the icon structure and name of a toolbar or sidebar on the screen.

[0140] For example, as illustrated in FIG. 6(a), a user pointing action on the user interface (200a) of the second device (200) can be recognized. As illustrated in FIG. 6(b), a plurality of icon graphic objects (610, 620) can be displayed on the display (100a) of the first device based on the recognition of the user pointing action. As illustrated in FIG. 6(c), the control unit (800) can generate guide information (e.g., “The first icon is glasses. The second icon is an abacus. The glasses icon has been selected. Insert the glasses icon into the slide”, 630) that explains the arrangement order and icon content of the plurality of icon graphic objects (610, 620) displayed on the display (100a) of the first device.

[0141] The control unit (800) can generate a prompt to be input to the artificial intelligence model (700) by using screen information and context information output to the display (100a) of the first device.

[0142] Here, “screen information” may include information describing components of an electronic document and an electronic document interface displayed on the display (100a) of the first device. For example, the control unit (800) may include information regarding the location, size, content, attributes, and display status of objects such as text blocks, images, shapes, buttons, and icons placed on the display (100a) of the first device. Additionally, the screen information may include screen image information (e.g., a captured image) displayed on the display (100a) of the first device.

[0143] And “contextual information” may include at least one of the currently active editing function, the editing history recently performed by the user, the current cursor position and selection area, and the electronic document type (attributes, e.g., presentation slides, meeting minutes, guide documents for the visually impaired, etc.).

[0144] The control unit (800) can generate a prompt including screen information and context information. The control unit (800) can process the prompt as input to an artificial intelligence model (700) to generate guide information for the user. The control unit (800) can provide the guide information (630) to the user as voice information through a speaker (600).

[0145] When a function graphic object associated with a specific editing function is output to the display (100a) of the first display (100a), the control unit (800) can generate a prompt containing information about the function graphic object (e.g., the icon graphic object, 610, 620, shown in (b) of FIG. 6) and process it as input to an artificial intelligence model. The control unit (800) can obtain guide information describing the function graphic object from the artificial intelligence model (700). The control unit (800) can convert the guide information into voice information and provide it through a speaker.

[0146] The control unit (800) can dynamically respond to the screen configuration of the display (100a) and the switching situation of editing functions so as to be able to interact in real time according to the user's electronic document editing workflow, and can output guide information in real time through the speaker (600). Whenever an editing action such as the user's pointing action, cursor movement, or function selection is detected, the control unit (800) can collect the latest editing status information corresponding to the action and provide guide information reflecting this through the speaker (600).

[0147] Furthermore, the control unit (800) can use the artificial intelligence model (700) to check whether the electronic document edited in the electronic document editing program corresponds to the user's intention.

[0148] As illustrated in FIG. 7(a), the control unit (800) can provide a conversation function between the artificial intelligence model (700) and the user based on user input to one of the multiple buttons (250) provided in the second device (200). In this case, the control unit (800) can provide a voice conversation function. The control unit (800) can receive the user's voice through a microphone and input the user's voice to the artificial intelligence model (700).

[0149] As illustrated in FIG. 7(b), the control unit (800) can analyze natural language commands or query sentences included in the voice of the user (U), convert the user voice input into text data, and then input it into the artificial intelligence model (700). The user's voice input may be a query reflecting the user's intent, such as “Does this paragraph fit the summary content?”, “What is the meaning of the image just inserted?”, or “Is this document suitable for a presentation?” Additionally, the user voice input may include content requesting an inspection of an electronic document. For example, it may include the type of electronic document and the intended editing result, such as “This material is meeting material. I inserted two pictures.”

[0150] The control unit (800) can generate a prompt requesting an electronic document inspection by using at least one of the following information: the user's voice input, the user's intention, the current editing status and editing history of the electronic document, the activated editing function, the type of inserted content, and the purpose of creating the electronic document.

[0151] More specifically, the control unit (800) can generate a prompt based on at least one of a text sentence converted from the user's voice input, a user intent analyzed from the user's voice input, an electronic document being edited, information about the electronic document (e.g., text of a specific paragraph, description of an inserted image, selected function, etc.), the user's editing history information, and information about the electronic document editing style preferred by the user.

[0152] The control unit (800) processes the prompt as input to the artificial intelligence model (700) to check whether the electronic document has been edited according to the user's intention.

[0153] The artificial intelligence model (700) can generate response information (or feedback information) based on the input prompt, including at least one of a judgment result on whether the content of the electronic document aligns with the user's intention or purpose of creation, a summary description, and a suggestion for editing direction.

[0154] The control unit (800) can convert response information obtained from the artificial intelligence model (700) into voice information and provide it to the user through the speaker (600). That is, the control unit (800) can provide feedback information that checks the editing status of the electronic document through voice information.

[0155] Through this, users can check the appropriateness of electronic documents via voice-based feedback without accessing visual information, and adjust the editing direction based on the analysis of an AI model. This configuration can be particularly useful for visually impaired users in terms of providing real-time editing review and intelligent guidance.

[0156] The multi-sensory interface and artificial intelligence-based electronic document editing method and system according to the present invention provide a user interface including a function selection area and an editing area to a second device in conjunction with the execution of the electronic document editing program on a first device, and can smoothly perform the creation, modification, and editing of electronic documents under various user conditions (e.g., visually impaired) or various device configuration environments.

[0157] Furthermore, the multi-sensory interface, artificial intelligence-based electronic document editing method, and system according to the present invention recognize a user’s pointing motion pointing to one area among the function selection areas of the second device through a camera, specify an editing function corresponding to one area among the function selection areas of the second device based on the pointing motion, and apply the specified editing function to the electronic document editing program running on the first device to edit the electronic document being displayed on the display of the second device. Through this, the present invention enables even visually impaired individuals with limitations in recognizing visual information to intuitively select an editing function using only hand movements (pointing motions) and to perform editing of the content of an electronic document without relying on sight, thereby providing enhanced accessibility and convenience to users with sensory limitations, including the visually impaired.

[0158] Furthermore, the multi-sensory interface and artificial intelligence-based electronic document editing method and system according to the present invention provide information corresponding to the electronic document editing result output on the display of the first device to the editing area of ​​the second device, and even a visually impaired person who has difficulty directly perceiving visual information can recognize the editing result in real time through the editing area of ​​the second device and can easily edit the electronic document independently without the help of another person.

[0159] Meanwhile, the present invention described above can be implemented as a program that is executed by one or more processes on a computer and can be stored on a computer-readable medium (or recording medium).

[0160] Furthermore, the present invention described above can be implemented as computer-readable code or instructions on a medium on which a program is recorded. That is, the present invention can be provided in the form of a program.

[0161] Meanwhile, computer-readable media include all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0162] Furthermore, the computer-readable medium may be a server or cloud storage that includes a storage and is accessible to an electronic device via communication. In this case, the computer may download the program according to the present invention from the server or cloud storage via wired or wireless communication.

[0163] Furthermore, in the present invention, the computer described above is an electronic device equipped with a processor, namely a CPU (Central Processing Unit), and no special limitations are placed on its type.

[0164] Meanwhile, the above detailed description should not be interpreted restrictively in all respects but should be considered exemplary. The scope of the invention shall be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are included within the scope of the invention.

Claims

Claim 1 A method for editing an electronic document using a first device and a second device different from the first device, comprising: a step of providing a user interface including a function selection area and an editing area to the second device in conjunction with the execution of an electronic document editing program on the first device, in a control unit included in the first device; a step of recognizing a user’s pointing motion pointing to one area among the function selection areas of the second device through a camera in a control unit included in the first device; a step of specifying an editing function corresponding to one area among the function selection areas of the second device based on the pointing motion in a control unit included in the first device; a step of applying the specified editing function to the electronic document editing program running on the first device in a control unit included in the first device to edit the electronic document being output on the display of the first device; and a step of providing information corresponding to the result of editing the electronic document output on the display of the first device to the editing area of ​​the second device in a control unit included in the first device. A multi-sensory interface and AI-based electronic document editing method, comprising: a step of checking the edited electronic document by utilizing an AI model that determines whether the edited electronic document was edited according to the user's intention, in response to receiving a user voice requesting to check the edited electronic document through a microphone in a control unit included in the first device; wherein in the checking step, a prompt is generated using at least one of text converted from the user voice, the edited electronic document, and information regarding the specified editing function; the prompt is processed as input to the AI ​​model to obtain feedback information including at least one of judgment and editing direction suggestions regarding the edited electronic document; and the feedback information is output through a speaker. Claim 2 A multi-sensory interface and an artificial intelligence-based electronic document editing method according to claim 1, wherein a plurality of tactile stimulus information corresponding to each of a plurality of editing functions is provided in the function selection area of ​​the second device, and among the plurality of tactile stimulus information, the tactile stimulus information corresponding to the specified editing function is continuously provided in the function selection area of ​​the second device, and the tactile stimulus information corresponding to an editing function different from the specified editing function disappears from the function selection area of ​​the second device. Claim 3 A multi-sensory interface and artificial intelligence-based electronic document editing method, characterized in that, in the second paragraph, when a specific editing function is selected by a user’s pointing motion in a function selection area of ​​the second device, a function graphic object associated with the specific editing function is provided in the electronic document editing program running on the first device, the user interface provided in the second device further includes a work area, and tactile function information corresponding to the function graphic object is provided in the work area of ​​the second device. Claim 4 A multi-sensory interface and AI-based electronic document editing method according to claim 3, further comprising the step of providing guide information related to the specified editing function through the speaker in the control unit, wherein the step of providing the guide information comprises: the step of generating a prompt containing information about the function graphic object in the control unit and processing it as input to an artificial intelligence model; the step of obtaining the guide information describing the function graphic object from the artificial intelligence model in the control unit; and the step of converting the guide information into voice information in the control unit and providing it through the speaker. Claim 5 delete Claim 6 A multi-sensory interface and artificial intelligence-based electronic document editing method, characterized in that, in the case where a pointing motion of the user pointing to the work area of ​​the second device is recognized using the camera, the control unit controls the electronic document editing program running on the first device in conjunction with the pointing motion for the work area to edit the electronic document, and information corresponding to the edited electronic document is provided in the editing area of ​​the second device. Claim 7 A multi-sensory interface and artificial intelligence-based electronic document editing method according to claim 1, wherein the user interface provided by the second device includes a tactile interface, and in the step of providing to the editing area, the control unit converts the electronic document edited by the first device into tactile stimulus information that can be provided to the tactile interface, and provides the tactile stimulus information as an editing result of the electronic document on the editing area of ​​the second device. Claim 8 A multi-sensory interface and artificial intelligence-based electronic document editing method, characterized in that, in the step of providing to the editing area, the electronic document before editing and the edited electronic document are compared to identify information changed according to the specified editing function, and based on the identified information, an update target area among the editing areas of the second device is specified, and tactile stimulus information corresponding to the changed information is provided to the update target area. Claim 9 A multi-sensory interface and artificial intelligence-based electronic document editing method, characterized in that, in the step of recognizing the pointing motion of the user, the control unit identifies a function selection area of ​​the second device and a preset marker from an image captured by the camera, and in the step of specifying the editing function, the control unit determines the specified editing function based on the relative positional relationship between the function selection area of ​​the second device and the marker. Claim 10 A first device on which an electronic document editing program is executed; a second device different from the first device and the first device; and, in conjunction with the execution of the electronic document editing program on the first device, a control unit included in the first device provides a user interface including a function selection area and an editing area to the second device, wherein the control unit included in the first device recognizes a user’s pointing motion pointing to one area of ​​the function selection area of ​​the second device through a camera, and based on the pointing motion, specifies an editing function corresponding to one area of ​​the function selection area of ​​the second device, applies the specified editing function to the electronic document editing program running on the first device to edit the electronic document being output on the display of the first device, provides information corresponding to the editing result of the electronic document output on the display of the first device to the editing area of ​​the second device, and in response to the reception of a user voice requesting an inspection of the edited electronic document through a microphone, inspects the edited electronic document by utilizing an artificial intelligence model that determines whether the edited electronic document was edited according to the user’s intention, and the control unit included in the first device includes text converted from the user voice, the edited A multi-sensory interface and artificial intelligence-based electronic document editing system characterized by generating a prompt using at least one of information regarding an electronic document and the aforementioned specific editing function, processing the prompt as input to the artificial intelligence model to obtain feedback information including at least one of judgment and editing direction suggestion for the edited electronic document, and outputting the feedback information through a speaker. Claim 11 A program that is executed by one or more processes in an electronic device and stored on a computer-readable medium, wherein the program is installed in a first device and an electronic document editing program is executed in the first device corresponding to the electronic device; a step of providing a user interface including a function selection area and an editing area to a second device in conjunction with the execution of the electronic document editing program in the first device; a step of recognizing a user’s pointing motion pointing to one area of ​​the function selection area of ​​the second device through a camera; a step of specifying an editing function corresponding to one area of ​​the function selection area of ​​the second device based on the pointing motion; a step of applying the specified editing function to the electronic document editing program running in the first device to edit the electronic document being output on the display of the first device; and a step of providing information corresponding to the result of editing the electronic document output on the display of the first device to the editing area of ​​the second device. A program stored on a computer-readable medium, comprising commands for performing a step of inspecting an edited electronic document by utilizing an artificial intelligence model that determines whether the edited electronic document was edited according to the user's intention, in response to receiving a user voice requesting inspection of the edited electronic document through a microphone, wherein in the inspection step, a prompt is generated using at least one of text converted from the user voice, the edited electronic document, and information regarding the specified editing function, and the prompt is processed as input to the artificial intelligence model to obtain feedback information including at least one of judgment and editing direction suggestions regarding the edited electronic document, and the feedback information is output through a speaker.

Images