System and method for providing screen sharing service using extension tools

Abstract

To provide a system and method for providing a screen sharing service using an extension tool that enables users other than developers to easily share the screen of a current Web page with other users. [Solution] In the screen sharing service providing system 1, the screen sharing service 10 includes an extension tool providing unit that registers a screen sharing related extension tool in the extension tool registration unit of the web browsing service server, a request receiving unit that receives a screen sharing request in response to execution of the screen sharing related extension tool installed in the first user terminal, and a screen sharing execution unit that transmits a screen sharing request notification message to a second user terminal corresponding to the screen sharing request, receives a screen sharing available state notification message from the second user terminal, transmits the screen sharing available state notification message to the first user terminal, and establishes a connection for screen sharing between the first user terminal 50 and the second user terminal 60.

Description

[Technical Field]

[0001] The present invention relates to a system and method for providing a screen sharing service using an extension tool. [Background technology]

[0002] Recently, digital transformation has been in the spotlight. Digital transformation refers to the digital transformation of business operations, including our daily lives, using new digital technologies.

[0003] Companies and organizations are increasingly using cloud services instead of building their own servers and operating environments. Using cloud services means that data and resources reside on an external cloud.

[0004] This has brought about a shift to a new work environment where various forms of collaboration and sharing are possible through mobile devices such as smartphones, and the need for business programs that can support collaboration and sharing is rapidly increasing in order to adapt to this.

[0005] Currently, the market is divided into platforms for mobile application (hereinafter referred to as apps) development and desktop application platforms. As the mobile ecosystem sees the emergence of mobile operating systems that lack interoperability, such as Apple's iOS and Google's Android, the burden of having to develop separate applications for specific operating systems is increasing. As a result, hybrid apps, which allow for the easy creation of a single app that can be efficiently used on various platforms such as iOS, Android, and Windows with a single development, are beginning to attract attention.

[0006] Furthermore, in recent years, non-face-to-face remote services have been increasing, and business processes suited to these services are now required.

[0007] The above-mentioned background art is technical information that the inventor possessed for the purpose of deriving the present invention or that he acquired in the process of deriving the present invention, and is not necessarily publicly known art that was made public to the general public prior to the filing of the present invention. [Prior art documents] [Patent documents]

[0008] Korean Patent Publication No. 10-2012-0063366 (Published on June 15, 2012) - Business Support System Summary of the Invention [Problem to be solved by the invention]

[0009] Therefore, the present invention has been devised to solve the above-mentioned problems, and an object of the present invention is to provide a system and method for providing a screen sharing service using an extension tool, which provides a screen sharing function using an extension tool of a web browser, allowing users other than developers to easily share the screen of the current web page with other users.

[0010] Another object of the present invention is to provide a system and method for providing a screen sharing service using an extension tool, which improves work efficiency by enabling non-face-to-face remote work processing by sharing a screen such as a user interface block implemented on the Web between users.

[0011] Still other problems to be solved by the present invention will be easily understood from the following description. [Means for solving the problem]

[0012] In order to achieve the above object, according to one aspect of the present invention, there is provided a computer program stored on a computer-readable medium for performing a method for providing a screen sharing service using an extension tool, the computer program causing a computer to perform the following steps, the steps being: registering a screen sharing related extension tool in an extension tool registration unit of a web browsing service server in an extension tool providing unit of a screen sharing service server; receiving a screen sharing request in response to execution of the screen sharing related extension tool installed in a first user terminal in a request receiving unit of the screen sharing service server; transmitting a screen sharing request notification message to a second user terminal corresponding to the screen sharing request in a screen sharing executing unit of the screen sharing service server; receiving a screen sharing available state notification message from the second user terminal in the screen sharing execution unit; and transmitting the screen sharing available state notification message to the first user terminal in the screen sharing execution unit, thereby establishing a connection for screen sharing between the first user terminal and the second user terminal, wherein the screen sharing related extension tool is an extension program installed in a web browser of the first user terminal, allowing items displayed on the web browser screen to be screen shared with the second user terminal, and the first user terminal requests the screen sharing related extension tool for installation in the web browser from the web browsing service server.

[0013] The method may further include the steps of: installing a screen sharing assistant when the screen sharing-related extension tool is executed on the first user terminal; automatically generating a screen sharing script using a screen sharing API in the screen sharing assistant; and providing a custom screen sharing script customized to the items displayed on the screen of the web browser.

[0014] The screen sharing script may include one or more of a sharing start API code, a connection URL, a service provision script, and a sharing end API code.

[0015] The method may further include the steps of: receiving a user query through a chat client provided by the screen sharing assistant; generating a context by calling context information corresponding to a category corresponding to the user query; calling a service module according to the category and adding the context to a prompt for each service provided by the service module to generate a prompt for LLM; querying a large-scale language model for the prompt for LLM through an LLM interface and receiving a corresponding response; and checking the validity of the response and then providing it to the chat client as a response to the user query.

[0016] The step of transmitting the screen sharing request notification message to the second user terminal may also provide a connection URL included in the screen sharing script, and the second user terminal may transmit the screen sharing available status notification message after connecting to a predetermined web page via the connection URL.

[0017] Meanwhile, according to another aspect of the present invention, there is provided a screen sharing service server included in a screen sharing service providing system using an extension tool, the screen sharing service server including: an extension tool providing unit that provides a screen sharing-related extension tool to a web browsing service server and registers the extension tool in an extension tool registration unit; a request receiving unit that receives a screen sharing request in response to execution of the screen sharing-related extension tool installed in a first user terminal; and a screen sharing execution unit that transmits a screen sharing request notification message to a second user terminal corresponding to the screen sharing request, receives a screen sharing available state notification message from the second user terminal, and transmits the screen sharing available state notification message to the first user terminal so that a connection for screen sharing is established between the first user terminal and the second user terminal, wherein the screen sharing-related extension tool is an extension program installed in a web browser of the first user terminal and allows the second user terminal to share items displayed on the web browser screen, and the first user terminal requests the screen sharing-related extension tool for installation in the web browser from the web browsing service server, and is provided with the screen sharing service server.

[0018] When the screen sharing-related extension tool is executed on the first user terminal, a screen sharing assistant is installed, and the screen sharing assistant automatically generates a screen sharing script using a screen sharing API, and a custom screen sharing script customized for the items displayed on the web browser screen can be provided.

[0019] The screen sharing script may include one or more of a sharing start API code, a connection URL, a service provision script, and a sharing end API code.

[0020] The screen sharing assistant calls context information corresponding to a category corresponding to a user query input through a chat client provided by the screen sharing assistant to generate a context, calls a service module according to the category, adds the context to a prompt for each service provided by the service module to generate a prompt for LLM, queries a large-scale language model for the prompt for LLM through an LLM interface, and generates a prompt for LLM according to the context information. answer receiving the answer The chat bot may further include an artificial intelligence chat bot response unit that checks the validity of the query and then provides the query to the chat client as a response to the user query.

[0021] When transmitting the screen sharing request notification message to the second user terminal, the screen sharing execution unit may also provide a connection URL included in the screen sharing script, and the second user terminal may connect to a predetermined web page via the connection URL and then transmit the screen sharing available status notification message.

[0022] Further aspects, features, and advantages will become apparent from the following drawings, claims, and detailed description of the invention. [Effects of the Invention]

[0023] According to the embodiment of the present invention, a screen sharing function is provided using an extension tool of a web browser, which has the effect of enabling users other than developers to easily share the screen of the current web page with other users.

[0024] In addition, it also has the effect of improving work efficiency by enabling non-face-to-face remote business processing by sharing a screen such as a user interface block realized on the web between users.

[0025] The effects obtained by the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those having ordinary skill in the art to which the present invention pertains from the following description. [Brief explanation of the drawings]

[0026] [Figure 1] 1 is a conceptual diagram of a screen sharing service providing system using an extension tool according to an embodiment of the present invention. [Figure 2] FIG. 10 is a block diagram of the configuration of a terminal provided with an extension tool. [Figure 3] 1 is a flowchart of a screen sharing service providing method performed in a screen sharing service providing system using an extension tool according to an embodiment of the present invention. [Figure 4] 1 is a flowchart of a method for using an assistant when installing an extension tool according to an embodiment of the present invention. [Figure 5] FIG. 10 is a diagram showing installation information of the screen sharing assistant (W-Sharing Assistant) provided by the extension tool after it has been installed in a web browser. [Figure 6] This image shows the icon added to the menu bar of a web browser after the Screen Sharing Assistant provided by the extension tool is installed. [Figure 7] 1 is an example of a screen sharing script using the Screen Sharing Assistant. [Figure 8] FIG. 1 is a conceptual diagram of a non-face-to-face remote business processing system using screen sharing. [Figure 9] 1 is a conceptual diagram of full sharing and partial sharing; [Figure 10] FIG. 1 is a conceptual diagram of signature data sharing. [Figure 11] FIG. 1 is a conceptual diagram for sharing image and audio data. [Figure 12] FIG. 1 is a conceptual diagram for history logging and replay functions. [Figure 13] FIG. 1 is a conceptual diagram of multiple sharing. [Figure 14]FIG. 1 is a diagram illustrating the configuration of a screen sharing service. [Figure 15] 1 is a diagram illustrating a configuration of a system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0027] Since the present invention can be modified in various ways and can have multiple embodiments, specific embodiments will be illustrated in the drawings and described in detail, but it should be understood that this is not intended to limit the present invention to the specific embodiments, and that the present invention encompasses all modifications, equivalents, and alternatives within the spirit and scope of the present invention.

[0028] When a component is referred to as being "coupled" or "connected" to another component, it should be understood that the component may be directly coupled or connected to the other component, but that there may be other components between them. On the other hand, when a component is referred to as being "directly coupled" or "directly connected" to another component, it should be understood that there are no other components between them.

[0029] Terms such as first and second may be used to describe various components, but the components should not be limited by these terms. These terms are used only to distinguish one component from another.

[0030] The terms used in this application are merely used to describe specific embodiments and are not intended to limit the present invention. The singular expressions include the plural expressions unless the context clearly dictates otherwise. In this application, the terms "comprise" or "have" are intended to specify only the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and should not be understood to preclude the presence or possible addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0031] As used herein, the term "module" includes hardware-implemented units, software-implemented units, and units implemented using both hardware and software. Furthermore, one unit may be implemented using two or more pieces of hardware, or two or more units may be implemented using a single piece of hardware. On the other hand, the term "module" is not limited to software or hardware; a "module" may be configured to reside on an addressable storage medium or to implement one or more processors. Thus, by way of example, a "module" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within a component and a "module" may be combined into fewer components and "modules" or further separated into additional components and "modules." Furthermore, a component and a "module" may be embodied to implement one or more CPUs within a device.

[0032] Furthermore, it is natural that the components of the embodiments described with reference to each drawing are not limited to being applied only to the embodiment in question, but may be embodied in a manner that is included in other embodiments within the scope that maintains the technical idea of ​​the present invention, and that even if separate explanations are omitted, multiple embodiments may be re-embodied as a single integrated embodiment.

[0033] In addition, when describing the present invention with reference to the accompanying drawings, the same or related reference numerals will be used to refer to the same components regardless of the reference numerals, and redundant descriptions thereof will be omitted. In describing the present invention, if a detailed description of related known art is deemed to unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

[0034] FIG. 1 is a conceptual diagram of a screen sharing service providing system using an extension tool according to an embodiment of the present invention.

[0035] A screen sharing service providing system 1 using an extension tool according to one embodiment of the present invention registers an extension tool related to screen sharing in a web browsing service server such as Chrome, provides the extension tool related to screen sharing in response to a user request through a web browser, and connects a first user terminal and a second user terminal in response to a screen sharing request using the extension tool, thereby enabling the entire screen or a part of the screen to be shared.

[0036] In addition, the extension tool helps to automatically generate API code for screen sharing, and utilizes an AI-based chatbot to help users easily customize the screen sharing API through dialogue.

[0037] 1, a screen sharing service providing system 1 using an extension tool may include a screen sharing service server 10, a web browsing service server 30, a first user terminal 50, and a second user terminal 60. Here, the screen sharing service server 10, the web browsing service server 30, the first user terminal 50, and the second user terminal 60 may be configured to be communicably connected to each other via a wired or wireless network.

[0038] The first user terminal 50 is a shared user terminal device that allows users to share their screen. The first user terminal 50 is a computing device that can connect to a web page through a web browser, and may be, for example, a smartphone, a tablet PC, a notebook, or a desktop PC.

[0039] The second user terminal 60 is a viewer user terminal device that shares and displays the first user's screen. The second user terminal 60 is a computing device that can connect to a web page through a web browser, and may be, for example, a smartphone, a tablet PC, a notebook, or a desktop PC.

[0040] The screen sharing service server 10 can provide the Web browsing service server 30 with an extension tool having a screen sharing function and register it in the Web store (extension tool registration unit 31).

[0041] The web browsing service server 30 communicates with the first user terminal 50 and provides screen sharing related extension tools registered in the web store in response to an extension tool request through the web browser of the first user terminal 50, thereby allowing the screen sharing function to be installed in the first user terminal 50.

[0042] In addition, the screen sharing service server 10 communicates with the second user terminal 60 that is the target of screen sharing in response to a screen sharing request from a first user terminal 50 that has a screen sharing function installed, and acts as an intermediary so that information regarding screen sharing is sent and received between the first user terminal 50 and the second user terminal 60, thereby enabling screen sharing between the first user terminal 50 and the second user terminal.

[0043] The screen sharing service server 10 may include an extension tool providing unit 11, a request receiving unit 12, and a screen sharing executing unit 13.

[0044] The extension tool providing unit 11 provides an extension tool, which is an executable program that runs on the first user terminal 50 and provides a screen sharing assistant that automatically generates a screen sharing script that operates to share all or part of a screen between two or more user terminals 50, 60. For example, the extension tool may have the form of a plug-in that is installed in a web browser that runs on the first user terminal 50.

[0045] The extension tool providing unit 11 may provide an extension tool component to the web browsing service server 30, which is embodied as a store where downloadable applications and extension tools are uploaded to the first user terminal 50. Alternatively, the extension tool providing unit 11 may directly transmit the extension tool to the user terminals 50 and 60.

[0046] The request receiving unit 12 receives a screen sharing request in response to execution of a screen sharing script that is automatically generated using the screen sharing assistant from the first user terminal 50 in which the extension tool is installed.

[0047] When a screen sharing request is received through the request receiving unit 12, the screen sharing executing unit 13 is activated and transmits a connection URL for screen sharing to the second user terminal 60, which is the target of screen sharing, in response to the screen sharing request from the first user terminal 50 in which the extension tool is installed.

[0048] When the second user terminal 60 connects via the connection URL and receives a notification that the screen is available for sharing, the screen sharing execution unit 13 notifies the first user terminal 50 of this, and various data and images are transferred between the first user terminal 50 and the second user terminal 60, thereby enabling screen sharing.

[0049] Referring to FIG. 2, there is shown a block diagram of the configuration of a terminal (first user terminal 50) provided with an extension tool.

[0050] The first user terminal 50 may include a communication module 71 , a web view module 72 , an interface module 73 , a sharing module 74 , an image / audio module 75 , and a device management module 76 .

[0051] The communication module 71 communicates with the screen sharing service server 10 and / or the web browsing service server 30 to transmit and receive various data and files. The communication module 71 may include a mobile communication device such as 3G, 4G, or 5G and / or a short-range communication device such as Bluetooth or Wi-Fi. Data and files transmitted and received via the communication module 71 may be encrypted via an encryption module (not shown) to protect against hacking and the like.

[0052] The web view module 72 displays a screen in response to execution of the web browser and / or the extension tool 80. The web view module 72 may be a display module.

[0053] The interface module 73 may be a native interface, for example a JavaScript native interface.

[0054] The image / audio module 75 supports enabling image or audio communication between staff and customers so that non-face-to-face (or untact) service can be provided during the course of service provision.

[0055] The device management module 76 manages devices related to screen sharing or image / audio services.

[0056] The sharing module 74 may be a screen sharing module that provides screen sharing functionality, which in addition to screen sharing may include data sharing, history logging and replay, event sharing, annotation functionality, and the like.

[0057] The screen sharing function enables remote business processing without face-to-face contact by sharing all or part of the screen of the first user terminal 50 and displaying it in a specified area of ​​the second user terminal 60 when a connection is established between multiple terminals that wish to be interconnected in a predetermined manner. Alternatively, the sharing user and the viewer user can be switched, and all or part of the screen of the second user terminal 60 can be shared and displayed in a specified area of ​​the first user terminal 50, allowing staff to smoothly carry out or assist with the business required by the customer without directly handling the second user terminal 60.

[0058] FIG. 3 is a flowchart of a screen sharing service providing method performed in a screen sharing service providing system using an extension tool according to one embodiment of the present invention, and FIG. 4 is a flowchart of a method of utilizing an assistant when installing an extension tool according to one embodiment of the present invention.

[0059] In this embodiment, we will assume that a screen is to be shared from a first user terminal 50 to a second user terminal 60, that is, the first user terminal 50 is a sharing user terminal and the second user terminal 60 is a viewer user terminal.

[0060] Referring to FIG. 3, the screen sharing service server 10 registers an extension tool having a screen sharing function in a downloadable manner in the Web browsing service server 30, particularly in a store operated by the server.

[0061] In this case, the first user terminal 50 requests the Web browsing service server 30 for a screen sharing-related extension tool that can be installed in the Web browser executed on the first user terminal 50 for screen sharing (step S200).

[0062] The extension tool registration unit 31 of the web browsing service server 30 provides a screen sharing-related extension tool corresponding to the request from among the extension tools registered in the store to the first user terminal 50 so that the extension tool can be installed (step S205). The extension tool can be provided in a downloading manner via the store.

[0063] The first user terminal 50 provided with the screen sharing-related extension tool installs the extension tool as one of the extension programs of the web browser. When the installation of the extension tool is completed, the first user terminal 50 can execute the extension tool while the web browser is running.

[0064] Referring to FIG. 4, when the screen sharing extension tool is executed (step S211), a screen sharing assistant for smoothly utilizing the screen sharing function can be installed (step S212).

[0065] The screen sharing assistant can use the screen sharing API to automatically generate a screen sharing script that allows sharing of a screen related to a web page currently displayed through a web browser (step S213). The screen sharing script can include at least one of a sharing start API code for starting screen sharing, a sharing end API code for ending screen sharing, a connection URL for accessing the web page for screen sharing from a second user terminal 60 that does not have the extension tool installed, and a service provision script for providing various related services (e.g., sharing the entire screen) during screen sharing (steps S300, S302, S304, S306).

[0066] The automatically generated screen sharing script is a custom screen sharing script for the current web browser display screen, and can be provided to the first user terminal 50 (step S214). Screen sharing can be performed with the second user terminal 60 that wishes to share the screen via the custom screen sharing script.

[0067] Referring again to FIG. 3, when a screen sharing script is automatically generated using the screen sharing assistant installed by executing the extension tool on the first user terminal 50, the screen sharing script is executed to transmit a request to start screen sharing with the second user terminal 60, along with identification information and a connection URL for the second user terminal 60 that wishes to share the screen, to the screen sharing service 10 (step S220).

[0068] In response to the screen sharing start request, the screen sharing service server 10, particularly the screen sharing execution unit 13, notifies the second user terminal 60 of the screen sharing request by a message, and the second user terminal 60 can receive the screen sharing request notification (step S225). Here, the screen sharing request notification can include a connection URL.

[0069] Although the second user terminal 60 does not have any programs or extension tools for screen sharing installed, it can access a specific web page for screen sharing with the first user terminal 50 via the connection URL included in the screen sharing request notification message.

[0070] When the second user terminal 60 connects to the corresponding web page via the connection URL, a notification message informing that the screen is in a state where screen sharing is possible is sent to the screen sharing service server 10, thereby making it possible to confirm that the second user terminal 60 is in a state where screen sharing is possible. In addition, the screen sharing possible state notification message is transmitted to the first user terminal 50, allowing the first user terminal 50 to know that screen sharing is possible.

[0071] Through this process, the first user terminal 50 and the second user terminal 60 establish an interconnection for screen sharing, and can enjoy the screen sharing function by sending and receiving information for screen sharing (e.g., various data, images, etc.) through the screen sharing service server 10 (steps S230 and S235).

[0072] The first user terminal 50 can start screen sharing by executing a sharing start API code automatically generated using the screen sharing assistant when the extension tool is executed, and can end screen sharing by executing a sharing end API code. In addition, various related services (such as sharing an entire web page, sharing a part of a web page, data sharing, history logging and replay, multi-party collaboration function, event sharing, annotation function, etc.) can be provided during screen sharing through the service provision script.

[0073] Figure 5 shows the installation information for the W-Sharing Assistant, an extension tool, after it has been installed in a web browser. It explains that it provides development convenience by allowing users to easily apply a service for sharing standard web screens with other devices and other browsers.

[0074] Figure 6 shows an image of the screen sharing assistant, which is provided as an extension tool, being installed and then added to the menu bar of a web browser. The icon can be activated when using the assistant.

[0075] Figure 7 shows an example of a screen sharing script using the Screen Sharing Assistant. The screen sharing start API code, the screen sharing end API code, various service scripts, and the connection URL for screen sharing on the web browser screen can be automatically generated and provided.

[0076] In addition, the screen sharing assistant is an AI-based chatbot that can search for and provide optimal answers to user queries regarding corrections and completions to the automatically generated screen sharing script in cooperation with LLM.

[0077] An artificial intelligence-based chatbot works as follows:

[0078] The API requests for user queries input from the chat client can classify the user queries into corresponding categories. The chat client is a web interface for providing chat with users (see FIG. 7). It also communicates with the screen sharing service server 10, particularly the AI ​​chatbot response unit 14, to request user queries and receive corresponding responses.

[0079] The AI ​​chatbot response unit 14 can generate a work plan for the initial user query, respond to queries in the LLM (Large Language Model) in order according to the work plan, and generate a response to the query requested by the initial user.

[0080] The AI ​​chatbot response unit 14 includes an API gateway, which is a Rest API controller, and can categorize user queries to generate contexts and execute individual services. Here, context information corresponding to a user query can be retrieved through a database search to generate a context.

[0081] It can call a service module according to the category, add the context to the prompt for each service provided by the service module, and generate a prompt for the LLM. Then, it queries the LLM prompt through the LLM interface and generates the corresponding answer The AI ​​chatbot response unit 14 can receive the answer After checking the validity of the , a response to the user query can be provided to the chat client.

[0082] In this embodiment, the category is a screen sharing category, and the called service module is a screen sharing category. NecessaryThe parameter values ​​corresponding to the request are entered, and the processed result through LLM can be provided as a response. The response can be a JSON string required to execute the screen sharing API.

[0083] Figures 8 to 13 show items for screen sharing services using extension tools. Figure 8 is a conceptual diagram of a non-face-to-face remote business processing system using screen sharing, Figure 9 is a conceptual diagram of full sharing and partial sharing, Figure 10 is a conceptual diagram of signature data sharing, Figure 11 is a conceptual diagram of image and audio data sharing, Figure 12 is a conceptual diagram of history logging and replay functions, and Figure 13 is a conceptual diagram of multiple sharing.

[0084] 8, a partial area 55 of a terminal screen 50 of a sharing user can be set as a shared screen, and a shared screen 65 can be displayed on a terminal screen 60 of a network-connected viewer user. In this embodiment, in addition to screen sharing, data sharing and event sharing are also possible, and an annotation function can also be provided.

[0085] In this embodiment, the user interface sharing may include the following items:

[0086] Share entire web pages: Ability to share a user's entire web page with other users

[0087] Web page portion sharing: Ability to share a portion of a user's web page with other users (see Figure 8)

[0088] Data sharing: Ability to transmit data between shared devices, signature data sharing

[0089] History Logging & Replay: A function to log and replay the operation history on the shared screen.

[0090] Multi-party collaboration: Multiple users can interactively control and update the shared screen.

[0091] Event sharing: Ability to share events such as touching / clicking / typing on the shared screen

[0092] Annotation: Support collaboration with highlighting and / or annotations on the shared screen

[0093] Referring to FIG. 9, the concepts of full sharing and partial sharing are illustrated.

[0094] In the case of full sharing, the entire page 52 implemented on the web is shared 62. The sharing user terminal and the viewer user terminal share the same content. The sharing user and the viewer user can be implemented to have control over the entire shared page.

[0095] In the case of partial sharing, only a portion 55 of a page implemented on the web is shared 65. By sharing only a portion, it is possible to share only the content necessary for security or business purposes. The sharing user and the viewer user can be implemented to have control over the shared portion.

[0096] Referring to Figure 10, the concept of signature data transfer is shown.

[0097] When a shared user terminal 50 that processes business needs to receive a handwritten signature from a viewer user, by selecting a partial area (signature area) where a handwritten signature is required, a signature creation view is provided on the viewer screen (viewer user terminal screen), and the user can input the handwritten signature using the pen function.

[0098] The signature creation view may include a pop-up window 67 that runs on the viewer screen to allow the viewer to input a handwritten signature. In a screen shared on the viewer screen through full sharing or partial sharing, the portion where the handwritten signature is required may appear small depending on the screen ratio. Therefore, by displaying the pop-up window 67, the viewer can more clearly see the portion where input is currently required, enabling accurate signature data input.

[0099] In this way, electronic documents signed through the real-time signature function can be saved in the context of a shared user interface.

[0100] Referring to FIG. 11, the concept of photo and audio data transmission is shown.

[0101] The viewer user terminal may be a mobile terminal. In this case, the viewer user terminal may have a camera that can be used to take a photo of the viewer user (e.g., face) and transmit the image data to the shared user terminal.

[0102] Alternatively, the viewer user's voice can be acquired by linking a recording function using a microphone provided in the viewer user's terminal, and the voice data can be transmitted to the shared user's terminal.

[0103] Here, the image data can be used for authentication purposes regarding documents (e.g., contract documents) being created through the user interface sharing.

[0104] The voice data is a recording of the contractor reading out important phrases in the contract document, and can also be used for identity verification.

[0105] Referring to FIG. 12, the concept of history logging and replay is illustrated. When using the user interface sharing function, history logging and replay functions can be provided. When the sharing user terminal and the viewer user terminal are connected and the user interface sharing function is activated, history recording related to shared processing information (e.g., a combination of a shared screen and an occurring event (e.g., user actions such as clicking, moving the mouse, typing, signing, annotating, etc.)) can be started / stopped for a certain period, i.e., a predetermined period, or depending on whether a predetermined event occurs.

[0106] At this time, the screen sharing execution unit 13 functions as a broker server and stores a history record of the shared processing information in a database.

[0107] The history record can be replayed upon request. The replay is a time-lapse reproduction of the shared processing information from the start to the end of the history record, and can be provided by displaying a shared screen and replaying the events that occurred.

[0108] For example, in the case of a non-face-to-face transaction that requires confirmation and signature of the contents of a multi-page contract document, the entire process from the time when the shared user or viewer user inputs data to the time when the final contract document is completed can be recorded as a history. By reviewing this history record through replay, it is possible to verify whether the contract was properly concluded.

[0109] Regarding history recording, the filter level, filter target, storage period, etc. can be set as options, which makes it possible to prevent unlimited replay and replay by unauthorized persons.

[0110] When recording history, the terminal sharing module 74 can activate the front camera of at least one of the sharing user terminal and the viewer user terminal to take a picture of the sharing user or the viewer user at any time when the sharing user or the viewer user cannot recognize them, and attach the image data and the information on the time of taking the picture to the history record as authentication materials and store them together.

[0111] This allows the user's identity to be confirmed in the sharing function by superimposing the image data taken at the time of shooting on the shared screen that is played back when the history record is later replayed, or by displaying it in a separate area.

[0112] For example, let's say an employee and a customer are signing an insurance contract remotely through the user interface sharing function. To successfully sign the insurance contract, the employee must share a number of contract-related pages on their screen with the customer and obtain their handwritten signature. This entire process is stored as a history record through the history logging function, making it possible to determine whether the insurance contract was signed successfully.

[0113] In addition, by using the camera on the customer terminal to take a picture of the signer's face at the moment the customer signs or agrees to and confirms the signature documents, it is possible to confirm whether the signature is made by someone other than the customer, thereby preventing the conclusion of unfair contracts and making it possible to use it as strong evidence when disputing the legitimacy of a contract after it has been concluded.Furthermore, important phrases can be read aloud, and the recorded audio data can be obtained and attached to the history record as authentication material.

[0114] Such signed documents, image data, and voice data can be encrypted in their original form and stored and managed in a database.

[0115] FIG. 13 also shows the concept of the multi-viewer function.

[0116] The content shared by the sharing user using the terminal can be shared by multiple viewer users using their own terminals.

[0117] Some viewer user terminals are only provided in simple viewer mode, and only designated viewer user terminals can be set to collaborative mode with the sharer user terminal, allowing corrections, annotations, etc. to be applied to shared content.

[0118] FIG. 14 is a configuration diagram of the screen sharing service.

[0119] The client and the counselor can connect to the screen sharing service server using their own user terminals. In this case, they use a shared line and connect to the screen sharing service web server (W-Sharing Web server) via a firewall and a load balancer (LB). The load balancer is a server that distributes processing among multiple servers when there is a lot of traffic generated by an Internet service such as the screen sharing service of this embodiment, and appropriately distributes processing taking into account the increase in server load rate, load volume, speed degradation, etc.

[0120] The mirroring screen sharing module is configured as follows.

[0121] The screen sharing service web server belongs to the DMZ (DeMilitarized Zone). DMZ refers to a subnet that provides external network services between the external and internal networks while protecting the internal network, i.e., a server area that is open to the outside. Firewalls can be installed before and after the DMZ. One is connected to the internal network and the other is connected to the external network.

[0122] The screen sharing service web server transmits HTML documents and objects (images, files, etc.) requested by the web browser via HTTP or HTTPS. The screen sharing service web server is only responsible for reverse proxy processing.

[0123] A reverse proxy performs the function of forwarding data without processing it.

[0124] The screen sharing service WAS receives and processes a requested screen sharing service message, and corresponds to the screen sharing execution unit 13.

[0125] The sharing engine is a screen sharing application that runs on a WAS server.

[0126] A JAVA package is a software package consisting of the tools necessary to develop and run Java-based software.

[0127] The mirroring recording module is configured as follows.

[0128] Record WAS is a server that records the content shared by the screen sharing service. Logging Engine is an application that records the content shared by the screen sharing service.

[0129] Message Queue is a message broker that delivers messages to many users and delegates requests to other APIs to provide quick responses if the processing time for the request is long.

[0130] The Record service provides management and video streaming functions.

[0131] The screen sharing service providing system using the extension tool according to this embodiment can be utilized as follows.

[0132] In a digital environment based on digital transformation in the financial sector, financial sales staff can collaborate and share tasks such as electronic signatures, product design, and payment requests between customers and staff in environments such as ODS (Out Door Sales) and Mobile Branch, regardless of the type of device.

[0133] When building a trap combat system in the defense field where security is essential, it enables access and control of various hardware such as weapons and sensors, supporting high-level security requirements and supporting offline execution to ensure normal operation when the network is disconnected.

[0134] In the field of e-learning, it enables interaction between teachers and students, and allows teachers to share their learning and course content with students' devices in remote locations to provide e-learning services. It also enables collaborative lessons through two-way communication by sharing electronic whiteboards and student devices, and the push function allows interactive collaborative lessons between teachers and students.

[0135] As for public services, it will enable sharing of identity authentication kiosks and RMB-free issuing machines with customers' terminal environments when entering and leaving airports and ports, enabling public institutions to provide both face-to-face and non-face-to-face shared services to citizens.

[0136] In a digital twin-based manufacturing environment, dynamic simulations of physical objects and systems, primarily in the manufacturing sector, can be digitized, providing a user terminal environment for simulating the design, operation, and maintenance of production equipment, as well as virtual training for on-site workers through collaboration and sharing.

[0137] In the field of digital medicine, it is being used to build remote medical treatment systems that support real-time sharing of biometric information collected using IoT devices, and it can be used to link medical devices with EMR (Electronic Medical Record) systems.

[0138] Collaborative software development environments can support shared development tools that enable collaboration, joint ownership, and horizontal knowledge sharing and real-time code review.

[0139] In the call center field, when a customer requests help while using an application, non-face-to-face support is possible by using a call and sharing service. The customer can share their screen with the customer through the sharing service to check the inquiry and provide guidance using the screen guide function. This makes it easier and faster to handle customer inquiries than by calling or messaging.

[0140] FIG. 15 is a diagram showing the configuration of a system according to an embodiment of the present invention.

[0141] 15, the screen sharing service server 10 includes a processor 410 and a memory 420. The memory 420 stores one or more instructions executable by the processor 410. The processor 410 executes one or more instructions stored in the memory 420. The processor 410 can perform one or more of the operations described above with reference to FIGS. 2 to 14 by executing the instructions. Furthermore, the configurations of the present invention described above with reference to FIG. 1 may be configured to be embodied by instructions executed by the processor 410.

[0142] The above-described method for providing a screen sharing service may also be embodied in the form of a recording medium containing computer-executable instructions, such as an application or program module executed by a computer. A computer-readable medium may be any available medium accessible by a computer, including both volatile and nonvolatile media, and both separate and non-separate media. Furthermore, a computer-readable medium may include a computer recording medium. A computer recording medium includes both volatile and non-volatile, separate and non-separate media embodied in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data.

[0143] The above-described screen sharing service providing method may be performed by an application (which may include a program included in a platform or operating system that is pre-installed on the terminal) that is pre-installed on the terminal, or may be performed by an application (i.e., a program) that a user directly installs on the master terminal via an application providing server such as an application store server, an application, or a web server related to the service. In this sense, the above-described screen sharing service generating method may be implemented by an application (i.e., a program) that is pre-installed on the terminal or that a user directly installs on the master terminal, and may be recorded on a computer-readable recording medium such as a terminal.

[0144] Although the present invention has been described above with reference to one embodiment, it will be understood by those skilled in the art that various modifications and variations of the present invention can be made without departing from the spirit and scope of the present invention as set forth in the claims below. [Explanation of symbols]

[0145] 1. Screen sharing service provision system 10 Screen Sharing Service Server 11 Extension Tool Provision Department 12 Request receiving section 13 Screen sharing execution unit 14. AI Chatbot Response Unit 30 Web browsing service servers 31 Extension Tool Registration Section 50 First User Terminal 60 Second User Terminal

Claims

1. A computer program stored on a computer-readable medium for causing a computer to perform a method for providing a screen sharing service using an extension tool, the computer program causing a computer to perform the following steps, the steps comprising: A step of registering a screen sharing related extension tool in an extension tool registration unit of the web browsing service server in an extension tool providing unit of the screen sharing service server; receiving a screen sharing request in response to execution of the screen sharing related extension tool installed in the first user terminal in a request receiving unit of the screen sharing service server; transmitting a screen sharing request notification message to a second user terminal corresponding to the screen sharing request in a screen sharing execution unit of the screen sharing service server; receiving a screen sharing available state notification message from the second user terminal in the screen sharing execution unit; transmitting the screen sharing available state notification message to the first user terminal in the screen sharing execution unit, so that a connection for screen sharing is established between the first user terminal and the second user terminal; The screen sharing related extension tool is an extension program installed in a web browser of the first user terminal, and allows items displayed on the screen of the web browser to be screen shared with the second user terminal; The first user terminal requests and receives the screen sharing related extension tool for installation in the web browser from the web browsing service server; When the screen sharing related extension tool is executed on the first user terminal, A computer program stored on a computer-readable medium, further comprising: a step of installing a screen sharing assistant; a step of automatically generating a screen sharing script using a screen sharing API in the screen sharing assistant; and a step of providing a custom screen sharing script customized to what is displayed on the screen of the web browser based on answers provided in cooperation with an LLM to user queries for modifications to the screen sharing script entered through a chat client provided by the screen sharing assistant.

2. receiving a user query through the chat client provided by the screen sharing assistant; retrieving context information corresponding to a category corresponding to the user query to generate a context; A step of calling a service module according to the category, and generating a prompt for the LLM by adding the context to a prompt for each service provided by the service module; querying the LLM via an LLM interface for a prompt for the LLM and receiving the response in response; 2. The computer program stored on a computer-readable medium of claim 1, further comprising: checking the validity of the answer before providing it to the chat client as a response to the user query.

3. 2. The computer program stored on a computer-readable medium according to claim 1, wherein the screen sharing script includes one or more of a sharing start API code, a connection URL for accessing a web page for screen sharing, a service providing script for providing various related services during the screen sharing, and a sharing end API code.

4. 3. The computer program stored on a computer-readable medium of claim 2, wherein the category is a screen sharing category, the called service module receives parameter values ​​corresponding to a screen sharing request, and the response is a result processed through the LLM and is a JSON string required to execute a screen sharing API.

5. The step of transmitting the screen sharing request notification message to the second user terminal includes providing a connection URL included in the screen sharing script together; 2. The computer program stored on a computer-readable medium according to claim 1, wherein the second user terminal transmits the screen sharing available status notification message after connecting to a web page for screen sharing via the connection URL.

6. A screen sharing service server included in a screen sharing service providing system using an extension tool, an extension tool providing unit that provides a screen sharing related extension tool to the web browsing service server and registers the screen sharing related extension tool in the extension tool registration unit of the web browsing service server; a request receiving unit that receives a screen sharing request in response to execution of the screen sharing related extension tool installed in the first user terminal; a screen sharing execution unit that transmits a screen sharing request notification message to a second user terminal corresponding to the screen sharing request, receives a screen sharing available status notification message from the second user terminal, and transmits the screen sharing available status notification message to the first user terminal so that a connection for screen sharing is established between the first user terminal and the second user terminal; The screen sharing related extension tool is an extension program installed in a web browser of the first user terminal, and allows the second user terminal to share the screen of the web browser with the first user terminal; The first user terminal requests and receives the screen sharing related extension tool for installation in the web browser from the web browsing service server; When the screen sharing related extension tool is executed on the first user terminal, A screen sharing service server having a screen sharing assistant installed therein, the screen sharing assistant automatically generating a screen sharing script using a screen sharing API, and providing a custom screen sharing script customized for items to be displayed on the screen of the web browser based on answers provided in cooperation with an LLM in response to user queries for modifications to the screen sharing script input through a chat client provided by the screen sharing assistant.

7. 7. The screen sharing service server of claim 6, further comprising an artificial intelligence chatbot response unit that calls context information corresponding to a category corresponding to a user query input through the chat client provided by the screen sharing assistant to generate a context, calls a service module according to the category, adds the context to a prompt for each service provided by the service module to generate a prompt for an LLM, queries the LLM for the prompt for the LLM via an LLM interface, receives the corresponding answer, checks the validity of the answer, and then provides the answer to the chat client as a response to the user query.

8. The screen sharing service server of claim 6, wherein the screen sharing script includes one or more of a sharing start API code, a connection URL for accessing a web page for screen sharing, a service provision script for providing various related services during the screen sharing, and a sharing end API code.

9. The screen sharing service server of claim 7, wherein the category is a screen sharing category, the called service module receives parameter values ​​corresponding to a screen sharing request, and the response is a JSON string processed through an LLM and required to execute a screen sharing API.

10. the screen sharing execution unit provides a connection URL included in the screen sharing script together with the screen sharing request notification message when transmitting the screen sharing request notification message to the second user terminal; The screen sharing service server according to claim 6, wherein the second user terminal connects to a web page for screen sharing via the connection URL and then transmits the screen sharing available status notification message.

Images