Display device and method of human-computer interaction

Abstract

The application discloses a display device and a man-machine interaction method, which is used for improving conference efficiency and improving user interaction experience in the way of simplifying man-machine interaction process and improving man-machine interaction efficiency. The display device comprises a display screen and a control circuit, wherein the display screen is used for displaying content; the control circuit comprises a processor and a memory, the memory is used for storing programs executable by the processor, and the processor is used for reading programs in the memory and performing the following steps: displaying application content of an application program; in response to a first operation of a user on the application content, triggering a menu item to be displayed; wherein the first operation comprises a handwriting operation, a handwriting trace of the handwriting operation and a position of the application content are corresponding, the menu item is used for performing corresponding processing on the application content, and the menu item and the application program are corresponding.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a display device and a method for human-computer interaction. Background Technology

[0002] Meetings are one of the most common forms of work, but with the continuous development of the times and the continuous advancement of technology, traditional meeting models are no longer suitable for the needs of modern offices. Meeting efficiency and interactive experience are becoming increasingly important in corporate meetings. This has led to the development of smart meetings, which are office products that integrate multiple devices such as electronic whiteboards, computers, and remote conferencing terminals, and possess various functions including interactive smart whiteboards.

[0003] With the rise of smart conferencing, meeting efficiency and interactive experience have become increasingly important in enterprise meetings. Improving these aspects is a key concern. Summary of the Invention

[0004] This invention provides a display device and a method for human-computer interaction, which improves meeting efficiency and enhances the user's interactive experience by simplifying the human-computer interaction process and improving human-computer interaction efficiency.

[0005] In a first aspect, an embodiment of the present invention provides a display device, which includes a display screen and a control circuit, wherein:

[0006] The display screen is used to display content;

[0007] The control circuit includes a processor and a memory. The memory stores programs executable by the processor, and the processor reads the programs from the memory and performs the following steps:

[0008] Displays the application's content;

[0009] In response to a user's first operation on the application content, a menu item is triggered to be displayed; wherein the first operation includes a handwriting operation, the handwriting of the handwriting operation corresponds to the position of the application content, the menu item is used to perform corresponding processing on the application content, and the menu item corresponds to the application.

[0010] Secondly, an embodiment of the present invention provides a human-computer interaction method, comprising:

[0011] Displays the application's content;

[0012] In response to a user's first operation on the application content, a menu item is triggered to be displayed; wherein the first operation includes a handwriting operation, the handwriting of the handwriting operation corresponds to the position of the application content, the menu item is used to perform corresponding processing on the application content, and the menu item corresponds to the application.

[0013] Thirdly, embodiments of the present invention also provide an electronic device, including a processor and a memory, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and perform the following steps:

[0014] Displays the application's content;

[0015] In response to a user's first operation on the application content, a menu item is triggered to be displayed; wherein the first operation includes a handwriting operation, the handwriting of the handwriting operation corresponds to the position of the application content, the menu item is used to perform corresponding processing on the application content, and the menu item corresponds to the application.

[0016] Fourthly, embodiments of the present invention also provide a human-computer interaction device, comprising:

[0017] The application display module is used to display the application content of the application.

[0018] A menu display module is used to trigger the display of menu items in response to a user's first operation on the application content; wherein the first operation includes a handwriting operation, the handwriting of the handwriting operation corresponds to the position of the application content, the menu item is used to perform corresponding processing on the application content, and the menu item corresponds to the application.

[0019] Fifthly, embodiments of the present invention also provide a computer storage medium having a computer program stored thereon, which, when executed by a processor, is used to implement the steps of the method described in any of the second aspects above.

[0020] In a sixth aspect, this application provides a computer program product comprising: computer program code, which, when run on a computer, causes the computer to perform the method described in any one of the second aspects.

[0021] These or other aspects of this application will become more apparent in the following description of embodiments. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A schematic diagram of a display device provided in an embodiment of the present invention;

[0024] Figure 2A-2B A schematic diagram of a portable user interface for a file manager provided in an embodiment of the present invention;

[0025] Figure 2C This is a schematic diagram of a file sharing display interface provided in an embodiment of the present invention;

[0026] Figure 2D A schematic diagram of a document printing display interface provided in an embodiment of the present invention;

[0027] Figure 2E This is a schematic diagram of a document translation display interface provided in an embodiment of the present invention;

[0028] Figures 3A-3B This is a schematic diagram of a portable operation interface for generating a table of screen-projected content, provided by an embodiment of the present invention.

[0029] Figure 3C This is a schematic diagram of a screen projection data table generation interface provided in an embodiment of the present invention;

[0030] Figures 4A-4B This is a schematic diagram of a convenient operation interface for searching screen-casting content, provided by an embodiment of the present invention.

[0031] Figure 4C This is a schematic diagram of a display interface for searching screen-projected content, provided by an embodiment of the present invention.

[0032] Figures 5A-5B This is a schematic diagram of an operation interface for generating screen projection content to be done, provided by an embodiment of the present invention.

[0033] Figure 5C This is a schematic diagram of a display interface for generating screen-projected content to be done, provided by an embodiment of the present invention.

[0034] Figures 6A-6B A schematic diagram of a portable operation interface for handwritten content provided in an embodiment of the present invention;

[0035] Figure 6C This is a schematic diagram of a handwritten content generation interface provided in an embodiment of the present invention;

[0036] Figure 7 A flowchart illustrating the implementation of a human-computer interaction method provided in an embodiment of the present invention;

[0037] Figure 8 A schematic diagram of an electronic device provided in an embodiment of the present invention;

[0038] Figure 9 This is a schematic diagram of a human-computer interaction device provided in an embodiment of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0040] In this embodiment of the invention, the term "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0041] The application scenarios described in the embodiments of this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. Those skilled in the art will understand that with the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems. In the description of this invention, unless otherwise stated, "multiple" means two or more.

[0042] Before introducing the display device and human-computer interaction method provided in the embodiments of this application, the technical background of the embodiments of this application will be described in detail below for ease of understanding.

[0043] Meetings are one of the most common forms of work, but with the continuous development of the times and the advancement of technology, traditional meeting models are no longer suitable for the needs of modern offices. Meeting efficiency and interactive experience are becoming increasingly important in corporate meetings. This has led to the development of smart meetings, office products that integrate multiple devices such as electronic whiteboards, computers, and remote conferencing terminals, possessing various functions including interactive smart whiteboards. With the rise of smart meetings, the importance of meeting efficiency and interactive experience in corporate meetings has become increasingly apparent. Therefore, how to improve the meeting efficiency and interactive experience of smart meetings is a major concern at present.

[0044] Taking the file manager as an example, currently, if users want to share or print a file, they need to long-press the document to enter edit mode, then select the file they want to work with, and then click the corresponding button in the bottom menu bar. The operation process is cumbersome and the user experience is poor. Taking the search function as another example, if users don't understand the content displayed on the meeting's screen and need to perform a search, they need to switch from the display device (i.e., the PC) to a third-party large model website to search. The switching process is complex, difficult to operate, and results in a poor user experience. Taking the handwriting interface as yet another example, if users want to logically organize their handwriting, they need to spend a lot of time manually organizing it after the meeting. This is inefficient, time-consuming, and labor-intensive, resulting in a poor user experience.

[0045] To address the aforementioned technical problems, this embodiment provides a display device and a method for human-computer interaction. The core idea is to improve human-computer interaction efficiency and enhance the user's interactive experience by simplifying the interaction process. This embodiment provides corresponding interaction solutions for different interaction scenarios, allowing users to simplify operation processes in any scenario, effectively reducing interaction steps and making human-computer interaction more convenient and efficient.

[0046] It should be noted that the display device in this embodiment refers to a large-size (generally 50 inches or more) intelligent interactive display device with touch handwriting function; the display device is equipped with a file manager, screen projection program, handwriting program, etc., and users can use their fingers, smart pens (active pens or passive pens), etc. to write or perform other touch operations on the display device. The display device may include display modules such as liquid crystal display (LCD), organic light-emitting semiconductor (OLED), and electronic ink, which can display the handwriting in real time.

[0047] like Figure 1 As shown, this embodiment provides a display device, which includes a display screen 100 and a control circuit 101, wherein:

[0048] The display screen 100 is used to display content;

[0049] The control circuit 101 includes a processor and a memory. The memory stores programs executable by the processor, and the processor reads the programs from the memory and performs the following steps:

[0050] Displays the application's content;

[0051] In response to a user's first operation on the application content, a menu item is triggered to be displayed; wherein the first operation includes a handwriting operation, the handwriting of the handwriting operation corresponds to the position of the application content, the menu item is used to perform corresponding processing on the application content, and the menu item corresponds to the application.

[0052] Optionally, the first operation in this embodiment includes a handwriting operation. The user uses a stylus or finger to perform a handwriting operation on the display interface. The position of the handwriting marks the location of the application content, thereby specifying the application content by handwriting and processing the application content according to the menu item. For example, selecting application content by handwriting selection effectively improves interaction efficiency. The position of the handwriting and the position of the application content correspond. For example, if the handwriting is a closed area, the application content within the closed area is the application content corresponding to that handwriting; if the handwriting is a line, the application content above the line is the application content corresponding to that handwriting.

[0053] In practice, this embodiment simplifies the human-computer interaction process through a one-step operation. By performing a first operation on the application content displayed on the application interface, menu items related to the application content are triggered. The user's operation intent is understood through a faster and simpler operation method, and relevant menu items are displayed to provide the user with convenient subsequent operations.

[0054] In some embodiments, the following steps may also be performed:

[0055] In response to a user's first operation on the application content, multiple menu items are displayed; in response to a user's instruction to select a menu item from the multiple menu items, corresponding processing is performed on the application content according to the selected menu item.

[0056] Optionally, the applications in this embodiment include, but are not limited to, at least one of: a file manager, a screen mirroring program, and a handwriting program. The file manager is a tool for managing files and folders, with main functions including, but not limited to, viewing, copying, moving, deleting files and folders, and creating new files or folders. The file manager provides a user graphical interface, allowing users to easily browse and manipulate the file system stored on the device. The file manager also supports batch operations, file search, and customizable display modes, enabling users to manage and find files more efficiently. The file manager also supports cloud storage connections, allowing users to upload local files to the cloud or download files from the cloud, achieving file sharing. A screen mirroring program is software or an application that allows users to wirelessly or via cable transmit the screen content of one device (such as a mobile phone, tablet, or computer) to another device, such as a large-screen display device, television, computer monitor, or other devices. Screen mirroring programs are typically used to display the content of mobile devices on a large screen, such as in conference presentations, video viewing, and live game streaming. Screen mirroring can be achieved via wireless connections (such as Wi-Fi Direct) or wired connections (such as HDMI (High Definition Multimedia Interface) or USB (Universal Serial Bus), depending on device and software compatibility and user needs. Screen mirroring programs include, but are not limited to, wireless screen mirroring (such as Share software screen mirroring) and wired screen mirroring (such as HDMI screen mirroring). Screen mirroring programs support different operating systems and devices, such as Android, iOS, Windows, and macOS, to ensure a broad user base. Screen mirroring programs can also allow users to control the original device (such as a mobile phone) from the mirroring device (such as a computer), which is very useful for remote presentations or operation guidance. Screen mirroring programs typically support video, audio, and potentially interactive content, such as games or applications. For enterprise and educational fields, screen mirroring programs also provide a certain level of security to prevent the leakage of sensitive information. Handwriting programs include, but are not limited to, applications such as electronic whiteboards, used to implement handwriting interaction functions. Electronic whiteboards are multimedia devices that combine computer technology, microelectronics technology, and electronic communication technology, and are widely used in teaching and conference settings. Electronic whiteboards have created a new, electronic, and intelligent teaching / conference model. They can not only display and operate multimedia content, but also allow users to write, draw, and interact with content using a touch screen. They combine the functions of traditional whiteboards with computer technology, enabling users to display and operate multimedia content.

[0057] In implementation, when the application is a file manager, its content includes, but is not limited to, at least one file or document. Users can operate on any one or more files, triggering the display of menu items. These menu items are used for file-related processing, such as printing, sharing, translating, and searching. When the application is a screen mirroring application, its content includes, but is not limited to, the mirrored screen. Users can operate on the mirrored screen, triggering the display of menu items. These menu items are used for processing the mirrored content, such as generating charts, searching, and generating to-do lists. When the application is a handwriting application, its content includes handwritten content. Users can operate on the handwritten content, triggering the display of menu items. These menu items are used for processing the handwritten content, such as generating tables, mind maps, and flowcharts. By providing menu items that meet the user's intent through a single user action on the application content, users can directly click on the menu items to process the application content accordingly. This simplifies the interaction process in different interaction scenarios. By setting menu items corresponding to the application, the menu items displayed after the user's first action in different application scenarios are those corresponding to that application scenario, thus satisfying the user's initial action intent and providing the menu items that the user may need in that application scenario. Users only need to click on the menu items again to process the application content, effectively simplifying the user interaction process, improving interaction efficiency, and enhancing the user's interaction experience.

[0058] Optionally, in this embodiment, the application and menu items are corresponding. Alternatively, the menu items corresponding to the application can be determined based on the type of application content specified in the first operation. For example, when the application is a screen mirroring application, if the application content specified in the first operation performed by the user is numbers, the displayed menu items include generating a table; if the application content specified in the first operation performed by the user is text, the displayed menu items include search, to-do list, translation, etc. The menu items displaying the corresponding functions are determined based on the type of application content.

[0059] In some embodiments, the processor in this embodiment is further configured to execute:

[0060] The application content specified by the first operation is determined based on the location of the handwritten strokes in the first operation.

[0061] In response to the user's second action on the menu item, the specified application content is processed and the processing result is displayed.

[0062] In practice, after performing the first operation, the user first determines the application content specified in the first operation. Then, the user performs a second operation on the menu items, selects one of the menu items, processes the specified application content according to the function of the menu item corresponding to the second operation, and displays the processing result.

[0063] For example, when the application is a file manager, its content includes, but is not limited to, at least one file or document. Users can operate on any one or more files, triggering the display of menu items. These menu items are used for file-related processing, such as printing, sharing, translating, and searching. A second operation on a menu item allows the user to perform similar actions on a specified file. Similarly, when the application is a screen mirroring application, its content includes, but is not limited to, the mirrored screen. Users can operate on the mirrored screen, triggering the display of menu items. These menu items are used for processing the mirrored content, such as generating charts, searching, and creating to-do lists. A second operation on a menu item allows the user to perform similar actions on a specified mirrored content. Finally, when the application is a handwriting application, its content includes handwritten content. Users can operate on the handwritten content, triggering the display of menu items. These menu items are used for processing the handwritten content, such as generating tables, mind maps, and flowcharts. Users can perform a second operation on the menu item to process the handwritten content they specify, such as generating tables, mind maps, or flowcharts.

[0064] In some embodiments, in order not to affect the display of application content, the processing result can be displayed through a floating window; or, the processing result can be displayed in the application interface of the application.

[0065] In practice, the processing result of the user's specified content in the first operation can be displayed in a floating window, or the processing result can be inserted into the application interface for display. There are no strict limitations on the position of the floating window or the inserted processing result; it can be placed in the center of the display interface or in other locations.

[0066] Optionally, users can also drag, full-screen, minimize, and close the floating window for convenient viewing. In practice, for example, the floating window can display download links or sharing QR codes for files, or display file content, translated content, etc. It can also display generated charts, search results, to-do items, generated flowcharts, mind maps, tables, and other content.

[0067] In some embodiments, after displaying the processing result via a floating window, the processing result can also be processed in response to a third user action on the processing result.

[0068] During implementation, users can download the corresponding file by scanning the shared QR code, save the generated charts, print the displayed file content, and browse, save, and share the translated content.

[0069] In some embodiments, the first operation in this embodiment may be to press and hold the smart pen button while using the smart pen to handwrite on the application content; or it may be to use a finger to click a virtual button while using the smart pen or a finger to handwrite on the application content. The handwriting operation includes, but is not limited to, drawing shapes, such as closed or semi-closed areas, horizontal lines, checkmarks, triangles, rectangles, circles, irregular shapes, etc. For example, the user may select application content and mark its location by drawing a shape. This embodiment does not impose excessive limitations on this.

[0070] It should be noted that the first operation in this embodiment can simultaneously achieve two functions: one is triggering entry into the selection mode, and the other is allowing the user to select application content. While the user selects application content, the system understands the user's intent and displays the corresponding menu item for that application scenario. Triggering the selection mode can be done by pressing and holding a button on the smart pen or by using a virtual button in the software. However, simultaneously, the user needs to handwrite on the application content, selecting the content based on the location of the handwriting. This not only confirms the selected application content but also simultaneously understands the user's intent to perform related processing on that application content, thereby simultaneously displaying the corresponding menu item for that application. This simplifies the user's interaction process for processing the application content and improves interaction efficiency.

[0071] Optionally, the second operation in this embodiment includes, but is not limited to, one or more of the following operations: clicking, double-clicking, and long-pressing.

[0072] In some embodiments, taking an application that includes a file manager as an example, the menu items corresponding to the file manager include a first menu item; the processor is specifically configured to execute:

[0073] In response to the user's first action on a file displayed in the file manager, a first menu item is triggered and displayed; wherein, the first menu item is used to perform related processing on the file. Optionally, the first menu item includes, but is not limited to: share, print, search, etc.

[0074] During implementation, such as Figure 2A-2B As shown, this embodiment provides a portable operating interface for a file manager. The application interface of the file manager displays information about multiple files. The user presses and holds the smart pen button while selecting (by drawing closed or open shapes with the smart pen) the files to be operated on. After releasing the button, multiple first menu items are displayed at the corresponding positions of the selected files. The first menu items include, but are not limited to, sharing, printing, and searching, allowing the user to perform sharing, printing, and searching operations.

[0075] In some embodiments, the processor is further configured to: determine the file specified by the first operation based on the position of the handwritten stroke of the first operation; process the specified file in response to a second operation by the user on the first menu item, and display the processing result.

[0076] In some embodiments, the first menu item is used for sharing a file; the processor is specifically configured to perform: in response to a second operation by the user on the first menu item, sending the specified file to the server; receiving a download link from the server and displaying the download link, wherein the download link is used by the user to access the specified file. Optionally, the download link includes, but is not limited to, a web link address or a QR code.

[0077] During implementation, such as Figure 2C As shown, this embodiment provides a file sharing display interface. After the user clicks share, it first determines whether the display device is connected to the network. If it is not connected to the network, the user is prompted to connect the display device to the network before proceeding. If it is connected to the network, the file is uploaded to the server in the background of the display device assistant mini-program. After successful upload, the server in the background will issue a QR code for file storage link. The display device displays the QR code, and the user can scan the QR code through the mini-program (i.e., the third operation performed by the user) to obtain the file. The user can also browse, save, and forward the file to other users.

[0078] In some embodiments, the first menu item is used for printing a document; the processor is specifically configured to perform: in response to a second operation by the user on the first menu item, launch and display the application interface of the printing program; and in response to a third operation by the user on the application interface, print the specified document. Optionally, the third operation in this embodiment includes, but is not limited to, one or more combinations of clicking, double-clicking, and long-pressing.

[0079] During implementation, such as Figure 2D As shown, this embodiment provides a document printing display interface. After the user clicks print, the display interface of the Android native printing APP is entered. The user can select printing-related settings in the printing display interface to perform third operations, such as selecting / adding a printer, selecting the range of pages to print, selecting printing colors, paper size, printing orientation, and number of copies, etc. After clicking OK, the document can be printed directly.

[0080] In some embodiments, the first menu item is used to translate a file; the processor is specifically configured to perform: in response to a second operation by the user on the first menu item, send the specified file to a third-party service; receive the translation sent by the third-party service, and display the translation.

[0081] During implementation, such as Figure 2E As shown, this embodiment provides a file translation display interface. After the user clicks "Translate" (the second operation performed by the user), it first determines whether the display device is connected to the network. If not connected, the user is prompted to connect the display device to the network before proceeding. If connected, the file selected by the user (i.e., the file specified in the first operation) is uploaded to a third-party translation API (Application Programming Interface). The user selects either foreign language to Chinese or Chinese to English translation. After the third-party translation is completed, the translated file is sent to the display device, which displays the translated file. The user can then perform a third operation on the display device, such as browsing, saving (to a local preset category directory), and sharing (using a mini-program function). The browser window displaying the translated file can also be dragged, turned full-screen, minimized, and closed for convenient viewing.

[0082] In some embodiments, taking an application that includes a screen mirroring program as an example; the menu item corresponding to the screen mirroring program includes a second menu item; the processor is specifically configured to execute:

[0083] In response to the user's first operation on the screen mirroring program, a second menu item is triggered and displayed; wherein, the second menu item is used to perform relevant processing on the screen mirroring content. Optionally, the second menu item includes, but is not limited to: table generation, search, to-do list generation, etc.

[0084] During implementation, such as Figures 3A-3B As shown, this embodiment provides a portable operation interface for generating tables from screen-projected content. When the screen-projected content is a table, the user presses and holds the smart pen button while selecting the desired table data range. After releasing the pen, multiple second menu items are displayed at the corresponding position of the selected table data range. The second menu items include, but are not limited to, table generation, search, and to-do list generation. It can provide three services: table generation, search, and to-do list generation and push. After clicking the table generation menu item, the user selects the desired chart type, such as bar chart, radar chart, pie chart, etc.

[0085] In some embodiments, the processor is further configured to: determine the projection content in the projection screen specified by the first operation based on the position of the handwriting in the first operation; process the projection content in response to the user's second operation on the second menu item, and display the processing result.

[0086] In some embodiments, the second menu item is used to generate a chart for the projected content; the processor is specifically configured to perform: in response to a second operation by the user on the second menu item, perform text recognition on the projected content to obtain a recognition result; generate a chart corresponding to the projected content based on the recognition result, and display the chart.

[0087] During implementation, such as Figure 3C As shown, this embodiment provides a screen projection data table generation interface. After the user selects the table generation menu item, the OCR (Optical Character Recognition) algorithm is used to recognize the selected screen projection content, i.e., the table data. After recognition, the corresponding table is generated using an Excel plugin or a generation logic consistent with Excel. Optionally, the user can also customize the table style. The generated table can be displayed in a floating window, allowing the user to preview the window and perform operations such as dragging, full-screen, minimizing, and closing the floating window. The user can also save a screenshot of the generated table as a JPG image to the local directory. The user can also share the generated table.

[0088] Optionally, screen mirroring can be done via shared mirroring or HDMI (High Definition Multimedia Interface) mirroring. When using HDMI mirroring, users can click to share and generate a QR code, which can then be scanned to obtain a screenshot of the table. When using shared mirroring, users can click to share the generated table to the mirroring terminal or share it via QR code. If sharing to the mirroring terminal is chosen, the shared mirroring transmission protocol will save the table screenshot to the local path of the mirroring terminal. This transmission protocol includes, but is not limited to, P2P (Peer-to-Peer) technology. P2P is a network transmission technology that allows users on a network to directly share files without a central server. This transmission method allows users to exchange files directly without a central server, thereby improving the efficiency and speed of file transfer.

[0089] In some embodiments, the second menu item is used for intent understanding processing of the projected content; the processor is specifically configured to perform: in response to a second operation by the user on the second menu item, triggering the display of a submenu of the second menu item; in response to a selection operation by the user on a submenu item in the submenu, triggering text recognition of the projected content, determining intent prompt information based on the recognition result and the submenu item corresponding to the selection operation; sending the intent prompt information to a third-party model, receiving the processing result sent by the third-party model, and displaying the processing result.

[0090] During implementation, such as Figures 4A-4B As shown, this embodiment provides a convenient interface for searching content on a projected screen. The projected meeting materials are displayed in the current display area. The user presses and holds the smart pen button while simultaneously selecting the text range to search. Upon releasing the pen, multiple second menu items appear at the corresponding position of the selected content. These second menu items include, but are not limited to, table generation, search (intent understanding), and to-do list generation. After selecting search (intent understanding), the user can further select at least one of the following functions: definition, brainstorming (expanded explanations of definitions), or Q&A. Definition, brainstorming, and Q&A are sub-menu items of intent understanding (second menu item). The prompts for inputting different second menu items into the third-party large-scale model are different. First, it is determined whether the display device is connected to the network. If not, the user is prompted to connect the display device to the network before searching. If connected, OCR recognition is performed on the text in the user-selected area. Different intent prompts, such as prompts, are used based on the user's selection. The OCR recognition result is input into the third-party large-scale model for inference, and the processing result is displayed through a floating window. Figure 4C As shown, this embodiment provides a display interface for searching projected content. If the user is not satisfied with the processing result, they can input their further requirements via voice or text in the input box. Simultaneously, the OCR recognition result and the user's further requirements are input into a third-party large model to obtain more detailed processing results. The user can also drag, full-screen, minimize, and close the floating window. The user can also save the processing results of the large model to the local file directory of the display device. A download link for the processing result, such as a QR code, can also be displayed, allowing the user to obtain the processing result by scanning the QR code. The user can browse, save, and share the processing results (through a mini-program). The display device is equipped with OCR recognition algorithms and voice input recognition algorithms, which can perform recognition without an internet connection. The third-party large model is a cloud-based algorithm that requires the display device to be connected to the internet to execute.

[0091] OCR (Optical Character Recognition) algorithms are technologies that convert text in images into editable text. They acquire text image information from paper through scanning and photography, analyze the morphological features of the text using pattern recognition algorithms, and convert the image information into a computer-recognizable text format.

[0092] In some embodiments, the second menu item is used to generate to-do items from the projected content; the processor is specifically configured to perform: in response to a second operation by the user on the second menu item, take a screenshot of the projected content to obtain a projected image, and perform text recognition on the projected content to obtain a recognition result; send the projected image and the recognition result to the to-do service program of the target user, wherein the target user refers to a user registered in the to-do service program that has interacted with the display device.

[0093] During implementation, such as Figures 5A-5B As shown, this embodiment provides an operation interface for generating to-do items for screen projection. The projected meeting materials are displayed in the current display area. The user presses and holds the smart pen button while simultaneously selecting the text area to be manipulated. After releasing the pen, multiple second menu items are displayed at the corresponding position of the selected content. These second menu items include, but are not limited to, table generation, search, and to-do item generation. Figure 5C As shown, this embodiment provides a display interface for generating to-do items for screen sharing. After the user selects to generate a to-do item, it first checks whether the display device is connected to the network. If not, the user is prompted to connect the display device to the network before proceeding. If connected, a contact list is displayed. This list includes user information who have interacted with the display device using the display device's assistant app (including users who have performed any of the following operations on the display device: file transfer, QR code check-in, voting, collaborative whiteboard, etc.). Clicking on a user in the list directly sends a screenshot of the selected area and the OCR-recognized text information to the app. The user can see the screenshot and text information in the application's service notification or the app. This facilitates notifying users who did not attend the meeting and generating to-do items for them. If the target user to whom the user wants to share has never used the app on the display device, a QR code can be generated. After the target user scans the code and adds the user to the contact list, the text information and screenshot can be shared with the target user. In other words, the target user first registers and is added to the contact list, and then the text or screenshot to be shared is provided to the user for download, improving the security and confidentiality of the meeting content. The display device is equipped with OCR recognition algorithms and voice input recognition algorithms, which can perform recognition without the need for an internet connection.

[0094] In some embodiments, taking an application that includes a handwriting program as an example; the menu item corresponding to the handwriting program includes a third menu item; the processor is specifically configured to execute:

[0095] In response to the user's first operation on the handwritten content displayed on the handwriting interface, a third menu item is triggered and displayed; wherein, the third menu item is used to perform related processing on the handwritten content. Optionally, the third menu item includes, but is not limited to: table generation, mind map generation, flowchart generation, etc.

[0096] In some embodiments, the processor is further configured to: determine the handwritten content specified by the first operation based on the position of the handwritten stroke of the first operation; process the specified handwritten content in response to the user's second operation on the third menu item, and display the processing result.

[0097] During implementation, such as Figures 6A-6B As shown, this embodiment provides a portable operation interface for handwritten content. In the handwriting application, the user performs handwriting operations, and the handwritten content is displayed. After the user finishes writing, they press and hold the smart pen button while simultaneously selecting the desired area for operation. After selection, multiple third menu items pop up, including but not limited to: table generation, mind map generation, and flowchart generation. The user can choose to generate tables, mind maps, flowcharts, etc.

[0098] In some embodiments, the processor is specifically configured to perform: in response to a second operation by the user on the third menu item, to recognize the specified handwritten content and determine intent prompt information based on the recognition result; to send the intent prompt information to a third-party model and to receive parsed text sent by the third-party model, wherein the parsed text is used to represent the hierarchical relationship of the handwritten content; to generate a graphical representation based on the parsed text and to display the graphical representation.

[0099] During implementation, such as Figure 6C As shown, this embodiment provides a handwritten content generation interface. After selecting one of the third menu items, it first determines whether the display device is connected to the network. If not, the user is prompted to connect the display device to the network before searching. If connected, the selected handwritten handwriting is OCR-recognized and recognized as a standard font. Based on the recognized content, appropriate intent prompts, such as prompt words, are selected and input into a third-party large model. The model then analyzes the hierarchy and flow relationships of the recognized content, generating hierarchical parsed text. This parsed text is returned to one or more of the built-in table generation, mind map generation, or flowchart generation tools on the display device to generate editable tables, mind maps, or flowcharts, which are then inserted into the handwriting interface for display. Users can edit the content and format of the generated tables, mind maps, or flowcharts. The display device is equipped with OCR recognition algorithms and voice input recognition algorithms, which can perform recognition without a network connection. The third-party large model requires the display device to be connected to the network to execute, thereby obtaining the parsed text from the third-party large model. This embodiment places the OCR recognition algorithm locally for processing, which can reduce server costs.

[0100] This embodiment provides a convenient operating experience for users in different application scenarios through a convenient triggering method. For example, when a user presses and holds the writing button on the smart pen to select the area to be operated, different operation menu items are recommended according to different application scenarios. It provides a new interactive method of selection based on intent understanding, as well as corresponding recommendation services in common large-screen scenarios such as file manager, screen projection, and handwriting. In the file manager scenario, menu items such as print, share, and translate can be recommended after the user selects the file. In the screen projection scenario, menu items such as charts, search, and to-do items can be recommended after the user selects the screen projection content. In the handwriting scenario, menu items such as generating tables, mind maps, and flowcharts can be recommended after the user selects the handwriting. The human-computer interaction method provided in this embodiment can solve the pain point of the current large-screen interaction requiring multiple cumbersome operations. Taking the file manager as an example, to share or print a file, one must first long-press the document to enter the editing mode, then select multiple files to operate on, and then click the corresponding button in the bottom menu bar. With this application, after selecting, the share or print menu item will pop up directly near the selected position, allowing the user to operate in one step. It can also address the pain point that current large-screen displays lack sufficient intelligence, requiring many operations to be completed on mobile phones or PCs. For example, in a meeting, if the content is unclear, users often need to switch to a large-scale website on a PC to search. This application allows users to perform a search after selecting the relevant information. In handwriting scenarios, users' handwriting can be messy, requiring significant time after the meeting to organize the logical relationships. This application's interactive method can directly generate structured diagrams such as tables, flowcharts, and mind maps. This effectively improves interaction efficiency, simplifies the interaction process, and enhances the user experience.

[0101] Based on the same inventive concept, this embodiment of the invention also provides a method for human-computer interaction. Since the principle of this method in solving the problem is similar to that of the display device, the implementation of this method can refer to the implementation of the display device, and the repeated parts will not be described again.

[0102] like Figure 7 As shown, the implementation flow of the human-computer interaction method provided in this embodiment is as follows:

[0103] Step 700: Display the application's content;

[0104] Step 701: In response to the user's first operation on the application content, trigger the display of menu items; wherein the first operation includes a handwriting operation, the handwriting of the handwriting operation corresponds to the position of the application content, the menu item is used to perform corresponding processing on the application content, and the menu item corresponds to the application.

[0105] As an optional implementation, the method further includes:

[0106] In response to the user's first action on the application content, multiple menu items are triggered to be displayed;

[0107] In response to a user's instruction to select a menu item from multiple menu items, the application content is processed accordingly based on the selected menu item.

[0108] As an optional implementation, the method further includes:

[0109] The application content specified by the first operation is determined based on the location of the handwritten strokes in the first operation.

[0110] In response to the user's second action on the menu item, the specified application content is processed and the processing result is displayed.

[0111] As an optional implementation, the method further includes:

[0112] The processing results are displayed via a floating window.

[0113] As an optional implementation, the method further includes:

[0114] In response to a third user action on the processing result, the corresponding processing is performed on the processing result.

[0115] As an optional implementation, the application includes a file manager; the menu items corresponding to the file manager include a first menu item; the step of triggering the display of the menu item in response to a user's first operation on the application content includes:

[0116] In response to the user's first action on the files displayed in the file manager, the first menu item is triggered and displayed.

[0117] The first menu item is used to perform related processing on the file.

[0118] As an optional implementation, the method further includes:

[0119] The file specified in the first operation is determined based on the location of the handwritten strokes in the first operation;

[0120] In response to the user's second operation on the first menu item, the specified file is processed and the processing result is displayed.

[0121] As an optional implementation, the first menu item is used for sharing files; the step of processing the specified file and displaying the processing result in response to a second operation by the user on the first menu item includes:

[0122] In response to the user's second operation on the first menu item, the specified file is sent to the server;

[0123] Receive a download link from the server and display the download link, wherein the download link is used by the user to access the specified file.

[0124] As an optional implementation, the first menu item is used for printing a document; the step of processing the specified document and displaying the processing result in response to a second operation by the user on the first menu item includes:

[0125] In response to the user's second operation on the first menu item, the application interface of the printing program is launched and displayed;

[0126] In response to a third action by the user on the application interface, the specified file is printed.

[0127] As an optional implementation, the first menu item is used for translating a file; the step of processing the specified file and displaying the processing result in response to a second operation by the user on the first menu item includes:

[0128] In response to the user's second action on the first menu item, the specified file is sent to a third-party service;

[0129] Receive translations sent by third-party services and display the translations.

[0130] As an optional implementation, the application includes a screen mirroring program; the menu items corresponding to the screen mirroring program include a second menu item; the step of triggering the display of the menu item in response to a user's first operation on the application content includes:

[0131] In response to the user's first operation on the screen mirroring screen of the screen mirroring program, the second menu item is displayed.

[0132] The second menu item is used to process the content projected onto the screen.

[0133] As an optional implementation, the method further includes:

[0134] Based on the location of the handwritten strokes in the first operation, determine the content to be projected in the screen specified by the first operation.

[0135] In response to the user's second operation on the second menu item, the projected content is processed and the processing result is displayed.

[0136] As an optional implementation, the second menu item is used to generate charts from the projected content; the step of processing the projected content and displaying the processing result in response to the user's second operation on the second menu item includes:

[0137] In response to the user's second operation on the second menu item, text recognition is performed on the projected content to obtain the recognition result;

[0138] Based on the recognition results, a chart corresponding to the projected content is generated and displayed.

[0139] As an optional implementation, the second menu item is used for intent understanding processing of the projected content; the step of processing the projected content and displaying the processing result in response to the user's second operation on the second menu item includes:

[0140] In response to the user's second operation on the second menu item, the submenu of the second menu item is displayed.

[0141] In response to the user's selection of a submenu item in the submenu, text recognition of the projected content is triggered, and intent prompt information is determined based on the recognition result and the submenu item corresponding to the selection operation.

[0142] The intent prompt information is sent to a third-party model, the processing result sent by the third-party model is received, and the processing result is displayed.

[0143] As an optional implementation, the second menu item is used to generate to-do items from the projected content; the processing of the projected content in response to the user's second operation on the second menu item includes:

[0144] In response to the user's second operation on the second menu item, a screenshot of the projected content is taken to obtain a projected image, and text recognition is performed on the projected content to obtain a recognition result;

[0145] The projected image and the recognition result are sent to the to-do service program of the target user, wherein the target user refers to a user registered in the to-do service program who has interacted with the display device.

[0146] As an optional implementation, the application includes a handwriting program; the menu items corresponding to the handwriting program include a third menu item; the step of triggering the display of the menu item in response to a user's first operation on the application content includes:

[0147] In response to the user's first action on the handwritten content displayed on the handwriting interface, the third menu item is triggered and displayed.

[0148] The third menu item is used to process handwritten content.

[0149] As an optional implementation, the method further includes:

[0150] Based on the position of the handwritten strokes in the first operation, determine the handwritten content specified in the first operation;

[0151] In response to the user's second operation on the third menu item, the specified handwritten content is processed and the processing result is displayed.

[0152] As an optional implementation, the step of responding to the user's second operation on the third menu item, processing the specified handwritten content, and displaying the processing result includes:

[0153] In response to the user's second operation on the third menu item, the specified handwritten content is recognized, and the intent prompt information is determined based on the recognition result;

[0154] The intent prompt information is sent to a third-party model, and the parsed text sent by the third-party model is received, wherein the parsed text is used to represent the hierarchical relationship of the handwritten content;

[0155] A graphical representation is generated based on the parsed text, and the graphical representation is displayed.

[0156] Based on the same inventive concept, the embodiments of the present invention also provide an electronic device. Since the electronic device is the same as the electronic device in the method of the embodiments of the present invention, and the principle of the electronic device in solving the problem is similar to that of the method, the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be described again.

[0157] like Figure 8 As shown, the electronic device includes a processor 800 and a memory 801. The memory 801 stores programs executable by the processor 800. The processor 800 reads the programs from the memory 801 and performs the following steps:

[0158] Displays the application's content;

[0159] In response to a user's first operation on the application content, a menu item is triggered to be displayed; wherein the first operation includes a handwriting operation, the handwriting of the handwriting operation corresponds to the position of the application content, the menu item is used to perform corresponding processing on the application content, and the menu item corresponds to the application.

[0160] Based on the same inventive concept, this embodiment of the invention also provides a human-computer interaction device. Since this device is the same as the device in the method of this embodiment of the invention, and the principle of the device in solving the problem is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0161] like Figure 9 As shown, the device includes:

[0162] Application display module 900 is used to display the application content of the application.

[0163] The menu display module 901 is used to trigger the display of menu items in response to a first operation by the user on the application content; wherein the first operation includes a handwriting operation, the handwriting of the handwriting operation corresponds to the position of the application content, the menu item is used to perform corresponding processing on the application content, and the menu item corresponds to the application.

[0164] Based on the same inventive concept, this disclosure provides a computer storage medium comprising: computer program code, which, when executed on a computer, causes the computer to perform any of the human-computer interaction methods described above. Since the principle by which the computer storage medium solves the problem is similar to that of the human-computer interaction methods, the implementation of the computer storage medium can be referred to the implementation of the method, and repeated details will not be elaborated further.

[0165] In specific implementation, computer storage media can include: Universal Serial Bus Flash Drive (USB), portable hard drive, Read-Only Memory (ROM), Random Access Memory (RAM), magnetic disk or optical disk, and other storage media that can store program code.

[0166] Based on the same inventive concept, this disclosure also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to execute any of the human-computer interaction methods described above. Since the principle by which the above-described computer program product solves the problem is similar to that of the human-computer interaction methods, the implementation of the above-described computer program product can be referred to the implementation of the method, and repeated details will not be described again.

[0167] Computer program products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0168] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0169] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 Devices that specify the functions in one or more boxes.

[0170] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction device, which is implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0171] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0172] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A display device, characterized in that, The display device includes a display screen and control circuitry, wherein: The display screen is used to display content; The control circuit includes a processor and a memory. The memory stores programs executable by the processor, and the processor reads the programs from the memory and performs the following steps: Displays the application's content; In response to a user's first operation on the application content, a menu item is triggered to be displayed; wherein the first operation includes a handwriting operation, the handwriting of the handwriting operation corresponds to the position of the application content, the menu item is used to perform corresponding processing on the application content, and the menu item corresponds to the application.

2. The display device according to claim 1, characterized in that, The processor is further configured to execute: In response to the user's first action on the application content, multiple menu items are triggered to be displayed; In response to a user's instruction to select a menu item from multiple menu items, the application content is processed accordingly based on the selected menu item.

3. The display device according to claim 1, characterized in that, The processor is further configured to execute: The application content specified by the first operation is determined based on the location of the handwritten strokes in the first operation. In response to the user's second action on the menu item, the specified application content is processed and the processing result is displayed.

4. The display device according to claim 3, characterized in that, The processor is further configured to execute: The processing results are displayed via a floating window.

5. The display device according to claim 3, characterized in that, The processor is further configured to execute: In response to a third user action on the processing result, the corresponding processing is performed on the processing result.

6. The display device according to claim 1, characterized in that, The application includes a file manager; the menu items corresponding to the file manager include a first menu item; the processor is specifically configured to execute: In response to the user's first action on the files displayed in the file manager, the first menu item is triggered and displayed. The first menu item is used to perform related processing on the file.

7. The display device according to claim 6, characterized in that, The processor is further configured to execute: The file specified in the first operation is determined based on the location of the handwritten strokes in the first operation; In response to the user's second operation on the first menu item, the specified file is processed and the processing result is displayed.

8. The display device according to claim 7, characterized in that, The first menu item is used for file sharing; the processor is specifically configured to execute: In response to the user's second operation on the first menu item, the specified file is sent to the server; Receive a download link from the server and display the download link, wherein the download link is used by the user to access the specified file.

9. The display device according to claim 7, characterized in that, The first menu item is used for printing the document; the processor is specifically configured to execute: In response to the user's second operation on the first menu item, the application interface of the printing program is launched and displayed; In response to a third action by the user on the application interface, the specified file is printed.

10. The display device according to claim 7, characterized in that, The first menu item is used for translating the file; the processor is specifically configured to execute: In response to the user's second action on the first menu item, the specified file is sent to a third-party service; Receive translations sent by third-party services and display the translations.

11. The display device according to claim 1, characterized in that, The application includes a screen mirroring program; the menu item corresponding to the screen mirroring program includes a second menu item; the processor is specifically configured to execute: In response to the user's first operation on the screen mirroring screen of the screen mirroring program, the second menu item is displayed. The second menu item is used to process the content projected onto the screen.

12. The display device according to claim 11, characterized in that, The processor is further configured to execute: Based on the location of the handwritten strokes in the first operation, determine the content to be projected in the screen specified by the first operation. In response to the user's second operation on the second menu item, the projected content is processed and the processing result is displayed.

13. The display device according to claim 12, characterized in that, The second menu item is used to generate charts from the projected content; the processor is specifically configured to execute: In response to the user's second operation on the second menu item, text recognition is performed on the projected content to obtain the recognition result; Based on the recognition results, a chart corresponding to the projected content is generated and displayed.

14. The display device according to claim 12, characterized in that, The second menu item is used for intent understanding processing of the projected content; the processor is specifically configured to execute: In response to the user's second operation on the second menu item, the submenu of the second menu item is triggered and displayed; In response to the user's selection of a submenu item in the submenu, text recognition of the projected content is triggered, and intent prompt information is determined based on the recognition result and the submenu item corresponding to the selection operation. The intent prompt information is sent to a third-party model, the processing result sent by the third-party model is received, and the processing result is displayed.

15. The display device according to claim 12, characterized in that, The second menu item is used to generate to-do items from the projected content; the processor is specifically configured to execute: In response to the user's second operation on the second menu item, a screenshot of the projected content is taken to obtain a projected image, and text recognition is performed on the projected content to obtain a recognition result; The projected image and the recognition result are sent to the to-do service program of the target user, wherein the target user refers to a user registered in the to-do service program who has interacted with the display device.

16. The display device according to claim 1, characterized in that, The application includes a handwriting program; the menu item corresponding to the handwriting program includes a third menu item; the processor is specifically configured to execute: In response to the user's first action on the handwritten content displayed on the handwriting interface, the third menu item is triggered and displayed. The third menu item is used to process handwritten content.

17. The display device according to claim 16, characterized in that, The processor is further configured to execute: Based on the position of the handwritten strokes in the first operation, determine the handwritten content specified in the first operation; In response to the user's second operation on the third menu item, the specified handwritten content is processed and the processing result is displayed.

18. The display device according to claim 17, characterized in that, The processor is specifically configured to execute: In response to the user's second operation on the third menu item, the specified handwritten content is recognized, and the intent prompt information is determined based on the recognition result; The intent prompt information is sent to a third-party model, and the parsed text sent by the third-party model is received, wherein the parsed text is used to represent the hierarchical relationship of the handwritten content; A graphical representation is generated based on the parsed text, and the graphical representation is displayed.

19. A method for human-computer interaction, characterized in that, The method includes: Displays the application's content; In response to a user's first operation on the application content, a menu item is triggered to be displayed; wherein the first operation includes a handwriting operation, the handwriting of the handwriting operation corresponds to the position of the application content, the menu item is used to perform corresponding processing on the application content, and the menu item corresponds to the application.

20. An electronic device, characterized in that, The device includes a processor and a memory for storing a program executable by the processor, and the processor for reading the program in the memory and performing the steps of the method of claim 19.

21. A computer storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in claim 19.

Images