Method, device and equipment for performing man-machine interaction with desktop virtual image and medium

Abstract

The invention discloses a method, device and equipment for performing man-machine interaction with a desktop virtual image, a medium and a product. The method comprises the following steps: in response to a trigger request for a target client for realizing a desktop virtual image function, displaying a client interface with a full-transparent background of the target client in a set screen area of a computer desktop; loading a target Web page with the full-transparent background in the client interface with the full-transparent background; performing dynamic rendering display on the desktop virtual image in the full-transparent background target Web page; and performing man-machine interaction with a user through the desktop virtual image. According to the technical scheme provided by the embodiment of the invention, the desktop interaction effect of the virtual image can be realized only by using the drawing rendering function of the target Web page integrated in the target client without deep cooperation of codes of a computer operating system, so that the development difficulty of developers is effectively reduced, and the development efficiency is improved. And the development efficiency of the desktop virtual image function software is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment, medium and product for human-computer interaction with a desktop virtual image. Background Art

[0002] With the continuous development of artificial intelligence (AI) technology, human-computer interaction is increasingly widely used in various electronic devices such as desktop computers or tablet computers.

[0003] In the related art, in-depth human-computer interaction can be carried out with users through desktop digital assistants. Specifically, a virtual image, such as a virtual person or a virtual pet, can be implemented on the computer desktop. The desktop virtual image can interact with the user in various ways based on the user's input voice, text or clicks.

[0004] In order to develop the desktop virtual image, it is necessary to use the development language supported by the operating system for software development. For those less popular operating systems, developers need to have very professional software development skills, and the operating system also needs to open up sufficient access rights. The development efficiency of the entire software is low and the development is difficult. At the same time, due to the natural gap between different operating systems, natively developed code is almost non-portable between different operating systems. In addition, when the operating system is upgraded, the desktop virtual image often requires a lot of adaptive changes and upgrades. In particular, when the ecology of some operating systems is relatively closed, third-party developers cannot build a desktop virtual image that is compatible with the operating system. Summary of the invention

[0005] The embodiments of the present invention provide a method, apparatus, device, medium and product for human-computer interaction with a desktop virtual image, creatively propose a desktop virtual image realized through Web development, and greatly reduce the difficulty of developing the desktop virtual image.

[0006] According to one aspect of an embodiment of the present invention, a method for human-computer interaction with a desktop virtual image is provided, comprising:

[0007] In response to a trigger request to a target client for implementing a desktop virtual image function, displaying a client interface with a fully transparent background of the target client in a set screen area of ​​a computer desktop;

[0008] In the client interface with a fully transparent background, load the target web page with a fully transparent background;

[0009] In a fully transparent background target Web page, the desktop virtual image is dynamically rendered and displayed, wherein the size of the client interface and the target Web page is adapted to the size of the desktop virtual image;

[0010] Human-computer interaction is performed with the user through the desktop virtual image.

[0011] According to another aspect of an embodiment of the present invention, there is also provided a device for human-computer interaction with a desktop virtual image, comprising:

[0012] A first transparent display module, for displaying a fully transparent background client interface of the target client in a set screen area of ​​the computer desktop in response to a trigger request to the target client for realizing the desktop virtual image function;

[0013] The second transparent display module is used to load a target Web page with a fully transparent background in a client interface with a fully transparent background;

[0014] A desktop virtual image display module is used to dynamically render and display the desktop virtual image in a fully transparent background target Web page, wherein the size of the client interface and the target Web page is adapted to the size of the desktop virtual image;

[0015] The human-computer interaction module is used to perform human-computer interaction with the user through the desktop virtual image.

[0016] According to another aspect of an embodiment of the present invention, an electronic device is provided, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for human-computer interaction with a desktop virtual image described in any embodiment of the present invention.

[0020] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for human-computer interaction with a desktop virtual image described in any embodiment of the present invention when executed.

[0021] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of the method for human-computer interaction with a desktop virtual image as described in any embodiment of the present invention.

[0022] The technical solution of the embodiment of the present invention, by responding to a trigger request of a target client for realizing a desktop virtual image function, displays a client interface with a fully transparent background of the target client in a set screen area of ​​the computer desktop; loads a target Web page with a fully transparent background in the client interface with a fully transparent background; and dynamically renders and displays the desktop virtual image in the target Web page with a fully transparent background; and realizes a human-computer interaction with a user through the desktop virtual image, without the need for in-depth cooperation with the computer operating system code, and only uses the drawing rendering function of the target Web page integrated in the target client to realize the desktop interaction effect of the virtual image, effectively reducing the development difficulty of developers and greatly improving the development efficiency of software for realizing the desktop virtual image function. Since the image rendering function is only realized on the Web page side, there is no need to adapt to the operating system, which greatly enhances the portability of the software. In addition, software development based on Web pages belongs to mainstream development, with more open source resources for reference and richer functions that can be realized, which can further reduce development costs and development time.

[0023] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 is a flow chart of a method for human-computer interaction with a desktop virtual image provided according to an embodiment of the present invention;

[0026] Figure 2 It is a schematic diagram of a bearing relationship between a target client, a target Web page and a desktop virtual image applicable to an embodiment of the present invention;

[0027] Figure 3 is a flowchart of another method for human-computer interaction with a desktop virtual image provided according to an embodiment of the present invention;

[0028] Figure 4 is a flowchart of another method for human-computer interaction with a desktop virtual image provided according to an embodiment of the present invention;

[0029] Figure 5 is a schematic structural diagram of a device for human-computer interaction with a desktop virtual image provided according to an embodiment of the present invention;

[0030] Figure 6 The present invention is a schematic diagram of the structure of an electronic device for implementing a method for human-computer interaction with a desktop virtual image according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Figure 1 A flowchart of a method for human-computer interaction with a desktop virtual image provided by an embodiment of the present invention. This embodiment is applicable to the case where a desktop virtual image is realized in terms of visual effects and a full human-computer interaction is performed with a user through the desktop virtual image. The method can be performed by a device for human-computer interaction with a desktop virtual image. The device can be implemented in the form of hardware and / or software and can generally be configured in a target client for realizing a desktop virtual image function. The target client can be installed in various electronic devices, such as a desktop computer, a laptop computer, or a tablet computer. Figure 1 As shown, the method includes:

[0034] S110 , in response to a triggering request of a target client for implementing a desktop virtual image function, displaying a client interface with a fully transparent background of the target client in a set screen area of ​​a computer desktop.

[0035] In this embodiment, the developer pre-develops a target client for realizing the desktop virtual image function. When the computer user needs to use the desktop virtual image function, the target client can be installed in the computer's operating system and triggered to start when needed. The computer can be pre-installed with any type of operating system, such as Windows system or masOS system.

[0036] Accordingly, when the computer user chooses to open the target client through a set interaction method, such as double-clicking the installation icon, a trigger request to the target client will be generated accordingly. After receiving the trigger request, the target client will open the corresponding client interface.

[0037] The desktop virtual image function can be understood as the target client independently displaying a specific virtual image, such as a virtual person, a virtual pet, or a virtual object (floating ball) in a specific screen area of ​​the computer desktop after being triggered and started. The computer user can meet various interaction needs by interacting with the virtual image through voice, text, or mouse actions. In addition, since the desktop virtual image only appears in a small area of ​​the computer desktop, the computer user can use other functions provided by the computer desktop normally while displaying the desktop virtual image.

[0038] In various embodiments of the present invention, in order to visually realize the desktop display effect of the virtual image, rather than actually integrating the virtual image function into the underlying system of the computer desktop, it is necessary to perform special configuration on the display effect of the client interface.

[0039] Specifically, considering that the desktop virtual image generally appears in the lower left corner or lower right corner of the computer desktop, the target client can be pre-set to be fixed in the lower left corner or lower right corner of the computer desktop with a fixed size after being triggered to display the client interface with a fully transparent background.

[0040] In this embodiment, the client interface with a fully transparent background can be understood as the entire client interface being transparent, or as a "fully transparent client interface". That is, although the client interface is displayed on the computer desktop, the computer user cannot observe the client interface at all through the "human eye".

[0041] When implementing the above fully transparent client interface, you may consider setting the window control buttons and window control bar in the interface to be hidden when constructing the client interface, and then setting the display property of the interface background of the client interface to be fully transparent.

[0042] Specifically, when writing the display effect code of the client interface, the display property of the interface background can be set to be fully transparent. At this time, although the client interface serves as the functional carrier of the target client, from a visual effect point of view, the user cannot observe the existence of the client interface at all.

[0043] S120: In the client interface with a fully transparent background, a target Web page with a fully transparent background is loaded.

[0044] Furthermore, after the target client opens the client interface, various client functions can be loaded in the client interface. In this embodiment, what needs to be loaded is a target web page with a fully transparent background. Similarly, when writing the display effect code of the target web page, the display property of the page background can be set to be fully transparent. From the visual effect, the user cannot observe the existence of the target web page at all.

[0045] It is understandable that when developing web pages based on Web technology, HTML5 drawing elements (typically, Canvas components) can be used, and mainstream JavaScript programming languages ​​can be used to render various graphics in Web pages. In this embodiment, the inventors can convert a desktop virtual image that requires complex programming into an animation effect that only needs to be implemented in the target Web page by cleverly performing two full-transparent display processes, and the difficulty of programming implementation is greatly reduced.

[0046] S130 . Dynamically render and display the desktop virtual image in the fully transparent background target Web page, wherein the sizes of the client interface and the target Web page are adapted to the size of the desktop virtual image.

[0047] As shown above, since the embodiments of the present invention realize the full transparent display of the client interface and the target Web page, it is only necessary to realize the dynamic rendering display of the desktop virtual image in the target Web page.

[0048] In an optional implementation of this embodiment, an animation effect can be designed in which, after a target client is started, a desktop virtual image moves from far to near, from small to large, to the lower left corner of the computer desktop. At this time, the target display area can be first determined on the computer desktop according to the moving trajectory of the desktop virtual image, and then the client interface with a fully transparent background of the target client is displayed in the target display area, and after the target with a fully transparent background is loaded and displayed in the target display area of ​​the Web page, the above-mentioned animation display effect of the desktop virtual image can be realized by using the target on the Web page as a display carrier. After completing the above-mentioned animation display effect, the size of the client interface and the target Web page can be dynamically reduced based on the actual display size of the desktop virtual image residing on the computer desktop, so as to avoid the part of the client interface and the target Web page that exceeds the size of the desktop virtual image from blocking other software or function controls on the computer desktop.

[0049] It is understandable that the process in which the above-mentioned virtual image enters the computer desktop and resides on the computer desktop for display, if implemented by programming with the help of programming tools adapted to the operating system, requires very professional developers based on a full understanding of the operating system to program it. The technical solution of the present invention only requires web page programming and drawing to achieve the above-mentioned animation effect, and the programming difficulty is greatly reduced.

[0050] To further facilitate understanding, Figure 2 FIG. 2 shows a schematic diagram of a bearing relationship between a target client, a target Web page, and a desktop virtual image applicable to an embodiment of the present invention. Figure 2 As shown, after the target client is turned on, the client interface is first presented, in which the target Web page can be loaded and displayed, and then, under the support of the target Web page, the desktop virtual image can be rendered and displayed with various animation effects. Figure 2 In the example, only for the purpose of display distinction, a clear gap is created between the client interface, the target web page, and the desktop virtual image. In fact, the gap between the three can be infinitely reduced within a feasible range to avoid the full transparent area from blocking other desktop elements to the greatest extent.

[0051] S140: Performing human-computer interaction with the user through the desktop virtual image.

[0052] In this embodiment, the target client can be set to always be displayed at the front end of the computer desktop and always be in an active state. Furthermore, after opening a new software on the computer desktop, the desktop virtual image can be displayed on the display interface of the new software. Accordingly, the target client that is always in an active state can capture the user's real-time input voice, text or mouse action, and use the desktop virtual image to perform real-time and efficient human-computer interaction with the user.

[0053] The technical solution of the embodiment of the present invention, by responding to a trigger request of a target client for realizing a desktop virtual image function, displays a client interface with a fully transparent background of the target client in a set screen area of ​​the computer desktop; loads a target Web page with a fully transparent background in the client interface with a fully transparent background; and dynamically renders and displays the desktop virtual image in the target Web page with a fully transparent background; and realizes a human-computer interaction with a user through the desktop virtual image, without the need for in-depth cooperation with the computer operating system code, and only uses the drawing rendering function of the target Web page integrated in the target client to realize the desktop interaction effect of the virtual image, effectively reducing the development difficulty of developers and greatly improving the development efficiency of software for realizing the desktop virtual image function. Since the image rendering function is only realized on the Web page side, there is no need to adapt to the operating system, which greatly enhances the portability of the software. In addition, software development based on Web pages belongs to mainstream development, with more open source resources for reference and richer functions that can be realized, which can further reduce development costs and development time.

[0054] Figure 3 A flowchart of another method for human-computer interaction with a desktop virtual image provided by an embodiment of the present invention is provided. This embodiment is optimized based on the above embodiments. In this embodiment, when the interaction mode selected by the user is interactive voice or screen interaction coordinates input by a mouse, the operation of human-computer interaction with the user through the desktop virtual image is concretized.

[0055] Correspondingly, such as Figure 3 As shown, the method includes:

[0056] S310: In response to a trigger request to a target client for implementing a desktop virtual image function, a client interface with a fully transparent background of the target client is displayed in a set screen area of ​​a computer desktop.

[0057] S320: In the client interface with a fully transparent background, load a target Web page with a fully transparent background.

[0058] S330: Dynamically render and display the desktop virtual image in the fully transparent background target Web page, wherein the size of the client interface and the target Web page is adapted to the size of the desktop virtual image.

[0059] S340, acquiring the user's interactive voice in real time, converting the interactive voice into interactive text, and sending it to the backend server, and executing S370.

[0060] In this embodiment, the target client may request authorization to use the computer microphone in advance. If the user completes the above authorization, the target client can monitor the user voice input by the computer user in real time when it is in an active state, or, in order to further ensure the user's information security, it can trigger the start of monitoring the user voice input by the user after the user clicks a set position in the desktop virtual image or presses a set shortcut key.

[0061] Specifically, during the process of real-time monitoring of user voice, if the user voice is detected to contain set keywords, the currently monitored user voice will be used as the user's interactive voice. Alternatively, after the user actively triggers a click or presses a set shortcut key, the user voice can be monitored and the monitored user voice can be directly used as the user's interactive voice.

[0062] After the target client confirms that it has obtained the user's interactive voice, it can convert the interactive voice into interactive text through speech-to-text technology, send it to the back-end server for intent recognition, and receive the interactive response result fed back by the back-end server.

[0063] The back-end server is used to obtain a response result and an interaction method for the response result according to the interaction text, and feed back the response result and the interaction method to the target client.

[0064] In this embodiment, an AI big model can be integrated in the back-end server, and the AI ​​big model performs model processing based on the interactive text sent by the target client to obtain corresponding response results and matching interaction methods.

[0065] Among them, the interactive method may include voice interaction, narration display or search result display, etc. Specifically, if the response result is a simple dialogue or chat content, the interactive method may be to control the desktop virtual image to directly broadcast the response result, if the response result is a relatively rich or large amount of text content, the interactive method may be to display the above response result through a text box, or if it is determined that the response result is to display the set search results, the interactive method may be to open a new browser page and display hyperlinks of various search results in the browser page, and the user can open the corresponding service webpage by clicking the above hyperlinks.

[0066] S350, receiving in real time the screen interaction coordinates input by the user through the mouse on the computer desktop, taking a screenshot of the computer desktop, and executing S360.

[0067] Furthermore, when the user uses the mouse to perform mouse operations on the computer desktop, the active target client can obtain in real time the screen position of the mouse single-click, double-click, or long-press, or obtain in real time the screen area selected by the user by clicking and dragging the mouse.

[0068] Accordingly, the screen interaction coordinates input by the user through the mouse on the computer desktop can be obtained, wherein the screen interaction coordinates can be one screen coordinate point or multiple screen coordinate points, for example, four screen coordinate corner points selected by the user clicking and dragging the mouse.

[0069] After obtaining the above screen interaction coordinates, the target client can obtain the current real-time display content of the computer desktop by means of a screenshot. Of course, it is understandable that when the target client is installing the software, it also needs to obtain the user's authorization to take a screenshot in advance. Only after obtaining the user's authorization can the target client obtain the screenshot of the computer desktop.

[0070] S360. According to the screen snapshot, locate the screen area image that matches the screen interaction coordinates, and send the screen area image to the backend server, and execute S370.

[0071] After obtaining the screenshot, a region coordinate range can be first determined based on the above screen interaction coordinates, and then the matching screen region image can be located based on the region coordinate range and the screenshot. In a specific example, if the screen interaction coordinates are only one screen coordinate point, the screen coordinate point can be used as the midpoint and extended to the left, left and right, or up, down, left and right to obtain the above region coordinate range. Alternatively, if the screen interaction coordinates are four screen coordinate corner points, the region coordinate range can be directly determined based on the four screen coordinate corner points.

[0072] After obtaining the screen area image, image recognition can be performed locally and the recognition result can be sent to the back-end server. Alternatively, in order to further improve the speed and efficiency of human-computer interaction, the above screen area image can be directly sent to the back-end server for image recognition.

[0073] On the basis of the above embodiments, after the screen area image is obtained, a privacy detection can also be triggered, for example, to detect whether the image content corresponding to the screen area image involves user privacy, for example, whether the content captured by the screen area image belongs to the set privacy software, or whether the text content contained in the screen area image may involve privacy content, etc. After determining that the above screen area image may involve privacy content, before sending the screen area image to the back-end server, a user privacy prompt can be firstly performed, and after obtaining the user's confirmation and authorization, the above screen area image can be sent to the back-end server.

[0074] The backend server is used to obtain user demand information matching the screen area image, obtain response results matching the user demand information and an interaction method for the response results, and feed back the response results and the interaction method to the target client.

[0075] Similarly, the back-end server can use a powerful AI model to identify text content, images or graphic content in the screen area image, and obtain user demand information based on the recognition results, such as information query, term explanation, Chinese-English translation or web page retrieval, etc., and then, it can obtain response results that match the user demand information and the interaction method for the response results.

[0076] S370: Perform human-computer interaction on the response result through the desktop virtual image according to the interaction method.

[0077] In an optional implementation of this embodiment, performing human-computer interaction on the response result according to the interaction mode through the desktop virtual image may include:

[0078] If the interaction mode is voice interaction, generate a response voice matching the response result; construct a virtual image broadcast animation through the desktop virtual image and the response voice, and dynamically render and display the virtual image broadcast animation;

[0079] If the interaction mode is narration display, after the sizes of the client interface and the target web page are expanded, a narration dialog box is displayed in the expanded area, and the response result is displayed in the narration dialog box, wherein the narration dialog box is set close to the desktop virtual image;

[0080] If the interaction mode is search result display, after the sizes of the client interface and the target Web page are expanded, an internal browser page is called up in the expanded area, and the search results matching the response result are displayed through the internal browser page.

[0081] Specifically, if the interaction mode is determined to be voice interaction, the desktop virtual image can be made to provide the answering voice to the user in the form of voice broadcast. For example, if the desktop virtual image is a virtual person, the virtual person can be controlled to broadcast the answering voice while constantly changing the mouth shape to achieve the above visual effect. In specific implementation, an adapted virtual image broadcasting animation can be simply constructed through Web development technology, and the virtual image broadcasting animation can be dynamically rendered and displayed in the target Web page.

[0082] Furthermore, if the interaction mode is voice-over display, a voice-over dialog box can be provided next to the desktop virtual image, for example, next to the mouth position, and the above-mentioned response result (a large paragraph of text or graphic information) is presented in the voice-over dialog box according to the set presentation effect. Since the above-mentioned display effect requires the expansion of the display area of ​​the target Web page in the client interface, it is necessary to first estimate the size of the required voice-over dialog box according to the data volume of the response result (for example, the number of binary symbols), and then, after the size of the client interface and the target Web page is expanded according to the size, the voice-over dialog box is displayed in the expanded area.

[0083] After the above narration display is completed, in response to the user's cancel display operation, the size of the client interface and the target Web page can be re-adjusted to match the size of the desktop virtual image to avoid blocking other desktop elements.

[0084] Furthermore, if the interaction method is to display search results, the size of the client interface and the target web page is also expanded, and the internal browser page of the target client is called in the expanded area, and the search results matching the response results are displayed through the internal browser page. Of course, in addition to calling the internal browser page of the target client, the native browser in the computer operating system can also be called, and the native browser displays the search results required by the user.

[0085] The technical solution of the embodiment of the present invention only uses the drawing rendering function of the target Web page integrated in the target client to achieve the desktop interactive effect of the virtual image, which effectively reduces the development difficulty of developers and greatly improves the development efficiency of software for realizing desktop virtual image functions. Since the image rendering function is only implemented on the Web page side, there is no need to adapt to the operating system, which greatly enhances the portability of the software. In addition, by using AI models (for example, various large voice models) as the provider of interactive response results, various needs of users in the process of using the computer can be fully met. Through a virtual image accompanying the desktop, the user's needs are effectively met in the form of animation, which improves the user's satisfaction in the process of using the computer and can improve the user's computer usage efficiency.

[0086] Figure 4 A flowchart of another method for human-computer interaction with a desktop virtual image provided by an embodiment of the present invention is provided. This embodiment is optimized based on the above embodiments. In this embodiment, when the interaction method selected by the user is a mouse gesture or interactive text input on the computer desktop through an input method, the operation of human-computer interaction with the user through the desktop virtual image is concretized.

[0087] Correspondingly, such as Figure 4 As shown, the method includes:

[0088] S410: In response to a trigger request to a target client for implementing a desktop virtual image function, a client interface with a fully transparent background of the target client is displayed in a set screen area of ​​a computer desktop.

[0089] S420: In the client interface with a fully transparent background, a target Web page with a fully transparent background is loaded.

[0090] S430: Dynamically render and display the desktop virtual image in the fully transparent background target Web page, wherein the size of the client interface and the target Web page is adapted to the size of the desktop virtual image.

[0091] S440, receiving in real time the mouse gestures input by the user on the computer desktop through the mouse, and sending the mouse gestures to the back-end server.

[0092] When the user uses the mouse to perform screen operations, the user can perform predefined operations by holding down the right mouse button and drawing a specific track on the screen, which is similar to the gestures used by people when communicating. In this embodiment, the mapping relationship between different mouse gestures and different functions of the desktop virtual image can be predefined. For example, when the user draws a nearly circular shape with the mouse, the desktop virtual image broadcasts the weather forecast, or when the user draws a wavy line with the mouse, the desktop virtual image is controlled to be in a set state (for example, resting or working state, etc.).

[0093] Optionally, the above mapping relationship can be configured on the back-end server, and at this time, when the target client detects the mouse gesture input by the user, the mouse gesture can be directly sent to the back-end server. Alternatively, the mapping relationship can be directly built into the target client, and the target client directly responds to the user's intention, which is not limited in this embodiment.

[0094] The backend server is used to obtain the user interaction intention according to the interaction gesture, and feed back the user interaction intention to the target client.

[0095] S450: Generate an interactive animation that matches the desktop virtual image according to the user interaction intention, and dynamically render and display the interactive animation.

[0096] S460: receiving in real time the interactive text input by the user on the computer desktop through the input method, and sending the interactive text to the back-end server.

[0097] The backend server is used to identify at least one knowledge entity according to the interactive text, and feed back the description information of each knowledge entity to the target client.

[0098] In this embodiment, the user may perform user behaviors such as work, study, or entertainment accompanied by the desktop virtual image. At this time, the user may need to use the input method to perform corresponding user input. At this time, if the user has authorized the target client to access the input method in advance, the target client can obtain various text inputs entered by the user through the input method in real time. At this time, the target client can perform full human-computer interaction with the user based on the above text input using the desktop virtual image as a medium.

[0099] For example, when a user enters a specific proper noun, such as a person's name or a place name, in the chat interface through the input method, the corresponding knowledge data, such as character introductions, place introductions, or surrounding points of interest media, can be obtained through the AI ​​model built into the backend server. Alternatively, when a user enters a specific proper noun in the office software setting through the input method, the corresponding noun explanation, paper search, or related news information can be obtained through the AI ​​model built into the backend server.

[0100] S470: Displaying the description information of each knowledge entity to the user in the form of voice broadcast or narration display through the desktop virtual image.

[0101] The technical solution of the embodiment of the present invention can fully meet various needs of users in the process of using computers by using AI models (for example, various large voice models) as the provider of interactive response results. Through a virtual image accompanying the desktop, the user's needs are effectively met in the form of animation, which improves the user's satisfaction in the process of using the computer and can improve the user's computer usage efficiency.

[0102] Figure 5 The present invention provides a schematic diagram of a device for human-computer interaction with a desktop virtual image. Figure 5 As shown, the device includes: a first transparent display module 510, a second transparent display module 520, a desktop virtual image display module 530 and a human-computer interaction module 540, wherein:

[0103] The first transparent display module 510 is used to display a fully transparent background client interface of the target client in a set screen area of ​​the computer desktop in response to a trigger request of the target client for realizing the desktop virtual image function.

[0104] The second transparent display module 520 is used to load a target Web page with a fully transparent background in a client interface with a fully transparent background.

[0105] The desktop virtual image display module 530 is used to dynamically render and display the desktop virtual image in a fully transparent background target Web page, wherein the size of the client interface and the target Web page is adapted to the size of the desktop virtual image.

[0106] The human-computer interaction module 540 is used to perform human-computer interaction with the user through the desktop virtual image.

[0107] The technical solution of the embodiment of the present invention, by responding to a trigger request of a target client for realizing a desktop virtual image function, displays a client interface with a fully transparent background of the target client in a set screen area of ​​the computer desktop; loads a target Web page with a fully transparent background in the client interface with a fully transparent background; and dynamically renders and displays the desktop virtual image in the target Web page with a fully transparent background; and realizes a human-computer interaction with a user through the desktop virtual image, without the need for in-depth cooperation with the computer operating system code, and only uses the drawing rendering function of the target Web page integrated in the target client to realize the desktop interaction effect of the virtual image, effectively reducing the development difficulty of developers and greatly improving the development efficiency of software for realizing the desktop virtual image function. Since the image rendering function is only realized on the Web page side, there is no need to adapt to the operating system, which greatly enhances the portability of the software. In addition, software development based on Web pages belongs to mainstream development, with more open source resources for reference and richer functions that can be realized, which can further reduce development costs and development time.

[0108] Based on the above embodiments, the human-computer interaction module 540 can be specifically used for:

[0109] Acquire the user's interactive voice in real time, convert the interactive voice into interactive text, and send it to the back-end server;

[0110] The backend server is used to obtain a response result and an interaction method for the response result according to the interaction text, and feed back the response result and the interaction method to the target client;

[0111] The response result is subjected to human-computer interaction through the desktop virtual image according to the interaction method.

[0112] Based on the above embodiments, the human-computer interaction module 540 can be specifically used for:

[0113] Receiving in real time the screen interaction coordinates input by the user through the mouse on the computer desktop, and taking a screenshot of the computer desktop;

[0114] According to the screen snapshot, locate the screen area image matching the screen interaction coordinates, and send the screen area image to the backend server;

[0115] The backend server is used to obtain user demand information matching the screen area image, obtain a response result matching the user demand information and an interaction method for the response result, and feed back the response result and the interaction method to the target client;

[0116] The response result is subjected to human-computer interaction through the desktop virtual image according to the interaction method.

[0117] Based on the above embodiments, the human-computer interaction module 540 can be further specifically used for:

[0118] If the interaction mode is voice interaction, generate a response voice matching the response result; construct a virtual image broadcast animation through the desktop virtual image and the response voice, and dynamically render and display the virtual image broadcast animation;

[0119] If the interaction mode is narration display, after the sizes of the client interface and the target web page are expanded, a narration dialog box is displayed in the expanded area, and the response result is displayed in the narration dialog box, wherein the narration dialog box is set close to the desktop virtual image;

[0120] If the interaction mode is search result display, after the sizes of the client interface and the target Web page are expanded, an internal browser page is called up in the expanded area, and the search results matching the response result are displayed through the internal browser page.

[0121] Based on the above embodiments, the human-computer interaction module 540 can be specifically used for:

[0122] Receive in real time the mouse gestures input by the user on the computer desktop through the mouse, and send the mouse gestures to the back-end server;

[0123] The backend server is used to obtain the user interaction intention according to the interaction gesture, and feed back the user interaction intention to the target client;

[0124] According to the user interaction intention, an interaction animation matching the desktop virtual image is generated, and the interaction animation is dynamically rendered and displayed.

[0125] Based on the above embodiments, the human-computer interaction module 540 can be specifically used for:

[0126] Receive interactive text input by the user on the computer desktop through an input method in real time, and send the interactive text to a back-end server;

[0127] The backend server is used to identify at least one knowledge entity according to the interactive text, and feed back the description information of each knowledge entity to the target client;

[0128] The description information of each knowledge entity is displayed to the user through the desktop virtual image in the form of voice broadcast or narration display.

[0129] The device for human-computer interaction with a desktop virtual image provided by an embodiment of the present invention can execute the method for human-computer interaction with a desktop virtual image provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0130] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0131] Figure 6 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0132] like Figure 6As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0133] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0134] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as executing the method for human-computer interaction with a desktop virtual image as described in any embodiment of the present invention.

[0135] That is, a method for human-computer interaction with a desktop virtual image, comprising:

[0136] In response to a trigger request to a target client for implementing a desktop virtual image function, displaying a client interface with a fully transparent background of the target client in a set screen area of ​​a computer desktop;

[0137] In the client interface with a fully transparent background, load the target web page with a fully transparent background;

[0138] In a fully transparent background target Web page, the desktop virtual image is dynamically rendered and displayed, wherein the size of the client interface and the target Web page is adapted to the size of the desktop virtual image;

[0139] Human-computer interaction is performed with the user through the desktop virtual image.

[0140] In some embodiments, the method for human-computer interaction with a desktop virtual image as described in any embodiment of the present invention may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for human-computer interaction with a desktop virtual image as described in any embodiment of the present invention described above may be executed. Alternatively, in other embodiments, the processor 11 may be configured to execute the method for human-computer interaction with a desktop virtual image as described in any embodiment of the present invention by any other appropriate means (e.g., by means of firmware).

[0141] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0142] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0143] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0144] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0145] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0146] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0147] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0148] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for human-computer interaction with a desktop virtual image, characterized in that: include: In response to a trigger request to a target client for implementing a desktop virtual image function, displaying a client interface with a fully transparent background of the target client in a set screen area of ​​a computer desktop; In the client interface with a fully transparent background, load the target web page with a fully transparent background; In a fully transparent background target Web page, the desktop virtual image is dynamically rendered and displayed, wherein the size of the client interface and the target Web page is adapted to the size of the desktop virtual image; Human-computer interaction is performed with the user through the desktop virtual image.

2. The method according to claim 1, characterized in that The human-computer interaction with the user is performed through the desktop virtual image, including: Acquire the user's interactive voice in real time, convert the interactive voice into interactive text, and send it to the back-end server; The backend server is used to obtain a response result and an interaction method for the response result according to the interaction text, and feed back the response result and the interaction method to the target client; The response result is subjected to human-computer interaction through the desktop virtual image according to the interaction method.

3. The method according to claim 2, characterized in that The human-computer interaction with the user is performed through the desktop virtual image, including: Receiving in real time the screen interaction coordinates input by the user through the mouse on the computer desktop, and taking a screenshot of the computer desktop; According to the screen snapshot, locate the screen area image matching the screen interaction coordinates, and send the screen area image to the backend server; The backend server is used to obtain user demand information matching the screen area image, obtain a response result matching the user demand information and an interaction method for the response result, and feed back the response result and the interaction method to the target client; The response result is subjected to human-computer interaction through the desktop virtual image according to the interaction method.

4. The method according to claim 2 or 3, characterized in that: The step of performing human-computer interaction on the response result according to the interaction mode through the desktop virtual image includes: If the interaction mode is voice interaction, generate a response voice matching the response result; construct a virtual image broadcast animation through the desktop virtual image and the response voice, and dynamically render and display the virtual image broadcast animation; If the interaction mode is narration display, after the sizes of the client interface and the target web page are expanded, a narration dialog box is displayed in the expanded area, and the response result is displayed in the narration dialog box, wherein the narration dialog box is set close to the desktop virtual image; If the interaction mode is search result display, after the sizes of the client interface and the target Web page are expanded, an internal browser page is called up in the expanded area, and the search results matching the response result are displayed through the internal browser page.

5. The method according to claim 1, characterized in that The human-computer interaction with the user is performed through the desktop virtual image, including: Receive in real time the mouse gestures input by the user on the computer desktop through the mouse, and send the mouse gestures to the back-end server; The backend server is used to obtain the user interaction intention according to the interaction gesture, and feed back the user interaction intention to the target client; According to the user interaction intention, an interaction animation matching the desktop virtual image is generated, and the interaction animation is dynamically rendered and displayed.

6. The method according to claim 1, characterized in that The human-computer interaction with the user is performed through the desktop virtual image, including: Receive interactive text input by the user on the computer desktop through an input method in real time, and send the interactive text to a back-end server; The backend server is used to identify at least one knowledge entity according to the interactive text, and feed back the description information of each knowledge entity to the target client; The description information of each knowledge entity is displayed to the user through the desktop virtual image in the form of voice broadcast or narration display.

7. A device for human-computer interaction with a desktop virtual image, characterized in that: include: A first transparent display module, for displaying a fully transparent background client interface of the target client in a set screen area of ​​the computer desktop in response to a trigger request to the target client for realizing the desktop virtual image function; The second transparent display module is used to load a target Web page with a fully transparent background in a client interface with a fully transparent background; A desktop virtual image display module is used to dynamically render and display the desktop virtual image in a fully transparent background target Web page, wherein the size of the client interface and the target Web page is adapted to the size of the desktop virtual image; The human-computer interaction module is used to perform human-computer interaction with the user through the desktop virtual image.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for human-computer interaction with a desktop virtual image according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for human-computer interaction with a desktop virtual image according to any one of claims 1 to 6 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements the method for human-computer interaction with a desktop virtual image according to any one of claims 1 to 6.

Images