Method and apparatus for processing image by using digital assistant, device, and storage medium

Abstract

Embodiments of the present invention provide a method and apparatus for processing an image by using a digital assistant, a device, and a storage medium. The method comprises: receiving an image captured from a target page, and sending the image to a digital assistant; on the basis of content of the image, determining a processing intent for processing the image; and on the basis of the processing intent, the digital assistant acquiring a processing result associated with the image. In this way, a captured image can be processed with the help of a digital assistant, and a processing result can be presented in a dialogue page. The present invention facilitates improvement of image processing efficiency, and reduction of the difficulty for users to acquire information, thereby improving the efficiency of the users acquiring information.

Description

Method, apparatus, device and storage medium for processing image by digital assistant TECHNICAL FIELD

[0001] Example embodiments of the present disclosure generally relate to the field of image processing, and in particular, to a method, apparatus, device and computer readable storage medium for processing image by digital assistant. BACKGROUND

[0002] Machine learning techniques have been widely used in various technical fields. In the prior art, a digital assistant is provided to assist users in various task processing needs in different applications and scenarios. The digital assistant usually has intelligent conversation and task processing capabilities. The digital assistant can be invoked in response to user input. For example, during the interaction with the digital assistant, the user inputs an interaction message, and the digital assistant provides a reply message in response to the user input.

[0003] SUMMARY

[0004] In a first aspect of the present disclosure, a method for processing an image by a digital assistant is provided. The method comprises: receiving an image captured from a target page, and sending the image to the digital assistant. Based on the content of the image, determining a processing intent for processing the image. Obtaining a processing result associated with the image, the processing result being determined by the digital assistant based on the processing intent.

[0005] In a second aspect of the present disclosure, an apparatus for processing an image by a digital assistant is provided. The apparatus comprises: a receiving module configured to receive an image captured from a target page, and send the image to the digital assistant; a determining module configured to determine a processing intent for processing the image based on the content of the image; and an obtaining module configured to obtain a processing result associated with the image, the processing result being determined by the digital assistant based on the processing intent.

[0006] In a third aspect of the present disclosure, an electronic device is provided. The device comprises at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. The instructions, when executed by the at least one processing unit, cause the device to perform the method of the first aspect of the present disclosure.

[0007] In a fourth aspect of the present disclosure, a computer readable storage medium is provided. The computer readable storage medium has stored thereon a computer program, the computer program being executable by a processor to implement the method of the first aspect of the present disclosure.

[0008] In a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the method of the first aspect of the present disclosure.

[0009] It is to be understood that the particulars shown herein are by way of example and for purposes of illustrative discussion of the embodiments of the present disclosure only and are not intended to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description, associated figures and appended claims. BRIEF DESCRIPTION OF DRAWINGS

[0010] The above-mentioned and other features and advantages of various embodiments of the present disclosure will become more apparent by reference to the following detailed description taken in conjunction with the accompanying drawings. In the drawings, like reference numerals denote like elements, wherein:

[0011] FIG. 1 illustrates a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0012] FIGS. 2A-2C illustrate schematic diagrams of example pages, in accordance with some embodiments of the present disclosure;

[0013] FIG. 3 illustrates a flow diagram of a process for processing an image with a digital assistant, in accordance with some embodiments of the present disclosure;

[0014] FIG. 4 illustrates a block diagram of an apparatus for processing an image with a digital assistant, in accordance with some embodiments of the present disclosure; and

[0015] FIG. 5 illustrates a block diagram of an electronic device in which one or more embodiments of the present disclosure can be implemented. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described hereinafter with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and fully convey the scope of the present disclosure to those skilled in the art. It is understood that the drawings and embodiments herein are for illustrative purposes only and are not intended to limit the scope of the present disclosure.

[0017] In the description of embodiments of the present disclosure, the term "including" and its derivatives, shall be understood as comprising but not limited to. The term "based on" shall be understood as "based, at least in part, on". The term "one embodiment" or "an embodiment" shall be understood as "at least one embodiment". The term "some embodiments" shall be understood as "at least some embodiments". Other explicit or implicit definitions can also be included below.

[0018] In this document, unless explicitly stated otherwise, performing a step "in response to" A does not mean that the step is performed immediately after A, but can include one or more intervening steps.

[0019] It can be understood that the data involved in the technical solutions of the present disclosure (including but not limited to the data itself, the obtaining or use of the data) should comply with the requirements of the corresponding laws and regulations and relevant provisions.

[0020] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the type of personal information involved in the present disclosure, the use range, the use scenario, etc. should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0021] For example, in response to receiving the active request of the user, prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will need to obtain and use the personal information of the user, so that the user can voluntarily choose whether to provide the personal information to the software or hardware such as electronic device, application program, server or storage medium, etc. that performs the operation of the technical solutions of the present disclosure according to the prompt information.

[0022] As an optional but non-limiting implementation manner, in response to receiving the active request of the user, the manner of sending prompt information to the user may, for example, be a pop-up window manner, and the prompt information may be presented in the form of text in the pop-up window. In addition, the pop-up window may also carry selection controls for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0023] It can be understood that the above notification and user authorization process is only illustrative and does not limit the implementation manner of the present disclosure, and other manners meeting the relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0024] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. Neural network models are an example of models based on deep learning. In this document, "model" can also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", which are used interchangeably herein.

[0025] A“neural network” is a machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs, which typically includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications typically include many hidden layers, increasing the depth of the network. The layers of a neural network are connected in sequence, such that the output of a previous layer is provided as input to a subsequent layer, with the input layer receiving the input to the neural network and the output of the output layer as the final output of the neural network. Each layer of the neural network includes one or more nodes (also referred to as processing nodes or neurons), each of which processes inputs from the previous layer.

[0026] Generally, machine learning can include three stages, namely a training stage, a testing stage, and an application stage (also referred to as an inference stage). In the training stage, a given model can be trained using a large amount of training data, iteratively updating parameter values until the model is able to obtain consistent inferences from the training data that satisfy an expected goal. Through training, the model can be considered to have learned the association (also referred to as the mapping) between the input and the output from the training data. The parameter values of the trained model are determined. In the testing stage, test inputs are applied to the trained model to test whether the model is able to provide correct outputs, thereby determining the performance of the model. The testing stage can sometimes be merged into the training stage. In the application or inference stage, the trained model can be used to process actual model inputs based on the parameter values obtained through training, to determine the corresponding model outputs.

[0027] FIG. 1 illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. In this example environment 100, a digital assistant 120 can be installed in a terminal device 110 (also referred to as the digital assistant 120 being deployed in the terminal device 110). A user 140 can interact with the digital assistant 120 via the terminal device 110 and / or an attached device of the terminal device 110.

[0028] In some embodiments, the digital assistant 120 can be included or implemented as a target application, which is an application already installed in the terminal device 110. The digital assistant 120 can be configured to have intelligent conversation capability and task processing capability within the target application. The digital assistant 120 is provided to assist the user with various task processing needs in different applications, scenarios. During the interaction with the digital assistant 120, the user inputs an interaction message, and the digital assistant 120 provides a reply message in response to the user input. The interaction message can include messages in multi-modal forms, such as text messages (e.g., natural language text), voice messages, image messages, video messages, etc. Generally, the digital assistant 120 is capable of supporting the user to input questions in a natural language manner, and perform tasks and provide replies based on the understanding of the natural language input and logical reasoning capability. In some embodiments, the target application can present a corresponding page 150 to the user 140 based on the operation of the user 140, to output and / or receive information from the user 140.

[0029] In some embodiments, the implementation of at least part of the functionality of the target application, and / or the implementation of at least part of the functionality of the digital assistant 120 in the target application can be implemented based on a target model. In some embodiments, the target model can be a machine learning model, a deep learning model, a learning model, a neural network, etc. In some embodiments, the model can be based on a language model (LM). The language model is capable of having question and answer capability by learning from a large amount of corpus. The target model can also be based on other appropriate models.

[0030] In some embodiments, the terminal device 110 communicates with the server 130 to implement the provisioning of the services of the target application and / or the digital assistant 120. The terminal device 110 can be any type of mobile terminal, fixed terminal, or portable terminal including a mobile handset, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media player, a multimedia player, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a game device, or any combination thereof, including accessories and peripherals of such devices, or any combination thereof. In some embodiments, the terminal device 110 can also support any type of interface to the user (such as "wearable" circuitry, etc.). The server 130 can be various types of computing systems / servers capable of providing computing capability, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0031] It should be understood that the structure and function of the various elements in the environment 100 are described for illustrative purposes only, without implying any limitation on the scope of the present disclosure.

[0032] In the fast-paced digital era, users are surrounded by various information and media content. Consuming and processing massive amounts of information has become a big challenge in users' daily life. For example, people often share interesting content or search for relevant information about an item. For another example, people need to quickly extract key information from documents including complex data, share with team in real time, and conduct in-depth analysis in work. For another example, students need to organize notes from online resources, and so on. Users expect a tool that can not only help them process information, but also adapt to their changing needs and personalized habits.

[0033] In view of this, according to an embodiment of the present disclosure, an improved scheme for processing an image by a digital assistant is provided. According to the scheme, an image captured from a target page is received, and the image is sent to a digital assistant. Based on the content of the image, a processing intent for processing the image is determined. A processing result associated with the image is obtained, which is determined by the digital assistant based on the processing intent. In this way, the captured image can be processed based on the processing intent by means of the digital assistant, and the processing result is presented in a conversation page. This helps to improve the efficiency of processing the image, reduce the difficulty of users obtaining information, and improve the efficiency of users obtaining information.

[0034] Some example embodiments of the present disclosure will be described in detail below with reference to examples of the accompanying drawings. In some embodiments, the method for processing an image by a digital assistant provided by the present disclosure can be implemented as a plug-in of the digital assistant, and the method is called in response to the activation of the plug-in. For example, the digital assistant can provide a switch indicating whether to start the method. The method for processing an image by a digital assistant can be executed only when the switch is in an enabled state.

[0035] FIGS. 2A-2C show schematic diagrams of example page 200A-200C (may also be referred to as example 200A-200C) according to some embodiments of the present disclosure. It should be understood that the pages shown in the drawings are only examples, and various page designs can actually exist. Various graphical elements in the page can have different arrangements and different visual representations, one or more elements of which can be omitted or replaced, and one or more other elements can also exist. Embodiments of the present disclosure are not limited in this respect.

[0036] The pages shown in examples 200A-200C can be presented at terminal device 110. For ease of discussion, examples 200A-200C will be described with reference to environment 100 of FIG. 1. Note that some operations described herein with reference to terminal device 110 can require assistance from server 130 to complete. Further, operations performed by terminal device 110 can be performed by a relevant application installed on terminal device 110 and / or digital assistant 120, specifically.

[0037] In embodiments of the present disclosure, terminal device 110 receives an image that is captured from a target page. The target page can be any suitable page from which content can be captured, including but not limited to a document page, a webpage, a video page, a media item browsing page, a message page, etc. As shown in FIGS. 2A and 2B, in example 200A, terminal device 110 can receive image 201 that is captured from target page 210, and in example 200B, terminal device 110 can receive image 202 that is captured from target page 210. Here, image 201 can be an image captured by invoking a screenshot function of an operating system, and / or an image captured by invoking a screenshot function of another application.

[0038] In embodiments of the present disclosure, terminal device 110 sends the image to digital assistant 120 for processing by digital assistant 120 (this process can also be referred to as invoking digital assistant 120 to process the image). In some embodiments, terminal device 110 can provide an invocation control (e.g., various controls in invocation page 220 shown in FIG. 2B). The invocation control can be provided in response to obtaining the image (i.e., the image captured from the target page described above), or can be presented by default, e.g., in a floating manner on top of the target page. Terminal device 110 can send the image to digital assistant 120 for processing by digital assistant 120, e.g., in response to receiving a user operation (e.g., a trigger operation by the user) on the invocation control.

[0039] In some embodiments, the terminal device 110 can further provide a control page for invoking the digital assistant 120 to process the image, and in response to receiving a first interaction request in the control page, invoke the digital assistant 120 to process the image. The control page may, for example, be a floating page displayed on top of the target page. As shown in FIG. 2A and FIG. 2B, the terminal device 110 can provide a control page 220, which is a floating page displayed on top of the target page 210. The control page 220 may, for example, include a plurality of interaction controls, including at least one interaction control having a function of invoking the digital assistant (e.g., the interaction control 222, the interaction control 223, and the interaction control 224 in example 200A, and the interaction control 225 and the interaction control 226 in example 200B). The terminal device 110 can invoke the digital assistant 120 to process the image in response to receiving a first interaction request (e.g., a user operation on a target operation control) on a target interaction control (any interaction control having a function of invoking the digital assistant).

[0040] In some embodiments, a processing intent for processing the image can be determined based on the content of the image. Assuming that the image includes text content, the processing intent can be determined to be automatic recognition, automatic translation, automatic search, and the like. Specifically, a refined processing intent can be determined based on text recognition, assuming that the image includes a name and a contact, the processing intent can be determined to be adding a contact, assuming that the image includes information such as time and place, the processing intent can be determined to be creating a schedule, and the like. At this time, the digital assistant 120 can be invoked to determine a processing result.

[0041] For another example, an object in the image can be recognized, and a corresponding processing intent can be determined. Assuming that the image includes a certain item, the processing intent can be determined to be querying detailed information, purchasing, and the like. In turn, the digital assistant 120 can be invoked to provide a corresponding query result or a purchase link, and the like.

[0042] In some embodiments, a conversation page can be provided, the conversation page being a conversational interaction page of the digital assistant, and the conversation page including the image, the processing intent, and the processing result. In embodiments of the present disclosure, the terminal device 110 can provide the conversation page in response to the digital assistant 120 being invoked, the conversation page being a conversational interaction page of the digital assistant, and the conversation page including the image. As shown in FIG. 2C, the terminal device 110 may, for example, present a conversation page 230 in the case of invoking the digital assistant 120 to process the image. The conversation page 230 is a conversation page between a user (e.g., the user 140) and the digital assistant 120. The conversation page 230 may, for example, also be a floating page displayed on top of the target page 210.

[0043] The dialogue page 230 can include an image 233, a processing intent (e.g., the conversation message 234 indicates that the processing intent is to find a relevant webpage), and a processing result (e.g., the conversation message 235 indicates a response from the digital assistant). In some embodiments, the image can be copied into the dialogue page by the user; alternatively and / or additionally, the image 233 can be automatically presented in the dialogue page in this way, which can further simplify user operations and improve the efficiency of obtaining information.

[0044] In some embodiments, for a certain image, multiple processing intents can be determined from multiple aspects. At this time, the processing intent includes multiple processing intents, and in the process of obtaining the processing result associated with the image, the dialogue page includes multiple prompts respectively corresponding to the multiple processing intents. At this time, the user can select the desired processing intent from the multiple prompts, thereby specifying the task of the digital assistant. Further, in response to receiving a selection request for a target prompt in the multiple prompts, the processing result is obtained based on a target processing intent in the multiple processing intents corresponding to the target prompt.

[0045] Continuing the example above, assuming that the image includes text content, multiple processing intents can be determined: automatic recognition, automatic translation, automatic search, etc. Multiple prompts corresponding to the above-mentioned multiple processing intents can be provided respectively, and then the image is processed based on the prompt selected by the user. Assuming that the user selects "automatic recognition", the processing result can represent the result of text recognition. Assuming that the user selects "automatic translation", the target language can be based on the language involved in the user's historical interaction with the digital assistant, alternatively and / or additionally, the user can be further required to specify the target language, and the corresponding translation result is provided. Assuming that the user selects "automatic search", the results corresponding to the text content can be searched within a specified range. At this time, the results can involve multiple media formats, such as text, image, audio, video, etc.

[0046] In some embodiments, obtaining the processing result associated with the image can further include obtaining context data of the image from the target page, and then obtaining the processing result associated with the image based on the context data. Assuming that the user intercepts an attached figure from a paper, the text content in the paper associated with the attached figure can be used as the context data of the image at this time. Further, the context data can be used to analyze the specific meaning of the attached figure. Assuming that the attached figure includes multiple curves of multiple parameters, the digital assistant can use the specific definitions of each parameter in the context data to explain the meaning of each curve, the relationship between multiple parameters, etc. For example, assuming that the user intercepts a photo from a news webpage, the digital assistant can use the text report in the news webpage as context data, and explain the identity of each person in the photo, etc.

[0047] In some embodiments, the terminal device 110 can move to a position of a page specified by a move request in response to receiving the move request for any of the move control page and the conversation page. The terminal device 110 may, for example, determine that a move request is received for the control page and / or the conversation page in response to receiving a drag operation for the control page and / or the conversation page. The terminal device 110 may, for example, determine a position where the drag operation ends as the position specified by the move request. Alternatively and / or additionally, the conversation page 230 can also be displayed at a fixed position.

[0048] In some embodiments, the terminal device 110 can cease presenting (i.e., hide) the control page 220 in response to determining that the conversation page 230 is presented, or in response to determining that the conversation page 230 is to be presented. In some embodiments, the terminal device 110 can maintain presentation of the control page 220 while presenting the conversation page 230 on top of the target page 210 (i.e., the control page 220 and the conversation page 230 can be presented together). The conversation page 230 includes a cancel control 232. The terminal device 110 can cease presenting the conversation page 230 in response to receiving a user operation for the cancel control 232. The conversation page 230 presents an image 233 therein, which may, for example, be an image previously captured by the terminal device 110 from the target page 210. The image 233 can be presented in the conversation page 230 in the form of a conversation message from a user.

[0049] In some embodiments, the control page includes a first interactive control for performing a first predetermined task. Returning to 2A and FIG. 2B, in the example 200A, the control page 220 can include the interactive control 222 for performing the elaboration task and the interactive control 223 for performing the web search task. In the example 200B, the control page 220 can include the interactive control 225 for performing the view term task and the interactive control 226 for performing the search related content task.

[0050] The terminal device 110 can provide a first result in the conversation page in response to receiving a second interaction request for the first interactive control, the first result being a result of performing the first predetermined task on the image by the digital assistant 120. The first predetermined task can be a task that is specified in advance. Illustratively, the terminal device 110 can perform the elaboration task in response to receiving a second interaction request for the interactive control 222. The conversation page 230 as shown in the example 200C can provide the first result of performing the elaboration task on the content in the image by the digital assistant 120, e.g., presenting text and / or other information extracted from the image, etc.

[0051] In some embodiments, the terminal device 110 can receive a third interaction request for adding a second interaction control to the control page. In some embodiments, the terminal device 110 can determine that the third interaction request for adding the second interaction control to the control page is received in response to receiving a user operation for the adding control. Illustratively, the control page can include a first adding control (e.g., the adding control 221 in the control page 220 shown in FIG. 2A and FIG. 2B), and the terminal device 110 can determine that the third interaction request for adding the second interaction control to the control page is received in response to receiving a user operation for the first adding control. In this way, the user can add personalized controls according to his / her own needs.

[0052] Illustratively, the conversation page can include a setting control (e.g., the setting control 231 in the conversation page 230 in FIG. 2C), and the terminal device 110 may, for example, present a setting page in response to receiving a user operation for the setting control. The setting page may, for example, include a second adding control. Alternatively or additionally, the conversation page can also directly include the second adding control. The terminal device 110 can determine that the third interaction request for adding the second interaction control to the control page is received in response to receiving a user operation for the second adding control.

[0053] In some embodiments, the terminal device 110 can determine that the third interaction request for adding the second interaction control to the control page is received in response to receiving a user operation for the first adding control. The terminal device 110 can also present a setting page for specifying the second predetermined task in response to receiving the user operation for the first adding control. The terminal device 110 can receive user input via the page and determine the second predetermined task based on the user input. For example, the predetermined task may, for example, include but is not limited to extracting text in an image, recognizing an object in an image, etc. In this way, it can be facilitated for the user to add a control to the control page to perform a predetermined task in a more simple manner.

[0054] In some embodiments, the third interaction request can also be a text input received in the conversation page, and the terminal device 110 can determine the second predetermined task based on the text input. For example, the terminal device 110 can determine that the second predetermined task is an XXXX task in response to receiving a text such as "add a control corresponding to the XXXX task to the control page" in the conversation page.

[0055] The terminal device 110 can further determine a second predetermined task corresponding to the second interaction control in response to receiving a third interaction request for adding the second interaction control into the control page, and add the second interaction control for invoking the digital assistant to perform the second predetermined task into the control page. Illustratively, if the terminal device 110 determines that the second predetermined task is the task of recognizing the formula in the image, the terminal device 110 can add the second interaction control for invoking the digital assistant to perform the task of recognizing the formula in the image into the control page.

[0056] In this way, the user can define the control for invoking the digital assistant in the control page by himself, and the efficiency of the digital assistant performing the corresponding task can be improved.

[0057] In some embodiments, the at least one interaction control for invoking the digital assistant in the control page can include an interaction control for invoking the digital assistant only (e.g., the interaction control 224 in FIG. 2A). The terminal device 110 can present a dialogue page in response to the interaction control being triggered. Further, the terminal device 110 can receive a user input via the dialogue page. The terminal device 110 can determine the received user input as a fourth interaction request. The terminal device 110 can provide a second result in the dialogue page in response to receiving the fourth interaction request for the dialogue page, the second result being a result of the digital assistant performing a task specified by the fourth interaction request on the image.

[0058] As shown in FIG. 2C, the terminal device 110 can receive a user input via the input box 236 and determine the user input as a fourth interaction request. The terminal device 110 can present the received user input in the form of a session message from the user (e.g., the session message 234 shown in the figure). The digital assistant 120 can perform the task specified by the fourth interaction request. The terminal device 110 can present the result obtained by the digital assistant 120 performing the task specified by the fourth interaction request, e.g., in the form of a session message from the digital assistant 120 (i.e., the session message 235 shown in the figure) in the dialogue page 230.

[0059] In some embodiments, the terminal device 110 can also provide a control for performing a subsequent task on the result in the dialogue page (i.e., the result of processing the image by the digital assistant) in the dialogue page. The control for performing the subsequent task can include, for example, an edit control, a send control, a favorite control, or any suitable control. The terminal device 110 can perform the subsequent task in response to receiving a fifth interaction request for the control. For example, the terminal device 110 can add the result to the user's favorites in response to receiving a user operation on the favorite control.

[0060] It should be noted that in some embodiments, in the case where the image is determined to be an image frame from a video clip, the terminal device 110 can also provide data associated with the video clip. Returning to FIG. 2B, if the target page 210 is a play page of a movie and the image 202 is an image including a certain character in the movie, the terminal device 110 can instruct the digital assistant 120 to determine which movie the image 202 is from and which character the character is in the movie, and provide information associated with the movie, such as a movie introduction, a character profile, a movie review, etc. in the conversation page.

[0061] In an education assistance scenario, when a user is reading an electronic textbook or viewing a screenshot of a teaching video, the terminal device 110 can display relevant historical figure profiles, scientific theory explanations, or solutions to mathematical problems for the user in the conversation page based on the content in the screenshot. This can improve the user's efficiency in acquiring knowledge.

[0062] In a shopping recommendation scenario, when a user is browsing product images or related advertisement screenshots, the terminal device 110 can provide price comparison information of the product in the conversation page based on the content in the screenshot, and recommend personalized coupons, related product combinations, or substitutes based on the user's shopping history and preferences, thereby improving shopping convenience and efficiency.

[0063] In a health consultation scenario, when a user is viewing a content screenshot related to health or diet, the terminal device 110 can provide nutritional component analysis, health advice, or recipe recommendations in the conversation page based on the content in the screenshot. For screenshots related to exercise, the terminal device 110 can recommend corresponding exercise methods or provide recommendations for exercise facilities near the user's location in the conversation page.

[0064] In a travel planning scenario, when a user sees a photo or screenshot of a tourist attraction, the terminal device 110 can provide an introduction of the location in the image and provide detailed travel guides, real-time weather, local cultural background knowledge, travel agency recommendations, and budget planning, etc. in the conversation page based on the content in the screenshot.

[0065] In a multi-language learning scenario, the terminal device 110 can provide instant translation in the conversation page based on the content in the screenshot. At the same time, it can also provide pronunciation, grammar analysis, usage scenarios, and practice conversations in the language, helping the user improve language learning efficiency.

[0066] In a home decoration scenario, when a user uploads a home decoration photo or design diagram screenshot, the terminal device 110 can provide decoration suggestions, such as color matching, furniture selection, and space utilization solutions, and even recommend local decoration service providers and related discount information in the conversation page based on the content in the screenshot.

[0067] In the scenario of content extension of a movie or a TV series, when a user views a screenshot of a movie or a TV series, the terminal device 110 can provide, in a dialogue page, role analysis, plot discussion, behind-the-scenes stories, and in-depth information about related literary works or historical events based on the content in the screenshot, and recommendations of similar movie or TV series, which can enhance the efficiency of information acquisition after movie or TV series watching.

[0068] In summary, according to the embodiments of the present disclosure, the screenshot can be automatically sent to the digital assistant, and then the digital assistant can process the intercepted image and present the processing result in the dialogue page. This helps to improve the efficiency of processing the image, reduce the difficulty of information acquisition of the user, and improve the efficiency of information acquisition of the user.

[0069] The specific details of each step of processing the image by the digital assistant have been described above, and a method of processing the image by the digital assistant is provided. FIG. 3 shows a flowchart of a process 300 of processing the image by the digital assistant according to some embodiments of the present disclosure. The process 300 can be implemented at the terminal device 110. The process 300 is described below with reference to FIG. 1.

[0070] At block 310, an image intercepted from a target page is received, and the image is sent to a digital assistant.

[0071] At block 320, a processing intent for processing the image is determined based on content of the image.

[0072] At block 330, a processing result associated with the image is obtained, and the processing result is determined by the digital assistant based on the processing intent.

[0073] In some embodiments, the process 300 further includes providing a dialogue page, the dialogue page being a dialogue interaction page of the digital assistant, and the dialogue page including the image, the processing intent, and the processing result.

[0074] In some embodiments, the processing intent includes a plurality of processing intents, and the dialogue page includes a plurality of prompts respectively corresponding to the plurality of processing intents, and obtaining the processing result associated with the image further includes: in response to receiving a selection request for a target prompt in the plurality of prompts, obtaining the processing result based on a target processing intent in the plurality of processing intents corresponding to the target prompt.

[0075] In some embodiments, obtaining the processing result associated with the image further includes: obtaining context data of the image from the target page; and obtaining the processing result associated with the image based on the context data.

[0076] In some embodiments, obtaining the processing result associated with the image further includes: providing a control page for invoking the digital assistant to process the image; and in response to receiving a first interaction request in the control page, invoking the digital assistant to process the image.

[0077] In some embodiments, the control page includes a first interaction control for performing a first predetermined task, and wherein providing the conversation page includes: in response to receiving a second interaction request for the first interaction control, providing a first result in the conversation page, the first result being a result of performing the first predetermined task on the image by the digital assistant.

[0078] In some embodiments, the process 300 further includes: in response to receiving a third interaction request for adding a second interaction control to the control page, determining a second predetermined task corresponding to the second interaction control; and adding the second interaction control to the control page, the second interaction control being for invoking the digital assistant to perform the second predetermined task.

[0079] In some embodiments, the third interaction request is a text input received in the conversation page, and determining the second predetermined task includes determining the second predetermined task based on the text input.

[0080] In some embodiments, the process 300 further includes: in response to receiving a fourth interaction request for the conversation page, providing a second result in the conversation page, the second result being a result of performing a task specified by the fourth interaction request on the image by the digital assistant.

[0081] In some embodiments, the process 300 further includes: providing a control in the conversation page for performing a subsequent task on a result in the conversation page, the result being a result of processing the image by the digital assistant; and in response to receiving a fifth interaction request for the control, performing the subsequent task.

[0082] In some embodiments, the process 300 further includes: in response to determining that the image is an image frame from a video clip, providing data associated with the video clip.

[0083] In some embodiments, the control page and the conversation page are floating pages displayed on top of a target page, and the process 300 further includes: in response to receiving a movement request for moving any of the control page and the conversation page, moving to a location of a page specified by the movement request.

[0084] In some embodiments, the process 300 is implemented as a plug-in of the digital assistant, and the process 300 is invoked in response to activation of the plug-in.

[0085] According to some embodiments of the present disclosure, an apparatus for processing an image with a digital assistant is also provided. FIG. 4 shows a block diagram of an apparatus 400 for processing an image with a digital assistant according to some embodiments of the present disclosure. The apparatus 400 can be implemented as or included in the terminal device 110. Various modules / components in the apparatus 400 can be implemented by hardware, software, firmware, or any combination thereof.

[0086] As shown in FIG. 4, the apparatus 400 includes a receiving module 410 configured to receive an image captured from a target page, and send the image to a digital assistant; a determining module 420 configured to determine a processing intent for processing the image based on content of the image; and an obtaining module 430 configured to obtain a processing result associated with the image, the processing result being determined by the digital assistant based on the processing intent.

[0087] In some embodiments, the apparatus 400 further includes a conversation page providing module configured to provide a conversation page, the conversation page being a conversational interaction page of the digital assistant, and the conversation page including the image, the processing intent, and the processing result.

[0088] In some embodiments, the processing intent includes a plurality of processing intents, and the conversation page includes a plurality of prompts respectively corresponding to the plurality of processing intents, and the obtaining module further includes a selected-based obtaining module configured to, in response to receiving a selection request for a target prompt in the plurality of prompts, obtain the processing result based on a target processing intent in the plurality of processing intents corresponding to the target prompt.

[0089] In some embodiments, the obtaining module further includes a context obtaining module configured to obtain context data of the image from the target page, and a context-based obtaining module configured to obtain the processing result associated with the image based on the context data.

[0090] In some embodiments, the obtaining module further includes a control page providing module configured to provide a control page for invoking the digital assistant to process the image, and an assistant invoking module configured to, in response to receiving a first interaction request in the control page, invoke the digital assistant to process the image.

[0091] In some embodiments, the control page includes a first interaction control for performing a first predetermined task, and the page providing module 430 includes a first result providing module configured to, in response to receiving a second interaction request for the first interaction control, provide a first result in the conversation page, the first result being a result of the digital assistant performing the first predetermined task on the image.

[0092] In some embodiments, the apparatus 400 further includes a second predetermined task determination module configured to, in response to receiving a third interaction request for adding a second interaction control into the control page, determine a second predetermined task corresponding to the second interaction control; and a control adding module configured to add the second interaction control into the control page, the second interaction control being used to invoke the digital assistant to perform the second predetermined task.

[0093] In some embodiments, the third interaction request is a text input received in the conversation page, and the second predetermined task determination module is specifically configured to determine the second predetermined task based on the text input.

[0094] In some embodiments, the apparatus 400 further includes a second result providing module configured to, in response to receiving a fourth interaction request for the conversation page, provide a second result in the conversation page, the second result being a result of the digital assistant performing a task specified by the fourth interaction request on the image.

[0095] In some embodiments, the apparatus 400 further includes a control providing module configured to provide, in the conversation page, a control for performing a subsequent task on a result in the conversation page, the result being a result of the digital assistant processing the image; and a subsequent task performing module configured to, in response to receiving a fifth interaction request for the control, perform the subsequent task.

[0096] In some embodiments, the apparatus 400 further includes a data providing module configured to, in response to determining that the image is an image frame from a video clip, provide data associated with the video clip.

[0097] In some embodiments, the control page and the conversation page are floating pages displayed on top of the target page, and the apparatus 400 further includes a page moving module configured to, in response to receiving a moving request for moving any one of the control page and the conversation page, move to a position of a page specified by the moving request.

[0098] In some embodiments, the apparatus 400 is implemented as a plug-in of the digital assistant, and the apparatus 400 is invoked in response to activation of the plug-in.

[0099] The units and / or modules included in the apparatus 400 can be implemented utilizing various means, including software, hardware, and / or firmware. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, e.g., machine-executable instructions stored on a storage medium. In addition to or alternatively, some or all of the units and / or modules in the apparatus 400 can be implemented at least partially by one or more hardware logic components. As an example and not by way of limitation, example types of hardware logic components that can be used include Field-Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application-Specific Standard Products (ASSPs), System-on-a-Chip (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0100] FIG. 5 shows a block diagram of an electronic device 500 in which one or more embodiments of the disclosure can be implemented. It should be understood that the electronic device 500 illustrated in FIG. 5 is merely an example and should not be construed as limiting on the functionality and scope of the embodiments described herein. The electronic device 500 illustrated in FIG. 5 can be used to implement the terminal device 110 of FIG. 1, the server 130, and / or the apparatus 400 of FIG. 4.

[0101] As shown in FIG. 5, the electronic device 500 is in the form of a general computing device. Components of the electronic device 500 can include, but are not limited to, one or more processors or processing units 510, a memory 520, a storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. The processing unit 510 can be a real or virtual processor and capable of executing various processing in accordance with programs stored in the memory 520. In a multi-processing system, multiple processing units execute computer-executable instructions in parallel to improve the processing power of the electronic device 500.

[0102] The electronic device 500 typically includes a plurality of computer storage media. Such media can be volatile and / or non-volatile storage media and removable and / or non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, and other data. The memory 520 can be volatile (such as register, cache, and / or random access memory (RAM)), non-volatile (such as read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), or flash memory), or some combination thereof. The storage device 530 can be a removable or non-removable media implemented in any method or technology for storage of information such as machine-readable instructions, data structures, program modules, and other data. Examples of storage media include, but are not limited to, flash drives, disks, and any other medium that can be used to store information and / or data and that can be accessed by the electronic device 500.

[0103] The electronic device 500 can further include additional detachable / non-detachable, volatile / non-volatile storage media. Although not shown in FIG. 5, a disk drive for reading from or writing to a detachable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a detachable, non-volatile optical disk (e.g., a CD-ROM) can be provided. In these cases, each drive can be connected to the bus (not shown) by one or more data media interfaces. The memory 520 can include a computer program product 525 having one or more program modules configured to carry out the various methods or actions of the various implementations of the present disclosure.

[0104] The communication unit 540 enables communication through communication media with other computing devices. Additionally, the functionality of the components of the electronic device 500 can be implemented in a single computing cluster or a plurality of computer machines that are capable of communicating through a communication connection. Thus, the electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0105] The input device 550 can be one or more input devices, such as a mouse, a keyboard, a trackball, etc. The output device 560 can be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 500 can also communicate with one or more external devices (not shown) such as a storage device, a display device, etc., one or more devices that enable a user to interact with the electronic device 500, or any devices (e.g., a network card, a modem, etc.) that enable the electronic device 500 to communicate with one or more other computing devices, as needed, through the communication unit 540. Such communication can be carried out via an input / output (I / O) interface (not shown).

[0106] According to an example implementation of the present disclosure, a computer readable storage medium is provided having computer executable instructions stored thereon, where the computer executable instructions are executed by a processor to implement the method described above. According to an example implementation of the present disclosure, a computer program product is also provided that is tangibly stored on a non-transitory computer readable medium and includes computer executable instructions, where the computer executable instructions are executed by a processor to implement the method described above.

[0107] Various aspects of the disclosure are now described with reference to the drawings. In general, the drawings described below are diagrammatic and schematic representations of actual or conceptual structures and processes, and are not limiting of the scope of the present disclosure. In the drawings, the size and relative positioning of components can be exaggerated for clarity and / or descriptive purposes. Also, the drawings represent examples of the various aspects of the disclosure and are not limiting of the scope of the present disclosure. It should be understood that the drawings are not necessarily drawn to scale.

[0108] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0109] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0110] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0111] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for processing an image with a digital assistant, comprising: receiving an image captured from a target page, the image being sent to the digital assistant; determining a processing intent for processing the image based on content of the image; and obtaining a processing result associated with the image, the processing result being determined by the digital assistant based on the processing intent.

2. The method of claim 1, further comprising: providing a conversation page, the conversation page being a conversational interaction page of the digital assistant, and the conversation page including the image, the processing intent, and the processing result. 3.The method of claim 1, wherein the processing intent includes a plurality of processing intents, and the conversation page includes a plurality of prompts respectively corresponding to the plurality of processing intents, obtaining a processing result associated with the image further includes: in response to receiving a selection request for a target prompt in the plurality of prompts, obtaining the processing result based on a target processing intent in the plurality of processing intents corresponding to the target prompt. 4.The method of claim 1, wherein obtaining a processing result associated with the image further includes: obtaining context data of the image from the target page; and and obtaining a processing result associated with the image based on the context data. 5.The method of claim 1, wherein obtaining a processing result associated with the image further includes: providing a control page for invoking the digital assistant to process the image; and and in response to receiving a first interaction request in the control page, invoking the digital assistant to process the image.

6. The method of claim 5, wherein the control page includes a first interactive control for performing a first predetermined task, and wherein providing the dialog page includes: in response to receiving a second interaction request for the first interaction control, providing a first result in the conversation page, the first result being a result of the digital assistant performing the first predetermined task for the image.

7. The method of claim 6, further comprising: in response to receiving a third interaction request for adding a second interaction control to the control page, determining a second predetermined task corresponding to the second interaction control; and and adding a second interaction control to the control page, the second interaction control being for invoking the digital assistant to perform the second predetermined task.

8. The method of claim 7, wherein the third interaction request is a text input received in the conversation page, and determining the second predetermined task comprises: determining the second predetermined task based on the text input. 9.The method of claim 2, further comprising: in response to receiving a fourth interaction request for the conversation page, providing a second result in the conversation page, the second result being a result of the digital assistant performing a task specified by the fourth interaction request for the image. 10.The method of claim 2, further comprising: providing a control in the conversation page for performing a subsequent task for a result in the conversation page, the result being a result of the digital assistant processing the image; and and in response to receiving a fifth interaction request for the control, performing the subsequent task.

11. The method of claim 2, further comprising: in response to determining that the image is an image frame from a video clip, providing data associated with the video clip.

12. The method of claim 5, wherein the control page and the dialog page are floating pages overlaid on the target page, and the method further comprises: In response to receiving a move request for moving any one of the control page and the dialog page, moving to a location of a page specified by the move request. 13.The method of claim 1, wherein the method is implemented as a plug-in of the digital assistant, and the method is invoked in response to activation of the plug-in. 14.An apparatus for processing an image with a digital assistant, comprising: a receiving module configured to receive an image captured from a target page, and send the image to the digital assistant; a determining module configured to determine, based on content of the image, a processing intent for processing the image; and an obtaining module configured to obtain a processing result associated with the image, the processing result being determined by the digital assistant based on the processing intent. 15.An electronic device, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1-13. 16.A computer-readable storage medium having stored thereon a computer program, the computer program being executable by a processor to implement the method according to any one of claims 1-13. 17.A computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1-13. ​ ​

Images