Method and device for processing image by using digital assistant, equipment and storage medium

Abstract

The embodiment of the invention provides a method and device for processing an image by using a digital assistant, equipment and a storage medium. In the method, an image intercepted from a target page is received, and the image is sent to a digital assistant. Based on the content of the image, a processing intent for processing the image is determined. A processing result associated with the image is acquired by the digital assistant based on the processing intention. In this way, the intercepted image can be processed by means of a digital assistant, and the processing result is presented in the dialogue page. Therefore, the image processing efficiency is improved, the information acquisition difficulty of the user is reduced, and the information acquisition efficiency of the user is improved.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of image processing, and in particular, to methods, devices, apparatuses, and computer-readable storage media for processing images using a digital assistant. Background Art

[0002] Machine learning technology has been widely used in a variety of technical fields. In the prior art, digital assistants are provided to assist users in various task processing needs in different applications and scenarios. Digital assistants usually have intelligent dialogue and task processing capabilities. Digital assistants can be called in response to user input. For example, in the process of interacting with a digital assistant, the user enters an interactive message, and the digital assistant provides a reply message in response to the user input. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for processing an image using a digital assistant is provided. The method comprises: receiving an image captured from a target page, and sending the image to a digital assistant. Based on the content of the image, determining a processing intent for processing the image. Obtaining a processing result associated with the image, the processing result being determined by the digital assistant based on the processing intent.

[0004] In a second aspect of the present disclosure, a device for processing an image using a digital assistant is provided. The device includes: a receiving module configured to receive an image captured from a target page and send the image to a digital assistant; a determining module configured to determine a processing intent for processing the image based on the content of the image; and an acquiring module configured to acquire a processing result associated with the image, the processing result being determined by the digital assistant based on the processing intent.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device performs the method of the first aspect of the present disclosure.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect of the present disclosure.

[0007] In a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method of the first aspect of the present disclosure when executed by a processor.

[0008] It should be understood that the contents described in this content section are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0011] FIG. 2A to FIG. 2C A schematic diagram showing an example page according to some embodiments of the present disclosure;

[0012] Figure 3 A flowchart showing a process of processing an image using a digital assistant according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A block diagram showing an apparatus for processing an image using a digital assistant according to some embodiments of the present disclosure; and

[0014] Figure 5 A block diagram of an electronic device is shown in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0015] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0016] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0017] Herein, unless explicitly stated, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0018] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0019] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0020] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information, so that the user can independently choose whether to provide personal information to software or hardware such as electronic devices, applications, servers or storage media that execute operations of the technical solution of the present disclosure based on the prompt message.

[0021] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0022] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0023] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multi-layer processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0024] A "neural network" is a machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs, and typically includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications typically include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence so that the output of the previous layer is provided as input to the next layer, where the input layer receives the input of the neural network and the output of the output layer serves as the final output of the neural network. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes input from the previous layer.

[0025] Generally, machine learning can be roughly divided into three stages, namely the training stage, the testing stage, and the application stage (also called the inference stage). In the training stage, a given model can be trained using a large amount of training data, and the parameter values ​​are continuously updated iteratively until the model can obtain consistent inferences that meet the expected goals from the training data. Through training, the model can be considered to be able to learn the association between input and output (also called input-to-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. The testing stage can sometimes be integrated into the training stage. In the application or inference stage, the trained model can be used to process the actual model input based on the parameter values ​​obtained through training to determine the corresponding model output.

[0026] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In the example environment 100, a digital assistant 120 may be installed in a terminal device 110 (it may also be referred to as a digital assistant 120 being deployed in the terminal device 110). A user 140 may interact with the digital assistant 120 via the terminal device 110 and / or an attached device of the terminal device 110.

[0027] In some embodiments, the digital assistant 120 may be included in or implemented as a target application, which is an application installed in the terminal device 110. The digital assistant 120 may be configured to have intelligent dialogue capabilities and task processing capabilities within the target application. The digital assistant 120 is provided to assist users in various task processing requirements in different applications and scenarios. In the process of interaction with the digital assistant 120, the user inputs an interactive message, and the digital assistant 120 provides a reply message in response to the user input. The interactive message may include a message in a multimodal form, such as a text message (e.g., a natural language text), a voice message, an image message, a video message, and the like. Typically, the digital assistant 120 can support users to input questions in a natural language, and perform tasks and provide replies based on the understanding and logical reasoning ability of the natural language input. In some embodiments, the target application can present a corresponding page 150 to the user 140 based on the operation of the user 140 to output and / or receive information from the user 140.

[0028] In some embodiments, the implementation of at least part of the functions of the target application and / or the implementation of at least part of the functions of the digital assistant 120 in the target application can be implemented based on the target model. In some embodiments, the target model can be a machine learning model, a deep learning model, a learning model, a neural network, etc. In some embodiments, the model can be based on a language model (LM). The language model can have question-answering capabilities by learning from a large amount of corpus. The target model can also be based on other appropriate models.

[0029] In some embodiments, the terminal device 110 communicates with the server 130 to provide services for the target application and / or the digital assistant 120. The terminal device 110 can be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the terminal device 110 can also support any type of interface for the user (such as a "wearable" circuit, etc.). The server 130 can be various types of computing systems / servers that can provide computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, and the like.

[0030] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0031] In the fast-paced digital age, users are surrounded by all kinds of information and media content. The consumption and processing of massive information has become a major challenge in users' daily lives. For example, people often share interesting content or find relevant information about an item. For example, people need to quickly extract key information from documents containing complex data, share it with the team instantly, and conduct in-depth analysis at work. For example, students need to organize notes from online resources, and so on. Users expect to be able to provide a tool that can not only help them process information, but also adapt to their changing needs and personalized habits.

[0032] In view of this, according to an embodiment of the present disclosure, an improved solution for processing images using a digital assistant is provided. According to the solution, an image captured from a target page is received and sent to a digital assistant. Based on the content of the image, a processing intent for processing the image is determined. A processing result associated with the image is obtained, and the processing result is determined by the digital assistant based on the processing intent. Thus, the captured image can be processed based on the processing intent with the help of a digital assistant, and the processing result can be presented in a conversation page. This helps to improve the efficiency of processing images, reduce the difficulty for users to obtain information, and improve the efficiency of users in obtaining information.

[0033] Some example embodiments of the present disclosure will be described in detail below with reference to the examples of the accompanying drawings. In some embodiments, the method for processing an image using a digital assistant provided by the present disclosure can be implemented as a plug-in of a digital assistant, and the method is called in response to activation of the plug-in. For example, the digital assistant can provide a switch indicating whether to turn on the method. The method for processing an image using a digital assistant can be executed only when the switch is in an enabled state.

[0034] FIG. 2A to FIG. 2C Schematic diagrams of example pages 200A to 200C (also referred to as examples 200A to 200C) according to some embodiments of the present disclosure are shown. It should be understood that the pages shown in the drawings are merely examples, and various page designs may actually exist. The various graphical elements in the pages may have different arrangements and different visual representations, one or more of which may be omitted or replaced, and one or more other elements may also exist. The embodiments of the present disclosure are not limited in this respect.

[0035] The pages shown in Examples 200A to 200C may be presented at the terminal device 110. For ease of discussion, reference will be made to Figure 1Examples 200A to 200C are described with reference to the environment 100. It should be noted that, in this document, some operations described with reference to the terminal device 110 may require the assistance of the server 130. In addition, the operations performed by the terminal device 110 may be performed by the relevant applications installed on the terminal device 110 and / or the digital assistant 120.

[0036] In an embodiment of the present disclosure, the terminal device 110 receives an image captured from a target page. The target page may be any appropriate page from which content can be captured, including but not limited to a document page, a web page, a video page, a media item browsing page, a message page, etc. Figure 2A and Figure 2B As shown, in example 200A, the terminal device 110 may receive an image 201 captured from a target page 210, and in example 200B, the terminal device 110 may receive an image 202 captured from the target page 210. Here, the image 201 may be an image captured by calling a screenshot function of an operating system and / or an image captured by calling a screenshot function of other applications.

[0037] In an embodiment of the present disclosure, the terminal device 110 sends an image to the digital assistant 120 so that the digital assistant 120 processes the image (this process may also be referred to as calling the digital assistant 120 to process the image). In some embodiments, the terminal device 110 may provide a call control (e.g., Figure 2B The calling control may be provided in response to acquiring an image (i.e., the image captured from the target page as described above), or it may be presented by default, for example, displayed in a floating manner on an upper layer of the target page. For example, the terminal device 110 may send an image to the digital assistant 120 in response to receiving a user operation on the calling control (e.g., a trigger operation on the calling control by the user), so that the digital assistant 120 processes the image.

[0038] In some embodiments, the terminal device 110 may also provide a control page for invoking the digital assistant 120 to process the image, and in response to receiving the first interaction request in the control page, invoking the digital assistant 120 to process the image. The control page may be, for example, a floating page displayed on a layer above the target page. Figure 2A and Figure 2BAs shown, the terminal device 110 may provide a control page 220, which is a floating page displayed on the upper layer of the target page 210. The control page 220 may include, for example, a plurality of interactive controls, including at least one interactive control having a function of calling a digital assistant (e.g., interactive controls 222, 223, and 224 in example 200A, and interactive controls 225 and 226 in example 200B). The terminal device 110 may call the digital assistant 120 to process the image in response to receiving a first interactive request (e.g., a user operation on a target operation control) for a target interactive control (any interactive control having a function of calling a digital assistant).

[0039] In some embodiments, a processing intent for processing an image may be determined based on the content of the image. Assuming that the image includes text content, the processing intent may be determined as: automatic recognition, automatic translation, automatic search, etc. Specifically, a refined processing intent may be determined based on text recognition. Assuming that the image includes a name and contact information, etc., the processing intent may be determined as adding a contact. Assuming that the image includes information such as time and location, the processing intent may be determined as creating a schedule, etc. At this point, the digital assistant 120 may be called to determine the processing result.

[0040] For another example, an object in an image may be identified, and then the corresponding processing intent may be determined. Assuming that the image includes an item, the processing intent may be determined to be: querying detailed information, purchasing, etc. Then, the digital assistant 120 may be called to provide the corresponding query results, or a purchase link, etc.

[0041] In some embodiments, a dialog page may be provided, which is a dialog-style interactive page of the digital assistant, and the dialog page includes an image, a processing intention, and a processing result. In an embodiment of the present disclosure, the terminal device 110 may provide a dialog page in response to the digital assistant 120 being called, which is a dialog-style interactive page of the digital assistant, and the dialog page includes an image. Figure 2C As shown, the terminal device 110 can present a dialogue page 230 when calling the digital assistant 120 to process an image. The dialogue page 230 is a dialogue page between a user (e.g., user 140) and the digital assistant 120. The dialogue page 230 can also be a floating page displayed on the upper layer of the target page 210.

[0042] The conversation page 230 may include an image 233, a processing intention (e.g., a conversation message 234 indicates that the processing intention is to find a related web page), and a processing result (e.g., a conversation message 235 indicates a response from the digital assistant). In some embodiments, the user may copy the image to the conversation page; alternatively and / or additionally, the image 233 may be automatically presented in the conversation page, in this way, the user operation may be further simplified and the efficiency of obtaining information may be improved.

[0043] In some embodiments, for a certain image, multiple processing intentions can be determined from multiple aspects. At this time, the processing intention includes multiple processing intentions, and in the process of obtaining the processing result associated with the image, the dialogue page includes multiple prompts corresponding to the multiple processing intentions respectively. At this time, the user can select the desired processing intention from the multiple prompts, thereby specifying the task of the digital assistant. Further, in response to receiving a selection request for a target prompt among the multiple prompts, the processing result is obtained based on the target processing intention corresponding to the target prompt among the multiple processing intentions.

[0044] Continuing with the above example, assuming that the image includes text content, multiple processing intentions can be determined: automatic recognition, automatic translation, automatic search, and so on. Multiple prompts corresponding to the above-mentioned multiple processing intentions can be provided respectively, and then the image can be processed based on the prompts selected by the user. Assuming that the user selects "automatic recognition", the processing result can represent the result of text recognition. Assuming that the user selects "automatic translation", the language involved in the user's historical interactions with the digital assistant can be used as the target language. Alternatively and / or additionally, the user can be further required to specify the target language and provide corresponding translation results. Assuming that the user selects "automatic search", the results corresponding to the text content can be searched within the specified range. At this time, the result can involve multiple media formats, such as text, images, audio, video, and so on.

[0045] In some embodiments, obtaining the processing results associated with the image may further include: obtaining context data of the image from the target page, and then obtaining the processing results associated with the image based on the context data. Assuming that the user has captured an accompanying figure from a paper, the text content associated with the accompanying figure in the paper can be used as the context data of the image. Further, the context data can be used to analyze the specific meaning represented by the accompanying figure. Assuming that the accompanying figure includes multiple curves of multiple parameters, the digital assistant can use the specific definition of each parameter in the context data to explain the meaning of each curve, the correlation between multiple parameters, and so on. For another example, assuming that the user has captured a photo from a news webpage, the digital assistant can use the text report in the news webpage as context data and explain the identity of each person in the photo, and so on.

[0046] In some embodiments, the terminal device 110 may respond to receiving a move request for moving any one of the control page and the dialog page and move to the position of the page specified by the move request. The terminal device 110 may, for example, respond to receiving a drag operation for the control page and / or the dialog page and determine that a move request for the control page and / or the dialog page has been received. The terminal device 110 may, for example, determine the position where the drag operation ends as the position specified by the move request. Alternatively and / or additionally, the dialog page 230 may also be displayed at a fixed position.

[0047] In some embodiments, the terminal device 110 may no longer present (i.e., hide) the control page 220 in response to determining that the dialog page 230 is presented, or in response to determining that the dialog page 230 is to be presented. In some embodiments, the terminal device 110 may maintain the presentation of the control page 220 while presenting the dialog page 230 on the upper layer of the target page 210 (i.e., the control page 220 and the dialog page 230 may be presented together). The dialog page 230 includes a cancel control 232. The terminal device 110 may cancel the presentation of the dialog page 230 in response to receiving a user operation for canceling the control 232. An image 233 is presented in the dialog page 230, and the image 233 may be, for example, an image previously captured by the terminal device 110 from the target page 210. The image 233 may be presented in the dialog page 230 in the form of a conversation message from the user.

[0048] In some embodiments, the control page includes a first interactive control for performing a first predetermined task. Return to 2A and Figure 2B In example 200A, control page 220 may include interactive control 222 for performing detailed description task and interactive control 223 for performing web page search task. In example 200B, control page 220 may include interactive control 225 for performing viewing entry task and interactive control 226 for performing search related content task.

[0049] The terminal device 110 may provide a first result in a conversation page in response to receiving a second interaction request for the first interactive control, the first result being the result of the digital assistant 120 performing a first predetermined task on the image. The first predetermined task may be a pre-specified task. Exemplarily, the terminal device 110 may perform a detailed description task in response to receiving a second interaction request for the interactive control 222. The conversation page 230 shown in example 200C may provide a first result obtained by the digital assistant 120 performing a task of providing a detailed description of the content in the image, for example, presenting text and / or other information extracted from the image, and so on.

[0050] In some embodiments, the terminal device 110 may receive a third interaction request for adding a second interactive control to the control page. In some embodiments, the terminal device 110 may determine that a third interaction request for adding a second interactive control to the control page is received in response to receiving a user operation for adding a control. Exemplarily, the control page may include a first add control (e.g., Figure 2A and Figure 2B In the example of adding a control 221 in the control page 220 shown in FIG. 1 , the terminal device 110 may determine that a third interaction request for adding a second interaction control to the control page is received in response to receiving a user operation for adding the first interaction control. In this way, the user may add personalized controls according to their own needs.

[0051] For example, the dialog page may include a settings control (e.g. Figure 2C In the example of the setting control 231 in the dialog page 230 in the text format, the terminal device 110 may, for example, present the setting page in response to receiving a user operation on the setting control. The setting page may, for example, include a second add control. Alternatively or additionally, the dialog page may also directly include the second add control. The terminal device 110 may, in response to receiving a user operation on the second add control, determine that a third interaction request for adding a second interaction control to the control page is received.

[0052] In some embodiments, the terminal device 110 may determine that a third interactive request for adding a second interactive control to the control page is received in response to receiving a user operation for adding a first control. The terminal device 110 may also present a setting page for specifying a second predetermined task in response to receiving a user operation for adding a first control. The terminal device 110 may receive user input via the page and determine the second predetermined task based on the user input. For example, the predetermined task may include, but is not limited to, extracting text from an image, identifying an object in an image, and the like. In this way, it is convenient for users to add controls to the control page, thereby performing predetermined tasks in a simpler manner.

[0053] In some embodiments, the third interaction request may also be a text input received in the dialog page, and the terminal device 110 may determine the second predetermined task based on the text input. For example, the terminal device 110 may determine that the second predetermined task is the XXXX task in response to receiving text such as "add a control corresponding to the XXXX task in the control page" in the dialog page.

[0054] The terminal device 110 may then determine the second predetermined task corresponding to the second interactive control in response to receiving the third interactive request for adding the second interactive control to the control page, and add the second interactive control for invoking the digital assistant to perform the second predetermined task to the control page. Exemplarily, if the terminal device 110 determines that the second predetermined task is the task of recognizing a formula in an image, the terminal device 110 may add the second interactive control for invoking the digital assistant to perform the task of recognizing a formula in an image to the control page.

[0055] Therefore, allowing users to define the controls in the control page for calling the digital assistant can improve the efficiency of the digital assistant in performing corresponding tasks.

[0056] In some embodiments, at least one interactive control in the control page for invoking a digital assistant may include an interactive control that is only used to invoke a digital assistant (e.g. Figure 2A The terminal device 110 may present a dialog page in response to the interactive control being triggered. Further, the terminal device 110 may receive user input via the dialog page. The terminal device 110 may determine the received user input as a fourth interactive request. In response to receiving the fourth interactive request for the dialog page, the terminal device 110 may provide a second result in the dialog page, the second result being a result of the digital assistant performing the task specified by the fourth interactive request for the image.

[0057] like Figure 2C As shown, the terminal device 110 can receive user input via the input box 236 and determine the user input as a fourth interaction request. The terminal device 110 can present the received user input in the form of a conversation message from the user (e.g., the conversation message 234 shown in the figure). The digital assistant 120 can perform the task specified by the fourth interaction request. The terminal device 110 can present the result obtained by the digital assistant 120 performing the task specified by the fourth interaction request, for example, in the form of a conversation message from the digital assistant 120 (i.e., the conversation message 235 shown in the figure) in the dialogue page 230.

[0058] In some embodiments, the terminal device 110 may also provide a control in the conversation page for performing subsequent tasks for the result in the conversation page (i.e., the result of the image being processed by the digital assistant). The control for performing subsequent tasks may include, for example, any appropriate control such as an edit control, a send control, a favorite control, etc. The terminal device 110 may perform the subsequent task in response to receiving the fifth interaction request for the control. For example, the terminal device 110 may add the result to the user's favorites in response to receiving a user operation for the favorite control.

[0059] It should be noted that, in some embodiments, when it is determined that the image is an image frame from a video clip, the terminal device 110 may also provide data associated with the video clip. Figure 2B If the target page 210 is a movie playback page and the image 202 is an image including a character in the movie, the terminal device 110 can instruct the digital assistant 120 to determine which movie the image 202 is from and which character the character is in the movie, and provide movie introductions, character profiles, movie reviews and other information associated with the movie in the conversation page.

[0060] In the educational assistance scenario, when a user is reading an electronic textbook or viewing a screenshot of a teaching video, the terminal device 110 can display relevant historical figures, scientific theory explanations, or solutions to mathematical problems on the dialogue page based on the content in the screenshot. This can improve the efficiency of the user in acquiring knowledge.

[0061] In a shopping recommendation scenario, when a user browses product pictures or related advertising screenshots, the terminal device 110 can provide price comparison information of the product in the conversation page based on the content in the screenshot, and recommend personalized coupons, related product combinations or alternatives based on the user's shopping history and preferences, thereby improving shopping convenience and efficiency.

[0062] In the health consultation scenario, when the user is viewing a screenshot of content related to health or diet, the terminal device 110 can provide nutritional analysis, health advice or recipe recommendations in the conversation page based on the content in the screenshot. For screenshots related to exercise, the corresponding exercise method can be recommended in the conversation page or a sports venue recommendation close to the user's location can be provided.

[0063] In a travel planning scenario, when a user sees photos or screenshots of a travel destination, the terminal device 110 can provide an introduction to the location in the image in the conversation page based on the content in the screenshot, and provide a detailed travel guide, real-time weather, local cultural background knowledge, travel agency recommendations, budget planning, etc.

[0064] In a multi-language learning scenario, the terminal device 110 can provide instant translation in the dialogue page based on the content in the screenshot. At the same time, it can also provide the pronunciation, grammar analysis, usage scenarios and practice dialogues of the language to help users improve language learning efficiency.

[0065] In the home decoration scenario, when a user uploads pictures of home decoration or screenshots of design drawings, the terminal device 110 can provide decoration suggestions in the dialogue page based on the content in the screenshot, such as color matching, furniture selection and space utilization plan, and can even recommend local decoration service businesses and related discount information.

[0066] In the film and television content expansion scenario, when a user watches a screenshot of a movie or TV series, the terminal device 110 can provide character analysis, plot discussion, behind-the-scenes footage, and in-depth information on related literary works or historical events in the conversation page based on the content in the screenshot, as well as recommendations for similar film and television works, which can enhance the efficiency of information acquisition after watching the film or TV series.

[0067] In summary, according to the embodiments of the present disclosure, the screenshots can be automatically sent to the digital assistant, and the digital assistant can then process the captured images and present the processing results in the conversation page. This helps to improve the efficiency of image processing, reduce the difficulty for users to obtain information, and improve the efficiency of users obtaining information.

[0068] The above has described the specific details of each step of processing an image using a digital assistant, and provided a method for processing an image using a digital assistant. Figure 3 FIG. 3 is a flowchart of a process 300 of processing an image using a digital assistant according to some embodiments of the present disclosure. The process 300 may be implemented at the terminal device 110. Figure 1 Process 300 is described.

[0069] In block 310, an image captured from a target page is received and sent to a digital assistant.

[0070] At block 320 , a processing intent is determined for processing the image based on the content of the image.

[0071] In box 330, a processing result associated with the image is obtained, and the processing result is determined by the digital assistant based on the processing intent.

[0072] In some embodiments, process 300 further includes: providing a dialog page, which is a conversational interaction page of the digital assistant, and the dialog page includes an image, a processing intention, and a processing result.

[0073] In some embodiments, the processing intent includes multiple processing intentions, and the dialog page includes multiple prompts corresponding to the multiple processing intentions respectively, and obtaining the processing result associated with the image further includes: in response to receiving a selection request for a target prompt among the multiple prompts, obtaining the processing result based on a target processing intent corresponding to the target prompt among the multiple processing intentions.

[0074] In some embodiments, obtaining the processing result associated with the image further includes: obtaining context data of the image from the target page; and obtaining the processing result associated with the image based on the context data.

[0075] In some embodiments, obtaining processing results associated with the image further includes: providing a control page for calling a digital assistant to process the image; and in response to receiving a first interaction request in the control page, calling the digital assistant to process the image.

[0076] In some embodiments, the control page includes a first interactive control for performing a first predetermined task, wherein providing a dialogue page includes: in response to receiving a second interactive request for the first interactive control, providing a first result in the dialogue page, the first result being a result of the digital assistant performing the first predetermined task on the image.

[0077] In some embodiments, process 300 further includes: in response to receiving a third interaction request for adding a second interactive control to the control page, determining a second predetermined task corresponding to the second interactive control; and adding a second interactive control to the control page, the second interactive control being used to call the digital assistant to perform the second predetermined task.

[0078] In some embodiments, the third interaction request is a text input received in the dialog page, and determining the second predetermined task includes: determining the second predetermined task based on the text input.

[0079] In some embodiments, process 300 further includes: in response to receiving a fourth interaction request for the conversation page, providing a second result in the conversation page, the second result being a result of the digital assistant performing a task specified by the fourth interaction request on the image.

[0080] In some embodiments, process 300 further includes: providing a control in the conversation page for performing a subsequent task for a result in the conversation page, wherein the result is a result of processing an image by the digital assistant; and performing the subsequent task in response to receiving a fifth interaction request for the control.

[0081] In some embodiments, process 300 further includes, in response to determining that the image is an image frame from the video segment, providing data associated with the video segment.

[0082] In some embodiments, the control page and the dialog page are floating pages displayed on top of the target page, and process 300 further includes: in response to receiving a move request for moving any one of the control page and the dialog page, moving to the position of the page specified by the move request.

[0083] In some embodiments, process 300 is implemented as a plug-in for a digital assistant, and process 300 is invoked in response to activation of the plug-in.

[0084] According to some embodiments of the present disclosure, a device for processing images using a digital assistant is also provided. Figure 4A block diagram of an apparatus 400 for processing an image using a digital assistant according to some embodiments of the present disclosure is shown. The apparatus 400 may be implemented as or included in a terminal device 110. Each module / component in the apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0085] like Figure 4 As shown, the device 400 includes: a receiving module 410, configured to receive an image captured from a target page and send the image to a digital assistant; a determining module 420, configured to determine a processing intent for processing the image based on the content of the image; and an acquiring module 430, configured to acquire a processing result associated with the image, and the processing result is determined by the digital assistant based on the processing intent.

[0086] In some embodiments, the device 400 further includes: a dialogue page providing module, configured to provide a dialogue page, the dialogue page is a conversational interactive page of the digital assistant, and the dialogue page includes an image, a processing intention, and a processing result.

[0087] In some embodiments, the processing intent includes multiple processing intentions, and the dialog page includes multiple prompts corresponding to the multiple processing intentions respectively, and the acquisition module further includes: a selection-based acquisition module, configured to obtain the processing result based on a target processing intention corresponding to the target prompt among the multiple processing intentions in response to receiving a selection request for a target prompt among the multiple prompts.

[0088] In some embodiments, the acquisition module further includes: a context acquisition module configured to acquire context data of the image from the target page; and a context-based acquisition module configured to acquire a processing result associated with the image based on the context data.

[0089] In some embodiments, the acquisition module further includes: a control page providing module, configured to provide a control page for calling a digital assistant to process an image; and an assistant calling module, configured to call a digital assistant to process an image in response to receiving a first interaction request in the control page.

[0090] In some embodiments, the control page includes a first interactive control for performing a first predetermined task, wherein the page providing module 430 includes: a first result providing module, configured to provide a first result in the dialogue page in response to receiving a second interactive request for the first interactive control, the first result being a result of the digital assistant performing the first predetermined task on the image.

[0091] In some embodiments, the device 400 further includes: a second predetermined task determination module, configured to determine a second predetermined task corresponding to the second interactive control in response to receiving a third interaction request for adding a second interactive control to the control page; and a control adding module, configured to add a second interactive control to the control page, the second interactive control being used to call the digital assistant to perform the second predetermined task.

[0092] In some embodiments, the third interaction request is a text input received in the dialog page, and the second predetermined task determination module is specifically configured to: determine the second predetermined task based on the text input.

[0093] In some embodiments, the device 400 further includes: a second result providing module, configured to provide a second result in the conversation page in response to receiving a fourth interaction request for the conversation page, the second result being a result of the digital assistant performing a task specified by the fourth interaction request on the image.

[0094] In some embodiments, the device 400 further includes: a control providing module, configured to provide a control in the conversation page for performing a subsequent task for a result in the conversation page, wherein the result is a result of the digital assistant processing an image; and a subsequent task execution module, configured to execute a subsequent task in response to receiving a fifth interaction request for the control.

[0095] In some embodiments, the apparatus 400 further comprises: a data providing module configured to provide data associated with the video segment in response to determining that the image is an image frame from the video segment.

[0096] In some embodiments, the control page and the dialog page are floating pages displayed on top of the target page, and the device 400 further includes: a page movement module, configured to move to the position of the page specified by the move request in response to receiving a move request for moving any one of the control page and the dialog page.

[0097] In some embodiments, device 400 is implemented as a plug-in of a digital assistant, and device 400 is invoked in response to activation of the plug-in.

[0098] The units and / or modules included in the device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units and / or modules in the device 400 can be implemented at least in part by one or more hardware logic components. As an example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0099] Figure 5 1 shows a block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 5 The electronic device 500 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to implement Figure 1 The terminal device 110, the server 130, and / or Figure 4 Device 400.

[0100] like Figure 5 As shown, the electronic device 500 is in the form of a general-purpose computing device. The components of the electronic device 500 may include, but are not limited to, one or more processors or processing units 510, a memory 520, a storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. The processing unit 510 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 520. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 500.

[0101] The electronic device 500 typically includes a plurality of computer storage media. Such media may be any available media accessible to the electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 520 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 530 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data and may be accessed within the electronic device 500.

[0102] The electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 5 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 520 may include a computer program product 525 having one or more program modules that are configured to perform various methods or actions of various implementations of the present disclosure.

[0103] The communication unit 540 implements communication with other computing devices through a communication medium. Additionally, the functions of the components of the electronic device 500 can be implemented in a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 500 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0104] The input device 550 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 560 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 500 may also communicate with one or more external devices (not shown) through the communication unit 540 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 500, or communicate with any device that allows the electronic device 500 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0105] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0106] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0107] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0108] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0109] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0110] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for processing an image using a digital assistant, comprising: receiving an image captured from a target page, and sending the image to the digital assistant; determining, based on content of the image, a processing intent for processing the image; as well as Obtain a processing result associated with the image, wherein the processing result is determined by the digital assistant based on the processing intent.

2. The method according to claim 1, further comprising: A dialogue page is provided, which is a conversational interaction page of the digital assistant, and the dialogue page includes the image, the processing intention, and the processing result.

3. The method according to claim 1, wherein the processing intention includes multiple processing intentions, and the dialog page includes multiple prompts corresponding to the multiple processing intentions, respectively, and obtaining the processing result associated with the image further includes: In response to receiving a selection request for a target prompt among the multiple prompts, the processing result is obtained based on a target processing intention among the multiple processing intentions corresponding to the target prompt.

4. The method according to claim 1, wherein obtaining the processing result associated with the image further comprises: Obtaining context data of the image from the target page; as well as Based on the context data, a processing result associated with the image is obtained.

5. The method according to claim 1, wherein obtaining a processing result associated with the image further comprises: providing a control page for invoking the digital assistant to process the image; as well as In response to receiving a first interaction request in the control page, calling the digital assistant to process the image.

6. The method according to claim 5, wherein the control page includes a first interactive control for performing a first predetermined task, wherein providing the dialog page includes: In response to receiving a second interaction request for the first interactive control, a first result is provided in the dialog page, wherein the first result is a result of the digital assistant performing the first predetermined task on the image.

7. The method according to claim 6, further comprising: In response to receiving a third interaction request for adding a second interactive control to the control page, determining a second predetermined task corresponding to the second interactive control; as well as A second interactive control is added to the control page, where the second interactive control is used to call the digital assistant to perform the second predetermined task.

8. The method of claim 7, wherein the third interaction request is a text input received in the conversation page, and determining the second predetermined task comprises: The second predetermined task is determined based on the text input.

9. The method according to claim 2, further comprising: In response to receiving a fourth interaction request for the conversation page, providing a second result in the conversation page, the second result being a result of the digital assistant performing a task specified by the fourth interaction request on the image.

10. The method according to claim 2, further comprising: Providing, in the conversation page, a control for executing a subsequent task with respect to a result in the conversation page, wherein the result is a result of the digital assistant processing the image; as well as In response to receiving a fifth interaction request for the control, performing the subsequent task.

11. The method according to claim 2, further comprising: In response to determining that the image is an image frame from a video segment, data associated with the video segment is provided.

12. The method according to claim 5, wherein the control page and the dialog page are floating pages displayed on top of the target page, and the method further comprises: In response to receiving a move request for moving any one of the control page and the dialog page, moving to a position of a page specified by the move request.

13. The method of claim 1, wherein the method is implemented as a plug-in of the digital assistant, and the method is invoked in response to activation of the plug-in.

14. A device for processing an image using a digital assistant, comprising: A receiving module, configured to receive an image captured from a target page and send the image to the digital assistant; a determination module configured to determine a processing intent for processing the image based on the content of the image; as well as An acquisition module is configured to acquire a processing result associated with the image, wherein the processing result is determined by the digital assistant based on the processing intent.

15. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 13 when executed by the at least one processing unit.

16. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 13.

17. A computer program product comprising a computer program, wherein the computer program implements the method according to any one of claims 1 to 13 when executed by a processor.

Images