Image processing method, electronic device and computer readable storage medium

Abstract

The application relates to the electronic technical field and provides an image processing method, an electronic device and a computer readable storage medium. The method is applied to the electronic device, a first plug-in is arranged in the electronic device, the method comprises the following steps: acquiring a to-be-processed image and a sentence input by a user, the sentence containing an intention of the user for processing the to-be-processed image; acquiring a task list corresponding to the intention, the task list at least comprising a first task; calling the first plug-in to execute the first task on the to-be-processed image, obtaining a first processing result; and displaying the first processing result. The above method can automatically process the image based on the sentence of the user, and the operation is convenient and efficient.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to an image processing method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the rapid development of electronic devices, people are able to process images in various ways to generate images that meet their needs.

[0003] For example, when a user takes a photo using an electronic device, if the photo contains an unwanted object, such as a passerby accidentally captured during the shot, the photo will not meet the user's expectations. In this case, the user can use the electronic device to remove the inadvertently captured part of the passerby from the photo, resulting in a photo containing only the subject. When using traditional image editing software, users usually need to manually execute various functions of the software to complete different image processing operations. For example, if a user wants to generate their own ID photo from a group photo, they need to first perform a cutout operation to obtain their own image, then select the head from the group image, and adjust its size and proportions to generate the ID photo.

[0004] Traditional operating methods have a high barrier to entry for users, requiring them to have professional skills related to image processing, and are time-consuming and labor-intensive. Summary of the Invention

[0005] This application provides an image processing method, apparatus, chip, electronic device, computer-readable storage medium, and computer program product that can understand and edit user intentions without requiring users to have image editing skills, thereby lowering the barrier to entry and improving user experience.

[0006] In a first aspect, an image processing method is provided, applied to an electronic device, wherein a first plug-in is disposed in the electronic device. The method includes: acquiring an image to be processed and a statement input by a user, wherein the statement contains the user's intention to process the image to be processed; acquiring a task list corresponding to the intention, wherein the task list includes at least a first task; calling the first plug-in to execute the first task on the image to be processed, thereby obtaining a first processing result; and displaying the first processing result.

[0007] The image processing method provided in this application can automatically understand the user's intent based on the user's input, then obtain and execute the corresponding task, thereby automatically completing the processing operation required by the user. This eliminates the need for the user to manually and repeatedly edit the image, making it more convenient and faster to use. Furthermore, it eliminates the need for users to possess image editing skills to understand and edit the image, lowering the barrier to entry and improving the user experience.

[0008] In some implementations, an assistant application and a cloud service are deployed in the electronic device to acquire an image to be processed and a statement input by the user, including: receiving the statement to acquire the image to be processed and the statement input by the user through the assistant application; acquiring a task list corresponding to the intent, including: sending the statement to a cloud-side server through the cloud service, the cloud-side server converting the statement into an intent and generating a task list corresponding to the intent; and receiving the task list.

[0009] By using a cloud-based server to perform intent recognition and task decomposition on statements, a task list consisting of individually executable tasks is obtained. Different tasks can be assigned to different plugins to invoke different tools to execute the corresponding tasks. This method can call corresponding plugins to achieve different image processing functions. When more processing functions are needed, the functionality can be expanded by adding plugins and corresponding interfaces, making it flexible and applicable to a wide range of scenarios.

[0010] In some implementations, calling the first plugin to perform a first task on the image to be processed and obtain a first processing result includes: sending a first loading request to the first plugin through the first plugin interface, the first loading request being used to instruct the first plugin to load; if the first plugin loads successfully, obtaining the identifier of the first plugin through the first plugin interface; and sending a first task to the first plugin according to the identifier of the first plugin, the first plugin being used to call the first tool corresponding to the first task to perform the first task and obtain the first processing result.

[0011] When a plugin loads successfully, the task can be sent to the corresponding plugin by returning the plugin's identifier, thereby ensuring that the correct tool is invoked and the task is successfully executed.

[0012] In some implementations, displaying the first processing result includes: sending the first processing result to an assistant application; and displaying the first processing result through a dialog box interface of the assistant application.

[0013] Users can see the processing results intuitively in the assistant application's dialog box interface, which improves interactivity and enhances the user experience.

[0014] In some implementations, the method further includes: sending a first response request to a cloud-side server via a cloud service, the first response request carrying information indicating that the first task was successfully executed, and the first response request being used to generate a first response message; receiving the first response message via the cloud service; sending the first response message to an assistant application; and displaying the first response message through a dialog box interface of the assistant application.

[0015] When the first task is successfully completed, the electronic device can also request the cloud server to generate the corresponding first response message through the cloud service. When the first response message is displayed to the user, the user can intuitively see the information that the first task has been successfully completed, which improves interactivity and enhances the user experience.

[0016] In some implementations, the method further includes: if the first plugin fails to load, sending a second response request to the cloud server via the cloud service, the second response request carrying information about the first plugin failing to load, and the second response request being used to generate a second response message; receiving the second response message via the cloud service; sending the second response message to the assistant application; and displaying the second response message through the dialog box interface of the assistant application.

[0017] When the first task fails, the electronic device can also request the cloud server to generate a corresponding second response message through the cloud service. When the second response message is displayed to the user, the user can intuitively see the information that the first task failed, which can avoid unnecessary waiting and improve the user experience.

[0018] In some implementations, the task list also includes a second task, and obtaining the task list corresponding to the intent includes: asynchronously receiving the first task and the second task.

[0019] Electronic devices can receive different tasks asynchronously without waiting for multiple tasks to be transmitted synchronously, thus improving the efficiency of task processing.

[0020] In some implementations, the method further includes: sending a second loading request to a second plugin through a second plugin interface; if the second plugin loads successfully, obtaining the identifier of the second plugin through the second plugin interface; sending a second task to the second plugin based on the identifier of the second plugin, wherein the second plugin is used to call the second tool corresponding to the second plugin to execute the second task and obtain a second processing result; sending the second processing result to an assistant application, wherein the second processing result and the first processing result are sent asynchronously; and displaying the second processing result through a dialog box interface of the assistant application.

[0021] Electronic devices can use different plugin interfaces to call different plugins, thereby calling different tools to perform different tasks through different plugins. Then, the processing results of different tasks are asynchronously returned to the dialog interface of the assistant application, so that users can intuitively see the results of each task execution, improving interactivity and enhancing user experience.

[0022] In some implementations, the intent includes the intention to remove pedestrians from the image to be processed, and the first processing result is an image after removing pedestrians from the image to be processed.

[0023] When a user wants to remove pedestrians from an image, the electronic device can automatically understand the user's intent based on the input statement, generate the corresponding removal task, and execute it automatically. This eliminates the need for the user to manually select the objects to be removed, making it more convenient and faster. Furthermore, users don't need to learn how to select or remove objects; the device understands the user's intent and performs automatic removal, lowering the barrier to entry and improving the user experience.

[0024] In some implementations, the first task is to remove passersby, and the first plugin is an image removal plugin.

[0025] Electronic devices can automatically invoke image removal plugins to remove unwanted people, enhancing the user's experience. Optionally, when a user's message expresses multiple intentions, the electronic device can also automatically invoke other plugins to perform additional processing, enabling the simultaneous execution of multiple tasks. This avoids the tediousness of repeated manual operations, improves multitasking efficiency, and enhances the user experience.

[0026] Secondly, an image processing method is provided, applied to an electronic device. The electronic device includes an assistant application, a media proxy module, and a cloud service. The media proxy module includes a media proxy service, a plug-in interface module, and at least one plug-in. The at least one plug-in includes a first plug-in. The plug-in interface module includes at least one first plug-in interface. The method includes: the assistant application acquiring an image to be processed and a user-inputted statement; the assistant application sending the statement to the media proxy service; the media proxy service sending the statement to the cloud service, whereby the cloud service calls a cloud-side server to recognize the statement, obtain the user's intent, and generate a task list based on the intent, the task list including a first task; the media proxy service receiving the task list through the cloud service; and the media proxy service sending a first task to the first plug-in through the first plug-in interface. The process involves the following steps: The media proxy service receives the identifier of the first plugin returned by the first plugin through the first plugin interface upon successful loading of the first plugin. Based on the identifier, the media proxy service sends the first task to the first plugin. The first plugin calls the first tool corresponding to the first task to execute the first task and obtain a first processing result. The first plugin sends the first processing result to the media proxy service. The media proxy service forwards the first processing result to the assistant application. The assistant application displays the first processing result. The media proxy service sends a first reply request to the cloud server through the cloud service. The cloud server is also used to generate a first reply message based on the first reply request. The media proxy service receives the first reply message through the cloud service. The media proxy service returns the first reply message to the assistant application. The assistant application displays the first reply message.

[0027] The image processing method provided in this application can automatically understand the user's intent based on the user's input, then break it down into corresponding tasks and execute them, thereby automatically completing the processing operations required by the user. This eliminates the need for the user to manually and repeatedly edit the image, making it more convenient and faster to use. Furthermore, it eliminates the need for users to possess image editing skills to understand and edit the image, lowering the barrier to entry and improving the user experience. Moreover, this method can call corresponding plugins to implement image processing functions of different types. When more processing functions are needed, the functionality can be expanded by adding plugins and corresponding interfaces, making it flexible and applicable to a wide range of scenarios.

[0028] Thirdly, an image processing apparatus is provided, comprising a unit consisting of software and / or hardware, the unit being used to perform any one of the methods described in the first or second aspect.

[0029] Fourthly, embodiments of this application provide a chip, including a processor; the processor is configured to read and execute a computer program stored in a memory to perform any one of the methods described in the first or second aspect.

[0030] Optionally, the chip further includes a memory, which is connected to the processor via a circuit or wire.

[0031] Alternatively, the chip may further include a communication interface.

[0032] Fifthly, an electronic device is provided, comprising: a processor, a memory, and an interface; the processor, memory, and interface cooperate with each other to enable the electronic device to perform any one of the methods described in the first or second aspect.

[0033] In a sixth aspect, an electronic device is provided, which includes any one of the chips described in the fourth aspect.

[0034] In a seventh aspect, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, the processor performs any one of the methods described in the first or second aspect.

[0035] Eighthly, a computer program product is provided, the computer program product comprising: computer program code, which, when executed on an electronic device, causes the electronic device to perform any one of the methods described in the first or second aspect. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the structure of a terminal device 100 provided in an embodiment of this application;

[0037] Figure 2 This is a software structure block diagram of the terminal device 100 provided in the embodiments of this application;

[0038] Figure 3 This is a schematic diagram of an example image processing software architecture provided in an embodiment of this application;

[0039] Figure 4 This is a schematic diagram of an example of a dialog elimination interface provided in an embodiment of this application;

[0040] Figure 5 This is another example of a dialog box elimination interface provided in the embodiments of this application;

[0041] Figure 6 This is another example of a dialog box elimination interface provided in the embodiments of this application;

[0042] Figure 7 This is another example of a dialog box elimination interface provided in the embodiments of this application;

[0043] Figure 8 This is another example of a dialog box elimination interface provided in the embodiments of this application;

[0044] Figure 9 This is a flowchart of an example image processing method provided in an embodiment of this application;

[0045] Figure 10 This is a schematic diagram of an example of a dialog elimination interface provided in an embodiment of this application;

[0046] Figure 11 This is an interactive diagram of an example image processing method provided in an embodiment of this application;

[0047] Figure 12 This is an interactive diagram of another example of an image processing method provided in the embodiments of this application;

[0048] Figure 13 This is a flowchart of yet another image processing method provided in the embodiments of this application;

[0049] Figure 14 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0051] Hereinafter, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include one or more of that feature.

[0052] The image processing method provided in this application can be applied to terminal devices such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). This application does not impose any restrictions on the specific type of terminal device.

[0053] For example, Figure 1This is a schematic diagram of the structure of a terminal device 100 provided in an embodiment of this application. The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0054] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0055] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0056] The controller can serve as the central nervous system and command center of the terminal device 100. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0057] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0058] In some implementations, the processor 110 may include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0059] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may also adopt different interface connection methods or a combination of multiple interface connection methods as described in the above embodiments.

[0060] Terminal device 100 implements display functions through a GPU, display screen 194, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0061] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a minimized display, a microLED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, terminal device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0062] Terminal device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0063] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some implementations, the ISP can be integrated into the camera 193.

[0064] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the terminal device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0065] A digital signal processor (DSP) is used to process digital signals. Besides digital image signals, it can also process other digital signals. For example, when terminal device 100 selects a frequency, the DSP performs Fourier transforms on the frequency energy.

[0066] Video codecs are used to compress or decompress digital video. Terminal device 100 may support one or more video codecs. Thus, terminal device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0067] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in terminal devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0068] Terminal device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0069] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0070] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The terminal device 100 can listen to music or make hands-free calls through the speaker 170A.

[0071] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the terminal device 100 answers a phone call or voice message, the receiver 170B can be brought close to the listener's ear to hear the voice.

[0072] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Terminal device 100 may be equipped with at least one microphone 170C. In some embodiments, terminal device 100 may be equipped with two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, terminal device 100 may be equipped with three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0073] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0074] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of terminal device 100, in a different position than display screen 194.

[0075] The software system of terminal device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of terminal device 100.

[0076] Figure 2 This is a software structure block diagram of the terminal device 100 according to an embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some implementations, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer. The application layer may include a series of application packages.

[0077] like Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, SMS, and smart voice assistant.

[0078] For example, a smart voice assistant can be called YOYO Assistant, or simply YOYO. A smart voice assistant can interact with users, understand their needs, and provide corresponding services based on those needs.

[0079] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0080] like Figure 2 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0081] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0082] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0083] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0084] The phone manager is used to provide communication functions for terminal device 100. For example, it manages call status (including connection, hang-up, etc.).

[0085] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0086] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of download completion or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating the device, and flashing indicator lights.

[0087] The application framework layer can also set up a media agent module (MediaAgent). The media agent module is the base Android application package (APK) (BaseAPK), which is a standalone APK. It can call different plugins through the plugin interface module (Plugin SDK) to perform different functionalities on images, etc. Optionally, the media agent module can include one or more plugins. Different plugins can handle different types of tasks. These plugins can be called PluginAPKs.

[0088] The Plugin SDK is used for interface declaration and associates BaseAPK and PluginAPK, bridging them together. The association between BaseAPK and Plugin SDK is static, while the association between BaseAPK and PluginAPK is dynamic, requiring the association to be loaded during compilation.

[0089] Optionally, the media proxy module can also interact with the cloud-side server through interface services, thereby calling the large cloud-side model to complete some complex tasks.

[0090] Image removal plugins are a type of pluginAPK used to perform image removal tasks. Other pluginAPKs may include those for beautifying images, generating ID photos, and so on.

[0091] The Android runtime consists of core libraries and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system.

[0092] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0093] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0094] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0095] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0096] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0097] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0098] A 2D graphics engine is a graphics engine for 2D drawing.

[0099] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0100] Figure 3 The software architecture diagram related to the technical solutions of the embodiments of this application shows that task processing can be implemented through a combination of edge-side and cloud-side approaches. For example... Figure 3 As shown, the terminal (electronic device) is equipped with a cloud service (e.g., a voice cloud service, LLM service) for interacting with the cloud-side server. In this embodiment, the cloud-side server used for processing statements can be called a voice cloud.

[0101] The client-side also includes a Media Agent module. Within this module is a Media Agent Service, contained in the Base APK. This service provides inter-process communication, such as interacting with YOYO for request and image interactions. It can also communicate with the Voice Cloud service to make business requests to the cloud-side Large Language Model (LLM), processing user image editing requests, breaking down user statements (instructions) into tasks, generating responses, and obtaining image processing results and responses. The Voice Cloud service serves as the interface for the cloud-side Large Language Model (LLM), i.e., the interface service for the Voice Cloud.

[0102] The media proxy module also includes a plugin interface module (Plugin SDK) and at least one plugin. The media proxy service can also interact with the corresponding plugins through the plugin interface module. For example, the plugin interface module includes various plugin interfaces, such as the PhotoEraseInterface. The media proxy service can then interact with the photoerase plugin through the PhotoEraseInterface. Specifically, PhotoEraseInterface is included in the Plugin SDK and is used to provide business support for the media proxy service. Through the pluggable capabilities of the Microsoft Outsourcing Service Package (MOSP), the capabilities of the PluginAPK are provided to the media proxy service via interface method calls.

[0103] Optionally, the image removal plugin in the media proxy module is included in PluginApk. It can interact with the visual SDK corresponding to the visual cloud (a cloud testing server containing a large visual model) and call the visual cloud's large visual model to perform complex image processing tasks such as intelligent removal. The image removal plugin provides the PhotoEraseInterface image removal capability to BaseApk.

[0104] For ease of understanding, the following embodiments of this application will be described using the following methods: Figure 1 , Figure 2 and Figure 3 Taking the terminal device with the structure shown as an example, the technical solution provided by the embodiments of this application will be specifically described in conjunction with the accompanying drawings and application scenarios.

[0105] Taking the intelligent voice assistant YOYO as an example, YOYO can interact with users through natural language. For example, users can say a wake word to the electronic device, such as "Hello YOYO," to wake up YOYO. They can also bring the electronic device close to their mouth to wake up YOYO using breath.

[0106] When YOYO is activated, its program interface is displayed on the electronic device's screen. Optionally, YOYO's program interface can be displayed on the taskbar on the desktop. Users can enter statements in the taskbar, which can be in text or voice format. When in voice format, YOYO can recognize the speech, generate text, and obtain the user's command. YOYO can also display a dialog interface after being activated by the user. Users can enter statements to be processed in the dialog box.

[0107] For example, when a user wants to intelligently edit a picture, they can input the image and a sentence in the intelligent voice assistant's dialog box to execute the AI ​​editing function. Next, we will combine... Figures 4 to 8 The relevant interfaces of the AI ​​editing function provided in the embodiments of this application are described.

[0108] In some implementations, users can trigger the image processing function of the intelligent voice assistant in various ways to edit an image. The following will combine... Figures 4 to 8 This is illustrated by example:

[0109] The following is combined with Figure 4 The first method is illustrated by example:

[0110] like Figure 4 As shown in (a), the electronic device displays interface 21, which can display icons for multiple applications, assuming it includes an icon 211 for the gallery application. Optionally, interface 21 can be referred to as the main interface or desktop. Subsequently, the electronic device receives an operation applied to the icon 211 of the gallery application in interface 21, and in response to this operation, as shown in (a). Figure 4 As shown in (b), the electronic device can display an interface 22 of a gallery application. Interface 22 can display thumbnails of each of multiple images, assuming it includes a thumbnail 221 of image a. Subsequently, the electronic device receives an operation applied to thumbnail 221 of image a, and in response to this operation, as shown in (b). Figure 4 As shown in (c), the electronic device can display an interface 23 of a gallery application. Interface 23 can display image a, as well as multiple controls. For example, these multiple controls may include share, favorite, AI edit, delete, more, etc. Then, the electronic device receives an operation applied to image a and responds to that operation as follows: Figure 4As shown in (d), the electronic device can display the interface 24 of the gallery application. The interface 24 can display a full-screen view of image a. Optionally, the interface 24 can be referred to as a large page of image a. For example, in this embodiment, the operation received by the electronic device acting on the content displayed on the electronic device (such as icons, thumbnails, images, controls, or options) can be a voice operation, a gesture operation (such as clicking, long-pressing, or swiping), or a motion-sensing operation (such as device shaking or air gestures), etc., and this embodiment does not limit this.

[0111] like Figure 4 As shown in (d) above, in some embodiments, when the electronic device displays interface 24, it can automatically display the input box 241 of the intelligent voice assistant and multiple AI editing controls on interface 24. In other embodiments, the user can wake up the intelligent voice assistant while the electronic device displays interface 24. After the intelligent voice assistant is woken up, the electronic device can display the input box 241 of the intelligent voice assistant and multiple AI editing controls on interface 24. For example, the user can wake up the intelligent voice assistant using any of the following methods: voice wake-up, breath wake-up, button wake-up, navigation bar wake-up, etc. For example, the multiple AI editing controls may include a pedestrian removal control 242, a one-click skin beautification control 243, and an intelligent removal control 244. The pedestrian removal control 242 is used to trigger the removal of pedestrians in the image. The one-click skin beautification control 243 is used to trigger skin beautification of people in the image. The intelligent removal control 244 is used to trigger the removal of specific targets in the image. Specific targets may include one or more of people, animals, plants, objects, text, watermarks, etc. In this embodiment, "passersby" can refer to other figures in the image besides the main subject, such as pedestrians in the background, people passing by, or other individuals in the scene who do not directly interact with the main subject. The main subject typically refers to the central figure in the image, usually located in the center or a prominent position within the image.

[0112] In this scenario, if a user wants to remove passersby from image a, they can operate the "Remove Passersby" control 242. Upon receiving the operation on the "Remove Passersby" control 242, the electronic device can, in response to this operation, perform passersby removal processing on image a, and can display the chat window 26 of the intelligent voice assistant. For example, as... Figure 4As shown in (e), the chat window 26 can be displayed on the interface 24 in a floating window or similar manner, and the interface 24 and the chat window 26 together constitute the interface 25. When the chat window 26 is displayed on the interface 24, the chat window 26 does not completely cover the interface 24. The chat window 26 may include an input box 261 and a dialogue area 262. Optionally, during the process of removing pedestrians from image a, the electronic device may display the user message "Remove pedestrians" and the intelligent voice assistant message "Image generation in progress" in the dialogue area 262 to remind the user that pedestrians in image a are being removed.

[0113] Assuming the electronic device obtains image b after performing pedestrian removal processing on image a, the electronic device can display a thumbnail of image b in dialog area 262. For example, as... Figure 4 As shown in (e), after completing the pedestrian removal process for image a, the electronic device may no longer display the intelligent voice assistant message "Image generation in progress," but instead display the intelligent voice assistant messages "Completed, you can save directly or perform custom editing" and "Thumbnail of image b." Optionally, as... Figure 4 As shown in (e), the electronic device can display multiple controls while showing a thumbnail of image b, such as a sharing control 263, a save control 264, a custom editing control 265, and an AI retouching control 266. The sharing control 263 is used to share image b. The save control 264 is used to save image b locally. The custom editing control 265 is used for custom editing. The AI ​​retouching control 266 is used for AI retouching of image b. Users can manipulate image d using these controls.

[0114] The above text Figure 4 In the first approach provided by the embodiment, users can decide whether image processing is needed while viewing images in the gallery application, and instantly initiate image processing functions via a smart voice assistant when image processing is required. This improves ease of operation and makes image processing smoother and more intuitive.

[0115] The following is combined with Figure 5 and Figure 6 The second method is illustrated by example:

[0116] like Figure 5 As shown in (a), the electronic device displays interface 31, which can be any interface displaying an image. Here, it is assumed that interface 31 is a chat interface of an instant messaging application, where the user and / or their friend sends an image c to each other. Subsequently, the electronic device receives an operation applied to image c (such as a long press), and in response to this operation, the electronic device can display a thumbnail 321 of image c. For example, as... Figure 5As shown in (b), thumbnail 321 can be displayed on interface 31, and interface 31 and thumbnail 321 together constitute interface 32. When thumbnail 321 is displayed on interface 31, it can obscure the content at the corresponding position in interface 31. The user can drag thumbnail 321, and thumbnail 321 can move within interface 32 as the user drags it.

[0117] After that, as Figure 4 As shown in (c), when the electronic device detects that the thumbnail 321 has been dragged to near the edge of the screen, it can display interface 33. Interface 33 can display the thumbnail 321, screen 331, and service recommendation bar 332. Optionally, screen 331 can display the content of the scaled-down interface 31. For example, screen 331 is used to create the visual effect of interface 31 being pushed inward on the screen. Service recommendation bar 332 is used to display icons of one or more services, which may include the icon 333 of the intelligent voice assistant. Users can drag the thumbnail 321 to the position of an icon in service recommendation bar 332 according to their own needs and then release it to transmit image c to the service corresponding to the icon and start the corresponding service function. It should be noted that the service described in this application embodiment is a program used to implement a specific function, such as an application, applet, quick app (also known as an installation-free application), or certain functional modules of an application, etc., and this application embodiment does not limit this.

[0118] Subsequently, after detecting that thumbnail 321 has been dragged to and released from the location of the smart voice assistant icon 333, the electronic device can transmit image c to the smart voice assistant and display the smart voice assistant's chat window 35. For example, as... Figure 6 As shown in (a), the chat window 35 can be displayed on the interface 31 in a floating window or similar manner, and the interface 31 and the chat window 35 together constitute the interface 34. When the chat window 35 is displayed on the interface 31, the chat window 35 does not completely cover the interface 31. The chat window 35 may include an input box 351, multiple AI editing controls, and a dialogue area 352. The multiple AI editing controls may include a passerby removal control 353, a one-click skin beautification control 354, a smart removal control 355, etc. The functions of the multiple AI editing controls have been described above. Figure 4 The example is described below and will not be repeated here. The user message "thumbnail of image c" can be displayed in the dialog area 352.

[0119] In this scenario, if a user wants to remove pedestrians from image c, they can operate the pedestrian removal control 353. Upon receiving the operation on the pedestrian removal control 353, the electronic device can perform pedestrian removal processing on image c in response to the operation. Optionally, during the pedestrian removal processing of image c, the electronic device can display the user message "Remove Pedestrians" and the intelligent voice assistant message "Image Generating" in the dialog area 352 to remind the user that pedestrians are being removed from image c.

[0120] Assuming the electronic device obtains image d after performing pedestrian removal processing on image c, the electronic device can display a thumbnail of image d in dialog area 352. For example, as... Figure 6 As shown in (b), after completing the pedestrian removal process for image c, the electronic device may no longer display the intelligent voice assistant message "Image generation in progress," but instead display the intelligent voice assistant message "Completed, you can save directly or perform custom editing" and "Thumbnail of image d." Optionally, as... Figure 6 As shown in (b) above, the electronic device can display multiple controls simultaneously with the thumbnail of image d, such as a share control 361, a save control 362, a custom edit control 363, and an AI image editing control 364. The functions of these multiple controls have been described above. Figure 4 The examples are illustrated below and will not be repeated here. Users can manipulate image d using these controls.

[0121] The above text Figures 5 to 6 In the second method provided in the embodiment, users can quickly transmit images from any interface to the intelligent voice assistant for processing. This improves operational flexibility and convenience.

[0122] The following is combined Figure 7 The third method is illustrated by example:

[0123] The electronic device can display interface 41, which can be any interface. The user can wake up the intelligent voice assistant while the electronic device displays interface 41. After the intelligent voice assistant is woken up, the electronic device can display the input box 421 of the intelligent voice assistant. For example, such as... Figure 7 As shown in (a), the input box 421 can be displayed on the interface 41 in a floating window or similar manner. The interface 41 and the input box 421 can together constitute the interface 42. When the input box 421 is displayed on the interface 41, the input box 421 does not completely cover the interface 41. The input box 421 may include an add control 422 and an input mode switching control 423. The add control 422 is used to add images or files. The input mode switching control 423 is used to switch the input mode between voice input and text input.

[0124] In this situation, if the user wants to perform image processing, they can operate the added control 422. The electronic device receives the operation applied to the added control 422 and responds to the operation as follows: Figure 7 As shown in (b), the electronic device can display multiple controls in the input box 421. These controls may include an image control 424, a camera control 425, a file control 426, etc. Users can select a local image for processing using the image control 424, or capture an image for processing using the camera control 425. The following explanation uses the image control 424 as an example to illustrate how to input an image to the intelligent voice assistant. For the method of inputting an image to the intelligent voice assistant using the camera control 425, please refer to the following text. Figure 8 The relevant descriptions in the embodiments will not be repeated here.

[0125] Assuming the electronic device receives an operation applied to the image control 424, in response to that operation, such as Figure 7 As shown in (c), the electronic device can display an interface 43 of a gallery application. Interface 43 can display thumbnails of each of multiple images, assuming it includes a thumbnail 531 of image e. Subsequently, the electronic device receives an operation applied to thumbnail 531 of image e, and in response to this operation, as shown in (c). Figure 7 As shown in (d), the electronic device can display interface 44. Interface 44 can display image e, as well as a return control 441 and a completion control 442. Upon receiving an operation on the completion control 442, the electronic device can, in response to this operation, display the chat window 46 of the intelligent voice assistant. For example, as... Figure 7 As shown in (e), the chat window 46 can be displayed on the interface 41 in a floating window or similar manner. The interface 41 and the chat window 46 together constitute the interface 45. When the chat window 46 is displayed on the interface 41, the chat window 46 does not completely cover the interface 41. The chat window 46 may include an input box 461, multiple AI editing controls, and a dialogue area 462. The multiple AI editing controls may include a passerby removal control 463, a one-click skin beautification control 464, and a smart removal control 465, etc. The functions of the multiple AI editing controls have been described above. Figure 4 The example is described below and will not be repeated here. The user message "thumbnail of image e" can be displayed in the dialog area 462.

[0126] After that, as Figure 7 As shown in (e) to (f), the user can use the intelligent voice assistant to remove pedestrians in image e to obtain image f. Figure 7 The processes shown in (e) to (f) are similar to Figure 6 The processes shown in (a) to (b) are similar; please refer to the above text for details. Figure 6The relevant descriptions in the embodiments will not be repeated here.

[0127] The above text Figure 7 In the third method provided in this embodiment, the user can activate the intelligent voice assistant from any interface. The intelligent voice assistant serves as a centralized operation window, allowing users to easily add and process images using the provided controls. The process is simple and improves user efficiency.

[0128] The following is combined with Figure 8 The fourth method is illustrated by example:

[0129] The electronic device can display interface 51, which can be any interface. The user can wake up the intelligent voice assistant while the electronic device displays interface 51. After the intelligent voice assistant is woken up, the electronic device can display the input box 521 of the intelligent voice assistant. For example, such as... Figure 8 As shown in (a), the input box 521 can be displayed on the interface 51 in the form of a floating window or the like, and the interface 51 and the input box 521 can together constitute the interface 52. When the input box 521 is displayed on the interface 51, the input box 521 does not completely cover the interface 51.

[0130] The electronic device receives an operation applied to the input box 521 (such as a swipe-up operation starting from the input box 521), and responds to the operation as follows: Figure 8 As shown in (b), the electronic device can display an interface 53 for a smart voice assistant. Interface 53 can display an input box 531 and multiple options, including a YOYO Assistant option 532, recent options 533, and an AI agent option 534. The YOYO Assistant option 532 displays the dialogue page of the smart voice assistant. The recent options 533 display the AI ​​agents (artificial intelligence agents) recently used by the user. The AI ​​agent option 534 displays one or more AI agents that the user can use. An AI agent is an intelligent program with autonomous perception, decision-making, and execution capabilities, which can perform specific tasks through intent understanding, task decomposition, and resource scheduling.

[0131] Suppose the electronic device receives an operation applied to agent option 534, and in response to that operation, such as Figure 8 As shown in (c), the electronic device can display interface 54. Interface 54 can display multiple agent icons, which may include an icon 541 for an image editing agent, an icon 542 for a video editing agent, etc. The image editing agent is used to implement functions related to image editing. The video editing agent is used to implement functions related to video editing. The user can select the desired agent from the agents provided in interface 54. Optionally, as... Figure 8 As shown in (c), a search box 543 can also be displayed in the interface 54, through which the user can search for the desired intelligent agent.

[0132] Suppose the electronic device receives an operation on icon 541 applied to the image editing agent, and in response to that operation, such as Figure 8 As shown in (d), the electronic device can display the interface 55 of the image editing intelligent agent. Interface 55 can be a chat window for an intelligent voice assistant, displaying an input box 551, multiple AI editing controls, and a dialogue area 552. These multiple AI editing controls may include a pedestrian removal control 553, a smart removal control 554, a one-click skin beautification control 555, a smart background removal control 556, and a photo ID generation control 557. The smart background removal control 555 is used to trigger the separation of the subject (such as a person, animal, or object) in the image. The photo ID generation control 557 is used to trigger the generation of a photo ID for the person in the image. The functions of the pedestrian removal control 553, the smart removal control 554, and the one-click skin beautification control 555 have been described above. Figure 4 The examples are illustrated below and will not be repeated here.

[0133] In this situation, if the user wants to perform image processing, they can operate the pedestrian removal control 553. The electronic device receives the operation applied to the pedestrian removal control 553 and responds to the operation as follows: Figure 8 As shown in (e), the electronic device can display the user message "Remove passersby," the intelligent voice assistant message "Please upload a picture containing people," a camera control 558, and an image control 559 in the dialogue area 552. The user can take a picture using the camera control 558 and process it, or select a local image using the image control 559 and process it. The following explanation uses the method of inputting an image to the intelligent voice assistant via the camera control 558 as an example. For the method of inputting an image to the intelligent voice assistant via the image control 559, please refer to the above text. Figure 7 The relevant descriptions in the embodiments will not be repeated here.

[0134] Assuming the electronic device receives an operation applied to the camera control 558, it responds to the operation as follows: Figure 8 As shown in (f), the electronic device can display the camera application interface 56. Interface 56 is a shooting interface, which may include a shooting preview window 561 and a shooting control 562. After the user determines that the shooting screen in the shooting preview window 561 meets their needs, they can operate the shooting control 562. Upon receiving the operation applied to the shooting control 562, the electronic device, in response to the operation, can take a picture, obtaining image g, and, as shown... Figure 8As shown in (g), the electronic device can display interface 57. Interface 57 can display image g, a return control 571, and a completion control 572. Upon receiving an operation on the completion control 572, the electronic device can perform pedestrian removal processing on image g in response to this operation. Optionally, during pedestrian removal processing on image g, the electronic device can display the user messages "thumbnail of image g" and "remove pedestrians" in the dialogue area 552 of interface 55, and no longer display the photo control 558 and image control 559. Optionally, the electronic device can also display the intelligent voice assistant message "Image generation in progress" in the dialogue area 552 to remind the user that pedestrians in image g are being removed.

[0135] Assuming that the electronic device obtains image h after performing pedestrian removal processing on image g, the electronic device can display a thumbnail of image h in the dialog area 552. For example, as... Figure 8 As shown in (h), after the electronic device completes the pedestrian removal process for image g, it may no longer display the intelligent voice assistant message "Image generation in progress," but instead display the intelligent voice assistant message "Completed, you can save directly or perform custom editing" and "Thumbnail of image h." Optionally, as... Figure 8 As shown in (h), the electronic device can display multiple controls simultaneously with a thumbnail of image h, such as a share control, a save control, a custom edit control, and an AI image editing control. The functions of these multiple controls have been described above. Figure 4 The examples are illustrated below and will not be repeated here. Users can manipulate image h using these controls.

[0136] The above text Figure 8 In the fourth approach provided in the embodiment, by offering different intelligent agents, users can select the required intelligent agent for corresponding professional processing according to their own needs. For example, users can use the image editing intelligent agent to concentrate on image processing and avoid being disturbed by other functions or applications.

[0137] One point to note is that the above text Figures 4 to 8 This embodiment uses the example of triggering pedestrian removal processing on an image via a pedestrian removal control provided by a smart voice assistant for illustrative purposes. In practical applications, other controls provided by the smart voice assistant, such as smart removal controls, one-click skin beautification controls, smart background removal controls, and ID photo generation controls, can also be used to trigger other image processing. This embodiment does not limit this approach. Furthermore, image processing can be triggered not only by controls provided by the smart voice assistant but also by entering user commands in the smart voice assistant's input box. For example, a user can enter the command "remove pedestrians" in the smart voice assistant's input box via voice or text to trigger pedestrian removal processing on the image.

[0138] Another point to note is that the above text Figures 4 to 8 This embodiment only uses the functions provided by the pedestrian removal control, intelligent removal control, one-click skin beautification control, intelligent background cutout control, and ID photo generation control as examples for illustrative purposes. In practical applications, intelligent voice assistants can also provide other image processing functions. For example, they can add special effects or borders to images; add specific elements to images; and provide some simple image editing functions, such as cropping, filters, mosaic, and background blurring.

[0139] The embodiments in this application are only as described above. Figures 4 to 8 The process described in the embodiment is used as an example to illustrate the process of triggering the image processing function of the intelligent voice assistant. Figures 4 to 8 The processes described in the embodiments do not constitute a specific limitation on the implementation of the embodiments of this application. In practical applications, the image processing function of the intelligent voice assistant can also be triggered in other ways than the four methods described above, and the embodiments of this application do not limit this.

[0140] The aforementioned intelligent voice assistant messages can also be referred to as replies.

[0141] It should be noted that the embodiments in this application are only those described above. Figures 4 to 8 The process described in the embodiment is used as an example to illustrate the image editing process based on an intelligent voice assistant. Figures 4 to 8 The process described in the embodiments does not constitute a specific limitation on the implementation of the embodiments of this application. In practical applications, the image editing process based on the intelligent voice assistant can also be implemented in other ways different from the above methods.

[0142] The process described above, in which a user inputs their voice or command into a dialog box of an intelligent voice assistant and displays an AI-edited image and response through the dialog box, can be called a dialogue format.

[0143] The following section, with reference to the accompanying diagrams, provides a detailed explanation of how electronic devices perform image processing in a dialog-like manner.

[0144] For ease of description, the following text will use YOYO as an example of a smart voice assistant. Figure 9 A flowchart illustrating intelligent image removal is shown, including:

[0145] S901. Obtain the user-input statement and the image to be processed.

[0146] After opening YOYO, users can operate the YOYO interface, such as sending images and statements to be processed in the YOYO dialog box. The image to be processed is the image the user wants to edit. Optionally, users can click "Add Control" in the YOYO interface to select an image from their gallery and send it to the YOYO dialog box. Alternatively, YOYO can receive images sent (or shared) by other applications. Users can also click "Add Control" in the YOYO interface and then select the "Capture Control" to capture an image on-site. This application does not limit the source of the image to be processed.

[0147] The aforementioned statements can be voice input by the user, text input by the user manually using the keyboard, text converted from voice input, or pre-recorded audio or text selected by the user from other files by clicking the "Add Control" button. This application does not limit the method or type of statements input by the user.

[0148] If YOYO receives a speech, it can convert the speech into text and then send it to the speech engine. If YOYO receives a text, it can send the text directly to the speech engine, or perform preliminary processing, such as deleting invalid or duplicate characters, before sending it to the speech engine. Optionally, the speech can be a single sentence received at once, or multiple sentences received within a certain time period.

[0149] Optionally, the user can send the image to be processed first in the YOYO dialog box, and then enter a statement and send it; alternatively, the user can also enter a statement first and send it, and then send the image to be processed. This application embodiment does not limit the order in which the image to be processed and the statement are sent. When the electronic device receives the image to be processed and the statement through YOYO within a certain time period, it can determine that the semantics expressed by the statement represent a processing intention for the image to be processed. If the time interval between receiving the image to be processed and the statement is relatively long, it indicates that the semantics expressed by the user's input statement may not represent a processing intention for the image to be processed, and therefore no processing is required, or the system can wait for the next statement or image to be sent.

[0150] For example, after receiving the image and statement to be processed, YOYO can, as follows: Figure 10 As shown in (a), the image to be processed (e.g., picture 1) and the statement (e.g., eliminate passersby) are displayed in the dialog box. Figure 10 Example (a) is given in text form using statements.

[0151] Optionally, YOYO can also send the images to be processed to a storage cloud. The storage cloud, or Object Storage Service (OBS) cloud, can store large amounts of data for use in providing data to other cloud-side servers and edge-side devices. Images to be processed stored in the storage cloud can be directly accessed by the voice cloud and visual cloud, eliminating the need to retrieve them from the edge each time, thus saving bandwidth.

[0152] When the user inputs a statement that contains elimination semantics, it can also be regarded as an elimination request.

[0153] S902. Based on the statement, obtain the corresponding task list.

[0154] Electronic devices can invoke a local task orchestration model to perform intent recognition on statements, obtain at least one corresponding intent, and then decompose and orchestrate these intents to obtain a corresponding task list. Optionally, the task list may include at least one task.

[0155] Alternatively, the electronic device can send the statement to a cloud-based server, which will then perform intent recognition and task orchestration to obtain a corresponding task list. The cloud-based server will then return the task list to the electronic device.

[0156] Taking the decomposition of intent recognition and execution tasks by voice cloud as an example, when the intent recognized by voice cloud is multiple, voice cloud can decompose the multiple intents into tasks, and then arrange the multiple tasks obtained by decomposition into a task list.

[0157] For example, when recognizing the intentions of removing pedestrians and generating ID photos, the voice cloud can break these two intentions down into two tasks: removing pedestrians and generating ID photos. Then, the voice cloud can orchestrate these two tasks to obtain a task list that includes both tasks. Furthermore, in the task list, the task of removing pedestrians comes first, followed by the task of generating ID photos.

[0158] Optionally, the task list can be a list in the form of a DSL, where DSL stands for Domain-Specific Language. For example: [

[0160] {"taskId":"1","tool":"photo_erase","object_avatar_positions":[{"tag":"passerby",

[0161] "position":{"x":1.0f,"y":1.0f,"width":1.0f,"height":1.0f}}]},

[0162] {"taskId":"2","tool":"photo_id_generator"",""param":{"background_color":"blue",

[0163] "photo width":1.0f,"photo height":1.0f}},

[0164] ].

[0165] This DSL-style task list indicates that the task of eliminating passersby should be performed first, followed by the task of generating a passport photo with a blue background.

[0166] S903. Based on the tasks in the task list, call the corresponding plugin to process the image to be processed and obtain the processing result.

[0167] Optionally, each task needs to invoke a corresponding plugin to execute. In the task list, there may be one task corresponding to one plugin, or multiple tasks corresponding to the same plugin. When multiple tasks invoke the same plugin, the corresponding plugins can be invoked sequentially according to the order of the tasks in the task list.

[0168] At this point, the electronic device can either provide feedback on the processing results after completing all tasks, or it can return the result of the currently generated intermediate state after each task is completed.

[0169] When multiple tasks correspond to different plugins, these plugins can be invoked separately to execute the corresponding tasks. When any two tasks are executed, if one task does not depend on the results of the other two tasks, these two tasks can be executed simultaneously or not, and the execution order is not required to be sequential. For example, the statement is: "Remove pedestrians, brighten the background." The intent generated by the voice cloud is the intent to remove pedestrians and the intent to increase the brightness of the image background. Therefore, the corresponding generated tasks are: the task of removing pedestrians and the task of increasing the background brightness. Removing pedestrians requires invoking the image removal plugin, and increasing the background brightness requires invoking the image editing plugin. The electronic device can then invoke the image removal plugin and the image editing plugin to execute the corresponding tasks, either simultaneously or at different times, depending on the resource configuration of the electronic device.

[0170] Optionally, when there are multiple tasks, the electronic device can also return the processing results asynchronously when calling different plugins to process different characters.

[0171] Optionally, the processing result can be the result after the plugin completes the task processing, information representing the processing result, or intermediate information generated during the process of calling the processing tool to process the task.

[0172] S904. Generate a response based on the processing results.

[0173] Electronic devices can generate a matching response based on the received processing result, or they can send the processing result to the voice cloud, whereby the voice cloud generates a matching response based on the response strategy and feeds it back to the electronic device.

[0174] S905. Display the processing results and / or response.

[0175] Once the electronic device receives the processing result, it can display it in YOYO's dialog box. For example, if the result is an image after removing pedestrians, YOYO can display the image after removing pedestrians. When YOYO receives a reply, it can also display the reply in the dialog box.

[0176] The image processing method provided in this application can automatically understand the user's intent based on the user's input, then decompose it into corresponding tasks and execute them, thereby automatically completing the processing operations required by the user. This eliminates the need for the user to manually and repeatedly edit the image, making it more convenient and faster to use. Furthermore, it eliminates the need for users to possess image editing skills to understand and edit the image, lowering the barrier to entry and improving the user experience. Moreover, this method can call corresponding plugins to implement image processing functions of different types. When more processing functions are needed, the functionality can be expanded by adding plugins and corresponding interfaces, making it flexible and applicable to a wide range of scenarios.

[0177] Figure 11 An interactive diagram illustrating an example of an image intelligent removal process is shown, including:

[0178] S1101, YOYO obtains the image to be processed and the statement.

[0179] For a detailed description of S1101, please refer to the description of S901, which will not be repeated here.

[0180] S1102, YOYO statement sent to media proxy service.

[0181] After receiving the image and statement to be processed, YOYO can send the statement to the media proxy service.

[0182] Optionally, YOYO can also remove duplicates from the input before sending the deduplicated version to the speech engine. For example, when a user inputs, "Eliminate pedestrians, um, that is, remove the pedestrians in the picture," YOYO determines that there are duplicates in the user's input. YOYO first removes duplicates from the user's input, obtaining the deduplicated version "Eliminate pedestrians in the picture," and then sends the deduplicated version to the speech engine.

[0183] Optionally, when the statement is text, YOYO can send the text to the media proxy service; when the statement is speech, YOYO can convert the speech into text and then send the text to the media proxy service.

[0184] S1103, The media proxy service sends statements to the voice cloud via the voice cloud service.

[0185] The media proxy service interacts with the voice cloud service, sending statements to the voice cloud via the voice cloud service. The voice cloud then performs intent recognition on the statements to obtain at least one intent.

[0186] The media proxy service can interact with the voice cloud service and send sentences to the voice cloud. The voice cloud will then perform intent recognition on the sentences, obtain the intent corresponding to the sentences, and return the recognized intent.

[0187] For example, if a user inputs "Remove passersby from the picture," the voice cloud can recognize one intent: the intent to remove passersby. As another example, if the user inputs "Keep only Zhang San in the picture," the intent control center can recognize one intent: the intent to remove everyone except Zhang San. Furthermore, if the user inputs "Remove passersby from the picture and then beautify the image," the voice cloud can recognize two intents: the intent to remove passersby and the intent to beautify the image.

[0188] Optionally, the electronic device may first call its local intent recognition model to identify the intent contained in the statement. Optionally, if the local intent recognition model cannot identify the intent or the identified intent is incomplete, the electronic device may then send the statement to the voice cloud for intent recognition through a media proxy service.

[0189] After recognizing the intent, the voice cloud performs task breakdown and orchestration based on the intent to obtain a task list. Once the voice cloud has the task list, it returns it to the media proxy service.

[0190] Optionally, if the electronic device has a local model for task orchestration deployed locally, the media proxy service can also send intents to the local model for task breakdown and orchestration to obtain a task list.

[0191] Figure 11 The illustrated example uses a statement to represent a task that needs to be performed. For instance, when the statement is "eliminate pedestrians," the recognized intent is "eliminate pedestrians." Therefore, the task list would include the task "eliminate pedestrians."

[0192] S1104. The media proxy service receives the task list returned by the voice cloud through the voice cloud service.

[0193] For example, the media agency service receives a list of tasks via a voice cloud service, including tasks such as removing passersby.

[0194] S1105. The media proxy service requests the image removal plugin corresponding to the task through the plugin interface module and obtains the plugin loading result.

[0195] The media proxy service requests the loading of the target plugin corresponding to the task through the plugin interface module and obtains the plugin loading result of the target plugin.

[0196] The plugin interface module includes various plugin interfaces, such as an image removal interface, and may also include other plugin interfaces. Different plugin interfaces enable the media proxy service to call different functional plugins. For example, the media proxy service can use the image removal interface to call the image removal plugin for intelligent image removal.

[0197] After receiving the task list, the media proxy service can load the corresponding plugin for each task. Taking the task of removing passersby as an example, when the media proxy service determines that it supports the task of removing passersby, it can send a load request back to the image removal plugin through the image removal interface in the plugin interface module.

[0198] Regardless of whether the plugin loads successfully, the target plugin that receives the loading request can return the loading result to the media proxy service through the plugin interface module. Optionally, the loading result can include whether the loading was successful or failed.

[0199] In this embodiment, the target plugin is the image removal plugin, and the corresponding plugin interface is the image removal interface, which is described as an example.

[0200] The media proxy service requests and loads the image removal plugin corresponding to the task of removing passersby through the image plugin interface, and obtains the plugin loading result of the image removal plugin.

[0201] S1106A. When loading is successful, the media proxy service obtains the plugin identifier of the successfully loaded target plugin through the plugin interface module.

[0202] For example, once the image removal plugin has loaded successfully, it can return a success message to the media proxy service through the image removal interface in the plugin interface module. Simultaneously, the image removal plugin can also return its identifier to the media proxy service.

[0203] S1107, the media proxy service sends the task of eliminating passersby to the image elimination plugin for elimination.

[0204] When the media proxy service receives a message indicating successful loading, it can determine that the target plugin is working correctly and will then send the task to the target plugin for processing. For example, when the media proxy service receives a message indicating that the image removal plugin has loaded successfully, it can determine that the image removal plugin is working correctly and will then send the task of removing passersby to the image removal plugin for processing.

[0205] S1108, the image removal plugin calls the visual cloud through the visual interface to remove images and generate the removed image.

[0206] Optionally, the target plugin can call the cloud server through the corresponding interface to process complex tasks and generate processing results.

[0207] When the task is to remove pedestrians, the default interface can be the Vision Kit. The image removal plugin can also use the Vision Kit to call the Vision Cloud to remove pedestrians and obtain the image after removing pedestrians.

[0208] Optionally, the visual cloud can eliminate passersby other than the user-specified object, or it can use visual target localization to locate the target person the user wants to identify based on the semantics expressed in the user's statement. For example, if the user's statement is: "the person wearing red on the left," the visual cloud can locate the person wearing red on the left. After identifying the corresponding object, the visual cloud can perform target segmentation, such as segmenting out the object other than the indicated person wearing red on the left, and then segmenting out the passersby other than the person wearing red on the left. Then, the segmented passersby other than the person wearing red on the left can be eliminated, and then the image can be filled in to generate a complete image for the user.

[0209] Optionally, the target plugin can also invoke third-party tools on the device side to perform task processing on the image to be processed. For example, the image editing plugin can also invoke image editing tools installed on the electronic device to perform editing operations on the image to be processed, such as changing the background to blue, adding a frame, and adjusting the color tone, to obtain the processed image. This application embodiment does not limit the specific manner in which the target plugin performs tasks.

[0210] S1109 The image removal plugin receives the removal image returned by the visual cloud through the visual interface.

[0211] After the visual cloud generates the removed image, it can also be returned to the image removal plugin through the visual interface.

[0212] S1110, the image removal plugin returns the removed image to the media proxy service.

[0213] S1111, the media proxy service will remove the image and return it to YOYO for display.

[0214] S1112. The media proxy service sends the removal result to the voice cloud service and receives the returned response.

[0215] S1113, The media agency service will return the reply to YOYO for display.

[0216] The image removal plugin can return the removed image to the media proxy service, which then returns it to YOYO. Upon receiving the removed image, YOYO can display it in a dialog box for the user to see. For example, the removed image could be image 2 after removing the passerby. The image removal plugin can send image 2 to the media proxy service, which will then return it to YOYO for display. See [link to details] for more information. Figure 10 As shown in (d) in the figure.

[0217] Optionally, the media proxy service can also send the removal result to the voice cloud service and receive the returned response. For example, when the visual cloud returns an image showing the successful removal of a passerby, the media proxy service can determine that the removal result is successful and therefore send the successful removal result to the voice cloud service. Upon receiving the successful removal result, the voice cloud can generate a corresponding response. The voice cloud can then return the generated response to the media proxy service via the voice cloud service. The media proxy service can then send the response to YOYO for display. For example, when the passerby in image 1 is successfully removed, the response could be "Removal successful," or "Completed. You can save it directly or make custom edits," etc.

[0218] When removal fails, the visual cloud can return a message indicating failure; or if the electronic device loses internet access, the media proxy service can determine the removal result as failed and send the failed removal result to the voice cloud service. Upon receiving the failed removal result, the voice cloud can generate a corresponding response. The voice cloud can then return the generated response to the media proxy service via the voice cloud service. The media proxy service can then send the response to YOYO for display. For example, when removing the passerby in image 1 fails, the response could be "Removal failed," or "The network is down, please try again later," etc.

[0219] Upon successful removal, YOYO can display the removed image in a dialog box, and can also display the received response message in the dialog box, such as... Figure 10 As shown in (d) above. When elimination fails, YOYO can also display a response in the dialog box.

[0220] Optionally, during the processing of the image, YOYO's dialog box can also display which stage the current processing is in, for example... Figure 10 The process of "intelligently identifying pedestrians" and "intelligently eliminating pedestrians" is shown in (b) and (c) of the diagram. Optionally, the processing progress, such as 10% or 50%, can also be displayed. The status information displayed here can be information fed back to the visual interface by the visual cloud based on its own processing status and progress, and then fed back to the media proxy service by the visual interface through the image elimination plugin. Optionally, the visual cloud can generate corresponding status information based on its own processing status and progress, and feed it back to the media proxy service through the visual interface and the image elimination plugin, and then the media proxy service feeds back the status information to YOYO for display. Optionally, the media proxy service can generate corresponding responses based on the status information and send them to YOYO for display. Optionally, the media proxy service can also send the status information to the voice cloud to generate corresponding responses. Optionally, generate corresponding responses and send them to YOYO for display.

[0221] S1106B. When loading fails, the media proxy service sends a message about plugin loading failure to the voice cloud service and requests the generation of a response.

[0222] If the target plugin for the task does not exist, the media proxy service sends a plugin loading failure message to the voice cloud service via the voice cloud service, along with a request to generate a response. Alternatively, the media proxy service sends a request to the voice cloud plugin via the voice cloud service to generate a response, carrying a plugin loading failure message.

[0223] Based on the message indicating plugin loading failure, Voice Cloud generates a corresponding response. For example, if the image removal plugin is not present, the generated response could be: "Unable to perform removal task," or "The image removal plugin was not loaded correctly," indicating a failure. Voice Cloud can then send the response to the media proxy service via its Voice Cloud service.

[0224] S1114, The media proxy service receives a reply message indicating that the plugin failed to load.

[0225] S1115. The media proxy service will return a reply message indicating that the plugin failed to load to YOYO for display.

[0226] After receiving a response indicating plugin loading failure, the media proxy service can send the response to YOYO, which will then display it in a dialog box. Users can intuitively and promptly understand the processing progress and results through the content displayed by YOYO, achieving a conversational interaction and enhancing the user experience.

[0227] Optionally, the processed image may also display the source information, such as "Content provided by AI"; alternatively, the processed image may also display one or more of the following: a share control, a save control, and an edit control. When the user clicks the share control, the electronic device can share the processed image to other applications via YOYO, such as chat applications, email applications, etc., and can also share it to other devices, such as printing devices; when the user clicks the save control, the electronic device can save the processed image to the gallery; when the user clicks the edit control, the electronic device can edit the processed image based on the user's operation.

[0228] Optionally, when the task list includes multiple tasks corresponding to different plugins, the electronic device can repeatedly execute steps S1104 to S1115 above, except that the image removal plugin is replaced with another task plugin, and the image removal interface is replaced with another plugin interface. Furthermore, the other task plugins can execute their corresponding tasks by calling other processing tools.

[0229] Figure 12 The example uses an image removal plugin and a beauty plugin. When the user inputs: "Please remove passersby and apply a beauty filter," the corresponding task list includes both "remove passersby" and "beautify" tasks.

[0230] The above methods can also be used as follows Figure 12 As shown, this illustrates the process of loading the beauty plugin and executing the beauty task by calling the plugin. After completing the tasks of removing passersby and beautifying the image, the final result image can be generated.

[0231] When other tasks exist within the statement, the execution process of each task can be seen in the tasks of removing passersby and beautifying the user. These tasks can return results asynchronously during execution, allowing users to promptly understand the current processing progress and improving the user experience. See details in [link to documentation]. Figure 12 As shown, it includes:

[0232] S1101, YOYO obtains the image to be processed and the statement.

[0233] S1102, YOYO statement sent to media proxy service.

[0234] S1103, The media proxy service sends statements to the voice cloud via the voice cloud service.

[0235] S1104. The media proxy service receives the task list returned by the voice cloud through the voice cloud service.

[0236] Optionally, the media proxy service can asynchronously receive tasks returned by the voice cloud service. When the voice cloud recognizes an intent and obtains at least one task, and when the tasks are not strongly related (e.g., the execution order of the tasks is not specified), the voice cloud can return the first determined tasks to the media proxy service first, and the later determined tasks to the media proxy service later. That is, it asynchronously returns the task orchestration results, which allows the first determined tasks to be executed first, improving processing efficiency.

[0237] S1105. The media proxy service requests the image removal plugin corresponding to the task through the plugin interface module and obtains the plugin loading result.

[0238] When the image removal plugin is invoked, the plugin interface module is called through the image removal interface.

[0239] S1106A. When loading is successful, the media proxy service obtains the plugin identifier of the successfully loaded target plugin through the plugin interface module.

[0240] S1107, the media proxy service sends the task of eliminating passersby to the image elimination plugin for elimination.

[0241] S1108, the image removal plugin calls the visual cloud through the visual interface to remove images and generate the removed image.

[0242] S1109 The image removal plugin receives the removal image returned by the visual cloud through the visual interface.

[0243] S1110, the image removal plugin returns the removed image to the media proxy service.

[0244] S1111, the media proxy service will remove the image and return it to YOYO for display.

[0245] S1112. The media proxy service sends the removal result to the voice cloud service and receives the returned response.

[0246] S1113, The media agency service will return the reply to YOYO for display.

[0247] S1205. The media proxy service requests the loading of the beauty plugin corresponding to the task through the plugin interface module and obtains the plugin loading result.

[0248] When the beauty plugin is invoked, the plugin interface module calls the beauty interface.

[0249] The media proxy service requests the loading of the beauty plugin corresponding to the task through the plugin interface module and obtains the plugin loading result of the beauty plugin.

[0250] The media proxy service requests the loading of the beauty plugin corresponding to the beauty task through the beauty interface and obtains the plugin loading result.

[0251] S1206A. When loading is successful, the media proxy service obtains the plugin identifier of the successfully loaded beauty plugin through the plugin interface module.

[0252] For example, once the beauty plugin has loaded successfully, it can return a success message to the media proxy service through the beauty interface in the plugin interface module. Simultaneously, the beauty plugin can also return its identifier to the media proxy service.

[0253] S1207, the media proxy service sends the beautification task to the beautification plugin for beautification.

[0254] When the media proxy service receives a message indicating successful loading, it can determine that the beauty plugin can be used normally, and then sends any beauty task to the beauty plugin for processing.

[0255] S1208, the beauty plugin calls the interface to call the image editing tool to perform beauty enhancement and generate a beauty image.

[0256] S1209, The beauty plugin receives beautified images returned by the image editing tool.

[0257] After the image editing tool generates a beautified image, the beautified image can be returned to the beautification plugin through the corresponding interface.

[0258] S1210, the beauty plugin returns the beautified image to the media proxy service.

[0259] S1211, The media proxy service returns the beautified images to YOYO for display.

[0260] S1212, The media proxy service sends the beautification results to the voice cloud via the voice cloud service and receives the returned response.

[0261] S1213, The media agency service will return the reply to YOYO for display.

[0262] S1206B. When loading fails, the media proxy service sends a message about plugin loading failure to the voice cloud service and requests the generation of a response.

[0263] If the corresponding beauty plugin for the beauty task does not exist, the media proxy service sends a plugin loading failure message to the voice cloud service via the voice cloud service, along with a request to generate a response. Alternatively, the media proxy service sends a request to the voice cloud plugin via the voice cloud service to generate a response, carrying a plugin loading failure message.

[0264] Based on the message indicating plugin loading failure, Voice Cloud generates a corresponding response. For example, if the beauty plugin is not present, the generated response could be: "Unable to perform beauty task," or "Beauty plugin not loaded correctly," indicating a failure. Voice Cloud can then send the response to the media proxy service via its Voice Cloud service.

[0265] S1214, The media proxy service receives a reply message indicating that the plugin failed to load.

[0266] S1215, The media proxy service will return a reply message indicating that the plugin failed to load to YOYO for display.

[0267] When multiple plugins return a "plugin loading failed" message, the media proxy service sends this message to the voice cloud service via the voice cloud service and requests a response. The generated response can be asynchronously returned to the media proxy service for display. For example, when a beauty plugin returns a "plugin loading failed" message, the media proxy service sends this message to the voice cloud service via the voice cloud service and receives a response such as "Unable to perform beauty task." Similarly, when multiple image removal plugins return "plugin loading failed" messages, the media proxy service sends this message to the voice cloud service via the voice cloud service and receives a response such as "Unable to perform image removal task." These two responses can be different or similar.

[0268] Optionally, the replies generated from these multiple plugin loading failure messages can be asynchronous, rather than synchronous.

[0269] The image processing method of this application embodiment allows for the addition of corresponding functions by expanding its plugins when a specific function is required. This method is flexible and easily expandable. In addition to intelligent image removal, it can also perform skin beautification, ID photo generation, filter application, tagging, image sending, and many other functions. Simply add the corresponding plugins and plugin interfaces for invocation; further details are omitted here.

[0270] The foregoing has detailed examples of the methods provided in this application. It is understood that the corresponding apparatus, in order to achieve the above functions, includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0271] This application can divide the image processing apparatus into functional modules based on the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0272] Figure 13 A flowchart of an image processing method provided in this application is shown, including:

[0273] S1301. Obtain the image to be processed and the statement input by the user, the statement containing the user's intention to process the image to be processed;

[0274] S1302. Obtain the task list corresponding to the intent, the task list shall include at least the first task;

[0275] S1303: Call the first plugin to perform the first task on the image to be processed and obtain the first processing result;

[0276] S1304. Display the first processing result.

[0277] The implementation principle and technical effects of this method can be found in the description of the relevant embodiments and implementation methods, and will not be repeated here.

[0278] Figure 14A schematic diagram of the structure of an image processing apparatus provided in this application is shown. The apparatus 1400 includes:

[0279] The first acquisition module 1401 is used to acquire the image to be processed and the statement input by the user, the statement containing the user's intention to process the image to be processed;

[0280] The second acquisition module 1402 is used to acquire a task list corresponding to the intent, and the task list includes at least the first task.

[0281] Processing module 1403 is used to call the first plugin to perform a first task on the image to be processed and obtain a first processing result;

[0282] Display module 1404 is used to display the first processing result.

[0283] In some implementations, an assistant application and cloud services are arranged in the electronic device. The first acquisition module 1401 is specifically used to receive and acquire the image to be processed and the user input statement through the assistant application.

[0284] The second acquisition module 1402 is specifically used to send the statement to the cloud-side server via the cloud service. The cloud-side server is used to convert the statement into an intent and generate a task list corresponding to the intent; and to receive the task list.

[0285] In some implementations, the processing module 1403 is specifically used to send a first loading request to the first plugin through the first plugin interface, the first loading request being used to instruct the first plugin to load; if the first plugin loads successfully, the module obtains the identifier of the first plugin through the first plugin interface; according to the identifier of the first plugin, the module sends a first task to the first plugin, the first plugin being used to call the first tool corresponding to the first task to execute the first task and obtain a first processing result.

[0286] In some implementations, the display module 1404 is specifically used to send the first processing result to the assistant application; and to display the first processing result through the dialog box interface of the assistant application.

[0287] In some embodiments, the device 1400 further includes a response module, configured to send a first response request to a cloud-side server via a cloud service, the first response request carrying information indicating that the first task was successfully executed, the first response request being used to generate a first response message; receive the first response message via the cloud service; and send the first response message to an assistant application.

[0288] Display module 1404 is also used to display the first response through the dialog box interface of the assistant application.

[0289] In some implementations, the response module is further configured to, if the first plugin fails to load, send a second response request to the cloud server via cloud service, the second response request carrying information about the first plugin failing to load, and the second response request being used to generate a second response message; receive the second response message via cloud service; and send the second response message to the assistant application.

[0290] Display module 1404 is also used to display a second response through the dialog interface of the assistant application.

[0291] In some implementations, the task list also includes a second task, and a second acquisition module 1402 is specifically used to asynchronously receive the first task and the second task.

[0292] In some implementations, the processing module 1403 is further configured to send a second loading request to the second plugin through the second plugin interface; if the second plugin is successfully loaded, obtain the identifier of the second plugin through the second plugin interface; according to the identifier of the second plugin, send a second task to the second plugin, the second plugin is used to call the second tool corresponding to the second plugin to execute the second task and obtain a second processing result; send the second processing result to the assistant application, the second processing result and the first processing result are sent asynchronously;

[0293] Display module 1404 is also used to display the second processing result through the dialog interface of the assistant application.

[0294] In some implementations, the intent includes the intention to remove pedestrians from the image to be processed, and the first processing result is an image after removing pedestrians from the image to be processed.

[0295] In some implementations, the first task is to remove passersby, and the first plugin is an image removal plugin.

[0296] The specific manner in which the device 1400 performs the image processing method and the beneficial effects thereof can be found in the relevant descriptions in the method embodiments, and will not be repeated here.

[0297] This application also provides an electronic device, including the processor described above. The electronic device provided in this embodiment may be... Figure 1 The terminal device 100 shown is used to execute the image processing method described above. When using integrated units, the terminal device may include a processing module, a storage module, and a communication module. The processing module can be used to control and manage the actions of the terminal device; for example, it can support the terminal device in executing the steps performed by the display unit, detection unit, and processing unit. The storage module can be used to support the terminal device in executing stored program code and data. The communication module can be used to support communication between the terminal device and other devices.

[0298] The processing module can be a processor or a controller. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination of functions that implement computing capabilities, such as a combination of one or more microprocessors, a digital signal processor (DSP), and a microprocessor, etc. The storage module can be a memory. The communication module can specifically be a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, or other devices that interact with other terminal devices.

[0299] In one embodiment, when the processing module is a processor and the storage module is a memory, the terminal device involved in this embodiment can be a device having... Figure 1 The device with the structure shown.

[0300] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the image processing method described in any of the above embodiments.

[0301] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the image processing method described above.

[0302] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0303] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units. The replaced units may or may not be physically separate. The component shown as a unit may be one physical unit or multiple physical units, that is, it may be located in one place or distributed in multiple different places. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0304] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0305] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0306] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, Applied to an electronic device, wherein a first plug-in is disposed in the electronic device, the method includes: Obtain the image to be processed and the statement input by the user, the statement containing the user's intention to process the image to be processed; Obtain the task list corresponding to the intent, wherein the task list includes at least the first task; The first plugin is invoked to perform a first task on the image to be processed, and a first processing result is obtained; Display the result of the first processing.

2. The method according to claim 1, characterized in that, The electronic device includes an assistant application and cloud services. The process of acquiring the image to be processed and the user input includes: The assistant application receives the image to be processed and the user's input. Obtaining the task list corresponding to the intent includes: The statement is sent to a cloud-side server via cloud service. The cloud-side server is used to convert the statement into the intent and generate the task list corresponding to the intent. Receive the task list.

3. The method according to claim 2, characterized in that, The step of calling the first plugin to perform a first task on the image to be processed and obtaining a first processing result includes: A first loading request is sent to the first plugin through the first plugin interface, and the first loading request is used to instruct the first plugin to load. If the first plugin is loaded successfully, obtain the identifier of the first plugin through the first plugin interface; Based on the identifier of the first plugin, the first task is sent to the first plugin, and the first plugin is used to call the first tool corresponding to the first task to execute the first task and obtain the first processing result.

4. The method according to claim 3, characterized in that, The display of the first processing result includes: Send the first processing result to the assistant application; The first processing result is displayed through the dialog box interface of the assistant application.

5. The method according to claim 3, characterized in that, The method further includes: The cloud service sends a first response request to the cloud-side server. The first response request carries information that the first task was successfully executed. The first response request is used to generate a first response message. The first response message is received through the cloud service; Send the first reply to the assistant application; The first reply is displayed through the dialog box interface of the assistant application.

6. The method according to claim 3, characterized in that, The method further includes: If the first plugin fails to load, a second response request is sent to the cloud-side server through the cloud service. The second response request carries information about the first plugin failing to load and is used to generate a second response message. The second response message is received via the cloud service; Send the second reply to the assistant application; The second reply is displayed through the dialog box interface of the assistant application.

7. The method according to claim 3, characterized in that, The task list also includes a second task, and obtaining the task list corresponding to the intent includes: The first task and the second task are received asynchronously.

8. The method according to claim 7, characterized in that, The method further includes: Send a second loading request to the second plugin through the second plugin interface; If the second plugin loads successfully, obtain the identifier of the second plugin through the second plugin interface; Based on the identifier of the second plugin, the second task is sent to the second plugin, and the second plugin is used to call the second tool corresponding to the second plugin to execute the second task and obtain the second processing result; The second processing result is sent to the assistant application, and the second processing result and the first processing result are sent asynchronously. The second processing result is displayed through the dialog box interface of the assistant application.

9. The method according to claim 3, characterized in that, The intent includes the intent to remove pedestrians from the image to be processed, and the first processing result is the image after removing pedestrians from the image to be processed.

10. The method according to claim 9, characterized in that, The first task is to eliminate passersby, and the first plugin is an image removal plugin.

11. An image processing method, characterized in that, The method is applied to an electronic device, on which the following are deployed: an assistant application, a media proxy module, and a cloud service. The media proxy module includes: a media proxy service, a plugin interface module, and at least one plugin. The at least one plugin includes a first plugin. The plugin interface module includes at least one first plugin interface. The method includes: The assistant application acquires the image to be processed and the statements entered by the user; The assistant application sends the statement to the media proxy service; The media proxy service sends the statement to the cloud service, which calls the cloud-side server to identify the statement, obtain the user's intent, and generate a task list based on the intent. The task list includes a first task. The media proxy service receives the task list through the cloud service; The media proxy service sends a first loading request to the first plugin through the first plugin interface; When the first plugin is successfully loaded, the media proxy service receives the identifier of the first plugin returned by the first plugin through the first plugin interface. The media proxy service sends the first task to the first plugin based on the identifier of the first plugin; The first plugin calls the first tool corresponding to the first task to execute the first task and obtains the first processing result; The first plugin sends the first processing result to the media proxy service; The media proxy service forwards the first processing result to the assistant application; The assistant application displays the first processing result; The media proxy service sends a first response request to the cloud-side server through the cloud service, and the cloud-side server is also used to generate a first response message based on the first response request. The media proxy service receives the first response through the cloud service; The media proxy service returns the first response to the assistant application; The assistant application displays the first reply.

12. An electronic device, characterized in that, include: Processor, memory, and interface; The processor, the memory, and the interface cooperate with each other to enable the electronic device to perform the method as described in any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1 to 11.

Images