Target person determination method, electronic device and computer readable storage medium

Abstract

The application relates to the technical field of image processing, and provides a target person determination method, an electronic device and a computer readable storage medium. The method comprises the following steps: determining that a sentence input by a user contains a first personal pronoun, wherein the first personal pronoun at least comprises a person name and / or a person title; displaying first prompt information according to the first personal pronoun, wherein the first prompt information is used for prompting visual features of at least one person in a to-be-processed image; receiving first indication information input by the user, wherein the first indication information is used for indicating whether the first prompt information matches the first personal pronoun; and determining a target person indicated by the first personal pronoun in the to-be-processed image according to the first indication information. The above method can quickly determine the target person indicated by the user, and is convenient and fast.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a method for identifying a target person, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, people are able to perform intelligent image removal, intelligent image stitching, and other processing to generate images that meet people's needs.

[0003] For example, when a user takes a photo using an electronic device, if the photo contains an unwanted object, such as an unintentionally captured passerby, the photo will not meet the user's expectations. In this case, the user can use the electronic device to remove the unintentionally captured passerby from the photo, resulting in a photo that retains the intended subject.

[0004] Traditional smart elimination methods can only eliminate targets or passersby manually selected by the user, which is a relatively limited function. Summary of the Invention

[0005] This application provides a method, apparatus, chip, electronic device, computer-readable storage medium, and computer program product for identifying target persons. When a user-inputted sentence contains a person pronoun, it can accurately identify the person indicated by the person pronoun and then perform corresponding processing on the identified person, thus enriching the image processing function.

[0006] In a first aspect, a method for determining a target person in an image is provided, comprising: determining that a statement input by a user contains a first person pronoun, the first person pronoun including at least a person's name and / or title; displaying first prompt information based on the first person pronoun, the first prompt information indicating the visual features of at least one person in the image to be processed; receiving first indication information input by the user, the first indication information indicating whether the first prompt information matches the first person pronoun; and determining the target person indicated by the first person pronoun in the image to be processed based on the first indication information.

[0007] The aforementioned first-person pronouns indicate nouns that refer to people, such as any one or more of the person's name and title. Among them, the person's name can be the person's full name, nickname, or pseudonym, etc., and the person's title can be nouns that indicate the relationship between people, such as father, mother, father, daughter, etc.

[0008] When the user's input contains a person's name and / or title, the electronic device can display first prompt information representing visual features in the image to be processed that may correspond to the person the user wants to indicate, based on the presence of the person's name and / or title in the input. Then, the user determines whether the prompted person is the person they want to indicate based on the first prompt information, and then inputs first prompt information indicating whether the first prompt information matches the first person pronoun. The electronic device can then determine the target person in the image to be processed based on the user's feedback of the first prompt information.

[0009] Alternatively, the aforementioned first-person pronoun can also be the name of a fixed person, such as a well-known person's name, a historical figure, etc.

[0010] In this method, users only need to input any sentence without having to manually mark the person they want to instruct. The electronic device can automatically identify the target person the user wants to instruct. Based on the user's simple feedback, it can accurately determine the target person. The user operation is simple and improves the user experience.

[0011] In some implementations, displaying a first prompt message based on a first person pronoun includes: determining whether a target person identifier corresponding to the first person pronoun exists in a preset database, wherein the preset database includes a first correspondence between at least one person pronoun and at least one person identifier, wherein at least one person pronoun includes the first person pronoun, and at least one person identifier includes the target person identifier; if it exists, determining the target person indicated by the first person pronoun in the image to be processed based on the target person identifier; if it does not exist, displaying a first prompt message based on the first person pronoun.

[0012] If the database contains an identifier for the person whose first pronoun the user mentioned, the target person can be directly identified based on the database information. If the database does not contain an identifier for the person whose first pronoun the user mentioned, the target person can be clarified by asking a question. Based on the user's simple feedback, the target person can be accurately identified. The user operation is simple and improves the user experience.

[0013] In some implementations, determining the target person indicated by the first person pronoun in the image to be processed based on the target person identifier includes: determining the target facial feature information corresponding to the target person identifier based on the target person identifier; and determining the target person matching the target facial feature information from the image to be processed based on the target facial feature information.

[0014] Electronic devices can match the target person from the image to be processed based on the stored facial feature information by comparing facial feature information, making the identification of the target person more accurate.

[0015] In some implementations, the target facial feature information is the target person's avatar. Determining the target facial feature information corresponding to the target person's identifier based on the target person's identifier includes: obtaining the clustering results of a media database, the clustering results including a second correspondence between multiple person identifiers and multiple person avatars, the multiple person identifiers including the target person identifier, and the multiple person avatars including the target person's avatar; determining the person avatar corresponding to the target person identifier as the target person's avatar from the second correspondence; and matching the target person who is the same person as the target person's avatar from at least one person in the image to be processed.

[0016] Electronic devices can cluster images stored in a media database according to different people to obtain image sets (image atlases) corresponding to different people.

[0017] In some implementations, displaying a first prompt message based on a first person pronoun includes: detecting an image to be processed, obtaining at least one visual feature information, the at least one visual feature information representing at least one visual feature of a person in the image to be processed; and displaying the first prompt message, the first prompt message being used to prompt at least one visual feature, the at least one visual feature and the at least one visual feature information corresponding one-to-one.

[0018] Electronic devices can use detection models to detect visual feature information of people in an image to be processed. Visual feature information may include, but is not limited to, features that can be observed by the human eye, such as the style and color of clothing, the color and position of accessories, and hairstyle. Optionally, at least one visual feature can be detected for each person in the image to be processed. Optionally, the electronic device can also detect at least one visual feature for a specific person in the image to be processed.

[0019] In some implementations, the first person is a prominent person in the image to be processed.

[0020] Prominent figures or objects can be clearly visible to the human eye in an image, such as those occupying the center of the image or the largest area. Asking questions about prominent figures can more likely cover the person the user is referring to, and after one round of clarification, the target person can be identified, improving the efficiency of person clarification.

[0021] In some implementations, the first prompt information includes at least a first inquiry. Displaying the first prompt information includes: displaying the first inquiry, the first inquiry including first visual feature information of a first person, the first visual feature information being one of at least one visual feature information; receiving first instruction information input by the user, including: receiving a first reply input by the user; determining the target person indicated by the first person pronoun in the image to be processed based on the first instruction information, including: if the first reply indicates that the first visual feature information matches the first person pronoun, then determining the first person as the target person.

[0022] In some implementations, the method further includes: if a first response indicates that the first visual feature information does not match the first person pronoun, then displaying a second query, the second query being used to characterize the second visual feature information of a second person in the image to be processed, the second person being different from the first person; receiving the second response input by the user; if the second response indicates that the second visual feature information matches the first person pronoun, then determining the second person as the target person.

[0023] If the first visual feature information does not indicate the target person the user wants to point to, the user can enter an affirmative first response, at which point the electronic device can determine that the person indicated by the first visual feature information is the target person.

[0024] If the first visual feature information does not indicate the target person the user intends to point to, the user can input a negative response. The electronic device can then determine that the person indicated by the first visual feature information is not the target person. The electronic device can then proceed to a second round of questioning, displaying a second question to inquire about the second visual features of a second person, and receiving the user's second response regarding the second person's second visual feature information. Based on the second response, the device continues to determine whether the second person corresponding to the second visual feature information is the target person.

[0025] In this method, the electronic device sequentially asks different people about their visual characteristics in multiple rounds, and determines the target person based on the user's response. This achieves the goal of clarifying the target person through dialogue, accurately identifying the target person based on simple user feedback, and is simple for users to operate, thus improving the user experience.

[0026] In some implementations, the method further includes: if a second response indicates that the second visual feature information does not match the first person pronoun; then displaying a third query, the third query being used to characterize the third visual feature information of a third person in the image to be processed, the third person being different from both the first and second persons; receiving a third response input by the user; if the third response indicates that the third visual feature information matches the first person pronoun, then determining the third person as the target person; if the third response indicates that the third visual feature information does not match the first person pronoun, then outputting a second prompt message, the second prompt message being used to prompt the user to re-enter the statement.

[0027] If the target person has not been identified after three rounds of questioning, it indicates that the user's questioning method may be incorrect, making it impossible to accurately identify the target person. At this point, the electronic device can output a second prompt message asking the user to re-enter their question, avoiding multiple rounds of ineffective clarification and saving time.

[0028] In some implementations, the number of at least one visual feature information is multiple, and displaying a first prompt includes: displaying a fourth question, the fourth question including multiple visual feature information, the multiple visual feature information being used to characterize multiple visual features of a first person; receiving a first instruction information input by a user, including: receiving a fourth response input by a user; and determining the target person indicated by the first person pronoun in the image to be processed based on the first instruction information, including: if the fourth response indicates that multiple visual feature information matches the first person pronoun, then determining the first person as the target person.

[0029] Electronic devices can also display multiple visual features of the same person at once, allowing for a clearer and more precise identification of the person being inquired about. When the user responds, it can then be accurately determined whether the person is the target, resulting in more accurate clarification.

[0030] In some implementations, the number of at least one visual feature information is multiple, and displaying a first prompt includes: displaying a fifth inquiry, the fifth inquiry including multiple visual feature information, the multiple visual feature information being used to characterize the visual features of multiple people in the image to be processed, and the multiple visual feature information corresponding one-to-one with the multiple people; receiving a first instruction information input by the user, including: receiving a fifth reply input by the user, the fifth reply being used to indicate a fourth visual feature information among the multiple visual feature information; determining the target person indicated by the first person pronoun in the image to be processed according to the first instruction information, including: if the fourth visual feature information indicated by the fifth reply matches the first person pronoun, determining the person corresponding to the fourth visual feature information as the target person.

[0031] In some implementations, the method further includes: if the fifth response indicates that any one of the multiple visual feature information does not match the first person pronoun, then outputting a third prompt message, which prompts the user to re-enter the statement.

[0032] Electronic devices can also display multiple visual feature information of several different people simultaneously, allowing users to point to multiple people at once. Users can then filter out the visual features of the desired person based on this information and enter a response to confirm. After the response, the electronic device can accurately determine whether the person is the target, making the identification process more precise.

[0033] Furthermore, electronic devices can ask questions based on visual features, guiding users to respond and ask questions based on the visual features that the electronic device can recognize. Compared to continuing to ask about the relationship between people, this improves the success rate and efficiency of clarifying the relationship.

[0034] In some implementations, the method further includes: establishing a correspondence between the target person identifier and the first person pronoun; and adding the correspondence between the target person identifier and the first person pronoun to the first correspondence.

[0035] After identifying a target person, the electronic device can establish and store a correspondence between the target person's identifier and the first person's pronoun in a first correspondence for later retrieval. Based on this, when the user speaks the first person's pronoun again, the electronic device can directly retrieve the target person's identifier based on the new first correspondence, thereby quickly identifying the target person and improving processing efficiency.

[0036] In some implementations, before determining that the user-inputted statement contains a first person pronoun, the method further includes: acquiring the image to be processed and the user-inputted statement; the method further includes: acquiring the intent represented by the statement, the intent corresponding to the first person pronoun, and the intent being an elimination intent; after determining the target person indicated by the first person pronoun in the image to be processed based on the first indication information, the method further includes: performing an elimination task corresponding to the intent on the image to be processed based on the target person to obtain a result image; and displaying the result image.

[0037] When a user inputs a command to eliminate a first-person pronoun, the electronic device can quickly and accurately identify the target person and complete the elimination task. Based on simple user feedback, it precisely targets and eliminates the desired person, simplifying the process and enhancing the user experience. Displaying the eliminated result as an image allows users to visually understand the outcome, further improving the user experience.

[0038] In a second aspect, a target identification device is provided, comprising a unit consisting of software and / or hardware, the unit being used to perform any one of the methods described in the first aspect.

[0039] Thirdly, embodiments of this application provide a chip including a processor; the processor is configured to read and execute a computer program stored in a memory to perform any one of the methods described in the first aspect.

[0040] Optionally, the chip also includes a memory, which is connected to the processor via a circuit or wire.

[0041] Alternatively, the chip may further include a communication interface.

[0042] Fourthly, an electronic device is provided, comprising: a processor, a memory, and an interface; the processor, memory, and interface cooperate with each other to enable the electronic device to perform any one of the methods described in the first aspect.

[0043] Fifthly, an electronic device is provided, which includes any one of the chips described in the third aspect.

[0044] In a sixth aspect, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, the processor performs any one of the methods described in the first aspect.

[0045] In a seventh aspect, a computer program product is provided, the computer program product comprising: computer program code, which, when executed on an electronic device, causes the electronic device to perform any one of the methods described in the first aspect. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the structure of a terminal device 100 provided in an embodiment of this application;

[0047] Figure 2 This is a software structure block diagram of the terminal device 100 provided in the embodiments of this application;

[0048] Figure 3 This is a schematic diagram of an example of a dialog elimination interface provided in an embodiment of this application;

[0049] Figure 4 This is another example of a dialog box elimination interface provided in the embodiments of this application;

[0050] Figure 5 This is another example of a dialog box elimination interface provided in the embodiments of this application;

[0051] Figure 6 This is another example of a dialog box elimination interface provided in the embodiments of this application;

[0052] Figure 7 This is another example of a dialog box elimination interface provided in the embodiments of this application;

[0053] Figure 8 This is another flowchart of a dialogue elimination method provided in the embodiments of this application;

[0054] Figure 9 This is another example of a dialog elimination process interface provided in the embodiments of this application;

[0055] Figure 10 This is a software architecture diagram of an example image processing method provided in an embodiment of this application;

[0056] Figure 11 This is a flowchart of an example image processing method provided in an embodiment of this application;

[0057] Figure 12 This is a flowchart of a target person determination method provided in an embodiment of this application;

[0058] Figure 13 This is another example of an interface diagram illustrating the clarification of interpersonal relationships provided in this application embodiment;

[0059] Figure 14 This is a schematic diagram of an interface for clarifying interpersonal relationships provided in an embodiment of this application;

[0060] Figure 15 This is another example of an interface diagram illustrating the clarification of interpersonal relationships provided in this application embodiment;

[0061] Figure 16 This is another flowchart illustrating a method for clarifying interpersonal relationships provided in this application embodiment;

[0062] Figures 17A-17D This is a schematic diagram of an image processing method including the process of clarifying interpersonal relationships provided in an embodiment of this application;

[0063] Figure 18 This is a schematic diagram of the structure of a target person identification device provided in an embodiment of this application. Detailed Implementation

[0064] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0065] Hereinafter, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include one or more of that feature.

[0066] The target person determination method provided in this application embodiment can be applied to terminal devices such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). This application embodiment does not impose any restrictions on the specific type of terminal device.

[0067] For example, Figure 1This is a schematic diagram of the structure of a terminal device 100 provided in an embodiment of this application. The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0068] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0069] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0070] The controller can serve as the central nervous system and command center of the terminal device 100. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0071] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0072] In some implementations, the processor 110 may include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0073] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may also adopt different interface connection methods or a combination of multiple interface connection methods as described in the above embodiments.

[0074] Terminal device 100 implements display functions through a GPU, display screen 194, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0075] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a minimized display, a microLED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, terminal device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0076] Terminal device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0077] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some implementations, the ISP can be integrated into the camera 193.

[0078] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the terminal device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0079] A digital signal processor (DSP) is used to process digital signals. Besides digital image signals, it can also process other digital signals. For example, when terminal device 100 selects a frequency, the DSP performs Fourier transforms on the frequency energy.

[0080] Video codecs are used to compress or decompress digital video. Terminal device 100 may support one or more video codecs. Thus, terminal device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0081] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in terminal devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0082] Terminal device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0083] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0084] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The terminal device 100 can listen to music or make hands-free calls through the speaker 170A.

[0085] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the terminal device 100 answers a phone call or voice message, the receiver 170B can be brought close to the listener's ear to hear the voice.

[0086] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Terminal device 100 may be equipped with at least one microphone 170C. In some embodiments, terminal device 100 may be equipped with two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, terminal device 100 may be equipped with three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0087] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0088] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch screen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of terminal device 100, in a different position than display screen 194.

[0089] The software system of terminal device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of terminal device 100.

[0090] Figure 2 This is a software structure block diagram of the terminal device 100 according to an embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer. The application layer may include a series of application packages.

[0091] like Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, SMS, and smart voice assistant.

[0092] For example, a smart voice assistant can be called YOYO Assistant, or simply YOYO. A smart voice assistant can interact with users, understand their needs, and provide corresponding services based on those needs.

[0093] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0094] like Figure 2 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0095] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0096] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0097] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0098] The phone manager is used to provide communication functions for terminal device 100. For example, it manages call status (including connection, hang-up, etc.).

[0099] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0100] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of download completion or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating the device, and flashing indicator lights.

[0101] The application framework layer can also configure artificial intelligence agents (AI agents) to automate complex tasks. Optionally, there can be one or more AI agents. Different AI agents can handle different types of complex tasks.

[0102] Optionally, the intelligent agent can run in the same process as a system-level application (such as Brain) to interact with the intelligent voice assistant and process user commands received by the intelligent voice assistant.

[0103] Correspondingly, the intelligent agent can provide an intelligent agent lifecycle management service, which can also interact with other services. For example, the intelligent agent lifecycle management service can interact with perception services to obtain perceived data, such as the user's physiological state, the external environmental state, etc. Optionally, the functional module that provides the intelligent agent lifecycle management service can also be called an intelligent agent management module.

[0104] Optionally, the agent can also interact with cloud-based servers to invoke large cloud-based models to handle complex tasks.

[0105] The Android runtime consists of core libraries and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system.

[0106] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0107] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0108] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0109] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0110] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0111] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0112] A 2D graphics engine is a graphics engine for 2D drawing.

[0113] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0114] For ease of understanding, the following embodiments of this application will be described using the following methods: Figure 1 and Figure 2 Taking the terminal device with the structure shown as an example, the technical solution provided by the embodiments of this application will be specifically described in conjunction with the accompanying drawings and application scenarios.

[0115] Taking the intelligent voice assistant YOYO as an example, YOYO can interact with users through natural language. For example, users can say a wake word, such as "Hello YOYO," to wake up YOYO, or they can bring the electronic device close to their mouth to wake up YOYO using breath.

[0116] When YOYO is activated, its program interface is displayed on the electronic device's screen. Optionally, YOYO's program interface can be displayed on the taskbar on the desktop. Users can enter commands in the taskbar, which can be in text or voice format. When in voice format, YOYO can recognize the voice, generate text, and obtain the user's command. YOYO can also display a dialog interface after being activated by the user. Users can enter the commands they need to process in the dialog box.

[0117] For example, when a user wants to intelligently edit a picture, they can input the image and a sentence in the intelligent voice assistant's dialog box to execute the AI ​​editing function. Next, we will combine... Figures 3 to 7The relevant interfaces of the AI ​​editing function provided in the embodiments of this application are described.

[0118] In some implementations, users can trigger the image processing function of the intelligent voice assistant in various ways to edit an image. The following will combine... Figures 3 to 7 This is illustrated by example:

[0119] The following is combined Figure 3 The first method is illustrated by example:

[0120] like Figure 3 As shown in (a), the electronic device displays interface 21, which can display icons for multiple applications, assuming it includes an icon 211 for the gallery application. Optionally, interface 21 can be referred to as the main interface or desktop. Subsequently, the electronic device receives an operation applied to the icon 211 of the gallery application in interface 21, and in response to this operation, as shown in (a). Figure 3 As shown in (b), the electronic device can display an interface 22 of a gallery application. Interface 22 can display thumbnails of each of multiple images, assuming it includes a thumbnail 221 of image a. Subsequently, the electronic device receives an operation applied to thumbnail 221 of image a, and in response to this operation, as shown in (b). Figure 3 As shown in (c), the electronic device can display an interface 23 of a gallery application. Interface 23 can display image a, as well as multiple controls. For example, these multiple controls may include share, favorite, AI edit, delete, more, etc. Then, the electronic device receives an operation applied to image a and responds to that operation as follows: Figure 3 As shown in (d), the electronic device can display the interface 24 of the gallery application. The interface 24 can display a full-screen view of image a. Optionally, the interface 24 can be referred to as a large page of image a. For example, in this embodiment, the operation received by the electronic device acting on the content displayed on the electronic device (such as icons, thumbnails, images, controls, or options) can be a voice operation, a gesture operation (such as clicking, long-pressing, or swiping), or a motion-sensing operation (such as device shaking or air gestures), etc., and this embodiment does not limit this.

[0121] like Figure 3As shown in (d) above, in some embodiments, when the electronic device displays interface 24, it can automatically display the input box 241 of the intelligent voice assistant and multiple AI editing controls on interface 24. In other embodiments, the user can wake up the intelligent voice assistant while the electronic device displays interface 24. After the intelligent voice assistant is woken up, the electronic device can display the input box 241 of the intelligent voice assistant and multiple AI editing controls on interface 24. For example, the user can wake up the intelligent voice assistant using any of the following methods: voice wake-up, breath wake-up, button wake-up, navigation bar wake-up, etc. For example, the multiple AI editing controls may include a pedestrian removal control 242, a one-click skin beautification control 243, and an intelligent removal control 244. The pedestrian removal control 242 is used to trigger the removal of pedestrians in the image. The one-click skin beautification control 243 is used to trigger skin beautification of people in the image. The intelligent removal control 244 is used to trigger the removal of specific targets in the image. Specific targets may include one or more of people, animals, plants, objects, text, watermarks, etc. In this embodiment, "passersby" can refer to other figures in the image besides the main subject, such as pedestrians in the background, people passing by, or other individuals in the scene who do not directly interact with the main subject. The main subject typically refers to the central figure in the image, usually located in the center or a prominent position within the image.

[0122] In this scenario, if a user wants to remove passersby from image a, they can operate the "Remove Passersby" control 242. Upon receiving the operation on the "Remove Passersby" control 242, the electronic device can, in response to this operation, perform passersby removal processing on image a, and can display the chat window 26 of the intelligent voice assistant. For example, as... Figure 3 As shown in (e), the chat window 26 can be displayed on the interface 24 in a floating window or similar manner, and the interface 24 and the chat window 26 together constitute the interface 25. When the chat window 26 is displayed on the interface 24, the chat window 26 does not completely cover the interface 24. The chat window 26 may include an input box 261 and a dialogue area 262. Optionally, during the process of removing pedestrians from image a, the electronic device may display the user message "Remove pedestrians" and the intelligent voice assistant message "Image generation in progress" in the dialogue area 262 to remind the user that pedestrians in image a are being removed.

[0123] Assuming the electronic device obtains image b after performing pedestrian removal processing on image a, the electronic device can display a thumbnail of image b in dialog area 262. For example, as... Figure 3As shown in (e), after completing the pedestrian removal process for image a, the electronic device may no longer display the intelligent voice assistant message "Image generation in progress," but instead display the intelligent voice assistant messages "Completed, you can save directly or perform custom editing" and "Thumbnail of image b." Optionally, as... Figure 3 As shown in (e), the electronic device can display multiple controls while showing a thumbnail of image b, such as a sharing control 263, a save control 264, a custom editing control 265, and an AI retouching control 266. The sharing control 263 is used to share image b. The save control 264 is used to save image b locally. The custom editing control 265 is used for custom editing. The AI ​​retouching control 266 is used for AI retouching of image b. Users can manipulate image d using these controls.

[0124] The above text Figure 3 In the first approach provided by the embodiment, users can decide whether image processing is needed while viewing images in the gallery application, and instantly initiate image processing functions via a smart voice assistant when image processing is required. This improves ease of operation and makes image processing smoother and more intuitive.

[0125] The following is combined Figure 4 and Figure 5 The second method is illustrated by example:

[0126] like Figure 4 As shown in (a), the electronic device displays interface 31, which can be any interface displaying an image. Here, it is assumed that interface 31 is a chat interface of an instant messaging application, where the user and / or their friend sends an image c to each other. Subsequently, the electronic device receives an operation applied to image c (such as a long press), and in response to this operation, the electronic device can display a thumbnail 321 of image c. For example, as... Figure 4 As shown in (b), thumbnail 321 can be displayed on interface 31, and interface 31 and thumbnail 321 together constitute interface 32. When thumbnail 321 is displayed on interface 31, it can obscure the content at the corresponding position in interface 31. The user can drag thumbnail 321, and thumbnail 321 can move within interface 32 as the user drags it.

[0127] After that, as Figure 4As shown in (c), when the electronic device detects that the thumbnail 321 has been dragged to near the edge of the screen, it can display interface 33. Interface 33 can display the thumbnail 321, screen 331, and service recommendation bar 332. Optionally, screen 331 can display the content of the scaled-down interface 31. For example, screen 331 is used to create the visual effect of interface 31 being pushed inward on the screen. Service recommendation bar 332 is used to display icons of one or more services, which may include the icon 333 of the intelligent voice assistant. Users can drag the thumbnail 321 to the position of an icon in service recommendation bar 332 according to their own needs and then release it to transmit image c to the service corresponding to the icon and start the corresponding service function. It should be noted that the service described in this application embodiment is a program used to implement a specific function, such as an application, applet, quick app (also known as an installation-free application), or certain functional modules of an application, etc., and this application embodiment does not limit this.

[0128] Subsequently, after detecting that thumbnail 321 has been dragged to and released from the location of the smart voice assistant icon 333, the electronic device can transmit image c to the smart voice assistant and display the smart voice assistant's chat window 35. For example, as... Figure 5 As shown in (a), the chat window 35 can be displayed on the interface 31 in a floating window or similar manner, and the interface 31 and the chat window 35 together constitute the interface 34. When the chat window 35 is displayed on the interface 31, the chat window 35 does not completely cover the interface 31. The chat window 35 may include an input box 351, multiple AI editing controls, and a dialogue area 352. The multiple AI editing controls may include a passerby removal control 353, a one-click skin beautification control 354, a smart removal control 355, etc. The functions of the multiple AI editing controls have been described above. Figure 3 The example is described below and will not be repeated here. The user message "thumbnail of image c" can be displayed in the dialog area 352.

[0129] In this scenario, if a user wants to remove pedestrians from image c, they can operate the pedestrian removal control 353. Upon receiving the operation on the pedestrian removal control 353, the electronic device can perform pedestrian removal processing on image c in response to the operation. Optionally, during the pedestrian removal processing of image c, the electronic device can display the user message "Remove Pedestrians" and the intelligent voice assistant message "Image Generating" in the dialog area 352 to remind the user that pedestrians are being removed from image c.

[0130] Assuming the electronic device obtains image d after performing pedestrian removal processing on image c, the electronic device can display a thumbnail of image d in dialog area 352. For example, as... Figure 5As shown in (b), after completing the pedestrian removal process for image c, the electronic device may no longer display the intelligent voice assistant message "Image generation in progress," but instead display the intelligent voice assistant message "Completed, you can save directly or perform custom editing" and "Thumbnail of image d." Optionally, as... Figure 5 As shown in (b) above, the electronic device can display multiple controls simultaneously with the thumbnail of image d, such as a share control 361, a save control 362, a custom edit control 363, and an AI image editing control 364. The functions of these multiple controls have been described above. Figure 3 The examples are illustrated below and will not be repeated here. Users can manipulate image d using these controls.

[0131] The above text Figures 4 to 5 In the second method provided in the embodiment, users can quickly transmit images from any interface to the intelligent voice assistant for processing. This improves operational flexibility and convenience.

[0132] The following is combined Figure 6 The third method is illustrated by example:

[0133] The electronic device can display interface 41, which can be any interface. The user can wake up the intelligent voice assistant while the electronic device displays interface 41. After the intelligent voice assistant is woken up, the electronic device can display the input box 421 of the intelligent voice assistant. For example, such as... Figure 6 As shown in (a), the input box 421 can be displayed on the interface 41 in a floating window or similar manner. The interface 41 and the input box 421 can together constitute the interface 42. When the input box 421 is displayed on the interface 41, the input box 421 does not completely cover the interface 41. The input box 421 may include an add control 422 and an input mode switching control 423. The add control 422 is used to add images or files. The input mode switching control 423 is used to switch the input mode between voice input and text input.

[0134] In this situation, if the user wants to perform image processing, they can operate the added control 422. The electronic device receives the operation applied to the added control 422 and responds to the operation as follows: Figure 6 As shown in (b), the electronic device can display multiple controls in the input box 421. These controls may include an image control 424, a camera control 425, a file control 426, etc. Users can select a local image for processing using the image control 424, or capture an image for processing using the camera control 425. The following explanation uses the image control 424 as an example to illustrate how to input an image to the intelligent voice assistant. For the method of inputting an image to the intelligent voice assistant using the camera control 425, please refer to the following text. Figure 7The relevant descriptions in the embodiments will not be repeated here.

[0135] Assuming the electronic device receives an operation applied to the image control 424, in response to that operation, such as Figure 6 As shown in (c), the electronic device can display an interface 43 of a gallery application. Interface 43 can display thumbnails of each of multiple images, assuming it includes a thumbnail 431 of image e. Subsequently, the electronic device receives an operation applied to thumbnail 431 of image e, and in response to this operation, as shown in (c). Figure 6 As shown in (d), the electronic device can display interface 44. Interface 44 can display image e, as well as a return control 441 and a completion control 442. Upon receiving an operation on the completion control 442, the electronic device can, in response to this operation, display the chat window 46 of the intelligent voice assistant. For example, as... Figure 6 As shown in (e), the chat window 46 can be displayed on the interface 41 in a floating window or similar manner. The interface 41 and the chat window 46 together constitute the interface 45. When the chat window 46 is displayed on the interface 41, the chat window 46 does not completely cover the interface 41. The chat window 46 may include an input box 461, multiple AI editing controls, and a dialogue area 462. The multiple AI editing controls may include a passerby removal control 463, a one-click skin beautification control 464, and a smart removal control 465, etc. The functions of the multiple AI editing controls have been described above. Figure 3 The example is described below and will not be repeated here. The user message "thumbnail of image e" can be displayed in the dialog area 462.

[0136] After that, as Figure 6 As shown in (e) to (f), the user can use the intelligent voice assistant to remove pedestrians in image e to obtain image f. Figure 6 The processes shown in (e) to (f) are similar to Figure 5 The processes shown in (a) to (b) are similar; please refer to the above text for details. Figure 5 The relevant descriptions in the embodiments will not be repeated here.

[0137] The above text Figure 6 In the third method provided in this embodiment, the user can activate the intelligent voice assistant from any interface. The intelligent voice assistant serves as a centralized operation window, allowing users to easily add and process images using the provided controls. The process is simple and improves user efficiency.

[0138] The following is combined Figure 7 The fourth method is illustrated by example:

[0139] The electronic device can display interface 51, which can be any interface. The user can wake up the intelligent voice assistant while the electronic device displays interface 51. After the intelligent voice assistant is woken up, the electronic device can display the input box 521 of the intelligent voice assistant. For example, such as... Figure 7 As shown in (a), the input box 521 can be displayed on the interface 51 in the form of a floating window or the like, and the interface 51 and the input box 521 can together constitute the interface 52. When the input box 521 is displayed on the interface 51, the input box 521 does not completely cover the interface 51.

[0140] The electronic device receives an operation applied to the input box 521 (such as a swipe-up operation starting from the input box 521), and responds to the operation as follows: Figure 7 As shown in (b), the electronic device can display an interface 53 for a smart voice assistant. Interface 53 can display an input box 531 and multiple options, including a YOYO Assistant option 532, recent options 533, and an AI agent option 534. The YOYO Assistant option 532 displays the dialogue page of the smart voice assistant. The recent options 533 display the AI ​​agents (artificial intelligence agents) recently used by the user. The AI ​​agent option 534 displays one or more AI agents that the user can use. An AI agent is an intelligent program with autonomous perception, decision-making, and execution capabilities, which can perform specific tasks through intent understanding, task decomposition, and resource scheduling.

[0141] Suppose the electronic device receives an operation applied to agent option 534, and in response to that operation, such as Figure 7 As shown in (c), the electronic device can display interface 54. Interface 54 can display multiple agent icons, which may include an icon 541 for an image editing agent, an icon 542 for a video editing agent, etc. The image editing agent is used to implement functions related to image editing. The video editing agent is used to implement functions related to video editing. The user can select the desired agent from the agents provided in interface 54. Optionally, as... Figure 7 As shown in (c), a search box 543 can also be displayed in the interface 54, through which the user can search for the desired intelligent agent.

[0142] Suppose the electronic device receives an operation on icon 541 applied to the image editing agent, and in response to that operation, such as Figure 7As shown in (d), the electronic device can display the interface 55 of the image editing intelligent agent. Interface 55 can be a chat window for an intelligent voice assistant, displaying an input box 551, multiple AI editing controls, and a dialogue area 552. These multiple AI editing controls may include a pedestrian removal control 553, a smart removal control 554, a one-click skin beautification control 555, a smart background removal control 556, and a photo ID generation control 557. The smart background removal control 555 is used to trigger the separation of the subject (such as a person, animal, or object) in the image. The photo ID generation control 557 is used to trigger the generation of a photo ID for the person in the image. The functions of the pedestrian removal control 553, the smart removal control 554, and the one-click skin beautification control 555 have been described above. Figure 3 The examples are illustrated below and will not be repeated here.

[0143] In this situation, if the user wants to perform image processing, they can operate the pedestrian removal control 553. The electronic device receives the operation applied to the pedestrian removal control 553 and responds to the operation as follows: Figure 7 As shown in (e), the electronic device can display the user message "Remove passersby," the intelligent voice assistant message "Please upload a picture containing people," a camera control 558, and an image control 559 in the dialogue area 552. The user can take a picture using the camera control 558 and process it, or select a local image using the image control 559 and process it. The following explanation uses the method of inputting an image to the intelligent voice assistant via the camera control 558 as an example. For the method of inputting an image to the intelligent voice assistant via the image control 559, please refer to the above text. Figure 6 The relevant descriptions in the embodiments will not be repeated here.

[0144] Assuming the electronic device receives an operation on the camera control 558, it responds to the operation as follows: Figure 7 As shown in (f), the electronic device can display the camera application interface 56. Interface 56 is a shooting interface, which may include a shooting preview window 561 and a shooting control 562. After the user determines that the shooting screen in the shooting preview window 561 meets their needs, they can operate the shooting control 562. Upon receiving the operation applied to the shooting control 562, the electronic device, in response to the operation, can take a picture, obtaining image g, and, as shown... Figure 7As shown in (g), the electronic device can display interface 57. Interface 57 can display image g, a return control 571, and a completion control 572. Upon receiving an operation on the completion control 572, the electronic device can perform pedestrian removal processing on image g in response to this operation. Optionally, during pedestrian removal processing on image g, the electronic device can display the user messages "thumbnail of image g" and "remove pedestrians" in the dialogue area 552 of interface 55, and no longer display the photo control 558 and image control 559. Optionally, the electronic device can also display the intelligent voice assistant message "Image generation in progress" in the dialogue area 552 to remind the user that pedestrians in image g are being removed.

[0145] Assuming that the electronic device obtains image h after performing pedestrian removal processing on image g, the electronic device can display a thumbnail of image h in the dialog area 552. For example, as... Figure 7 As shown in (h), after the electronic device completes the pedestrian removal process for image g, it may no longer display the intelligent voice assistant message "Image generation in progress," but instead display the intelligent voice assistant message "Completed, you can save directly or perform custom editing" and "Thumbnail of image h." Optionally, as... Figure 7 As shown in (h), the electronic device can display multiple controls simultaneously with a thumbnail of image h, such as a share control, a save control, a custom edit control, and an AI image editing control. The functions of these multiple controls have been described above. Figure 3 The examples are illustrated below and will not be repeated here. Users can manipulate image h using these controls.

[0146] The above text Figure 7 In the fourth approach provided in the embodiment, by offering different intelligent agents, users can select the required intelligent agent for corresponding professional processing according to their own needs. For example, users can use the image editing intelligent agent to concentrate on image processing and avoid being disturbed by other functions or applications.

[0147] One point to note is that the above text Figures 3 to 7 This embodiment uses the example of triggering pedestrian removal processing on an image via a pedestrian removal control provided by a smart voice assistant for illustrative purposes. In practical applications, other controls provided by the smart voice assistant, such as smart removal controls, one-click skin beautification controls, smart background removal controls, and ID photo generation controls, can also be used to trigger other image processing. This embodiment does not limit this approach. Furthermore, image processing can be triggered not only by controls provided by the smart voice assistant but also by entering user commands in the smart voice assistant's input box. For example, a user can enter the command "remove pedestrians" in the smart voice assistant's input box via voice or text to trigger pedestrian removal processing on the image.

[0148] Another point to note is that the above text Figures 3 to 7 This embodiment only uses the functions provided by the pedestrian removal control, intelligent removal control, one-click skin beautification control, intelligent background cutout control, and ID photo generation control as examples for illustrative purposes. In practical applications, intelligent voice assistants can also provide other image processing functions. For example, they can add special effects or borders to images; add specific elements to images; and provide some simple image editing functions, such as cropping, filters, mosaic, and background blurring.

[0149] The embodiments in this application are only as described above. Figures 3 to 7 The process described in the embodiment is used as an example to illustrate the process of triggering the image processing function of the intelligent voice assistant. Figures 3 to 7 The process described in the embodiments does not constitute a specific limitation on the implementation of the embodiments of this application. In practical applications, the image processing function of the intelligent voice assistant can also be triggered in other ways than the four methods described above, and the embodiments of this application do not limit this.

[0150] The aforementioned intelligent voice assistant messages can also be referred to as replies.

[0151] It should be noted that the embodiments in this application are only those described above. Figures 3 to 7 The process described in the embodiment is used as an example to illustrate the image editing process based on an intelligent voice assistant. Figures 3 to 7 The process described in the embodiments does not constitute a specific limitation on the implementation of the embodiments of this application. In practical applications, the image editing process based on the intelligent voice assistant can also be implemented in other ways different from the above methods.

[0152] The process described above, in which a user inputs their voice or command into a dialog box of an intelligent voice assistant and displays an AI-edited image and response through the dialog box, can be called a dialogue format.

[0153] The following section, with reference to the accompanying diagrams, provides a detailed explanation of how electronic devices perform image processing in a dialog-like manner.

[0154] For ease of description, the following text will use YOYO as an example of a smart voice assistant. Figure 8 An example image processing flow provided in an embodiment of this application is shown, including:

[0155] S701, YOYO obtains the image and statement to be processed.

[0156] After opening YOYO, users can operate the YOYO interface, such as sending images and statements to be processed in the YOYO dialog box. The image to be processed is the image the user wants to edit. Optionally, users can click "Add Control" in the YOYO interface to select an image from their gallery and send it to the YOYO dialog box. Alternatively, YOYO can receive images sent (or shared) by other applications. Users can also click "Add Control" in the YOYO interface and then select the "Capture Control" to capture an image on-site. This application does not limit the source of the image to be processed.

[0157] The aforementioned statements can be voice input by the user, text input by the user manually using the keyboard, text converted from voice input, or pre-recorded audio or text selected by the user from other files by clicking the "Add Control" button. This application does not limit the method or type of statements input by the user.

[0158] If YOYO receives a speech, it can convert the speech into text and then send it to the speech engine. If YOYO receives a text, it can send the text directly to the speech engine, or perform preliminary processing, such as deleting invalid or duplicate characters, before sending it to the speech engine. Optionally, the speech can be a single sentence received at once, or multiple sentences received within a certain time period.

[0159] Optionally, the user can send the image to be processed first in the YOYO dialog box, and then enter a statement and send it; alternatively, the user can also enter a statement first and send it, and then send the image to be processed. This application embodiment does not limit the order in which the image to be processed and the statement are sent. When the electronic device receives the image to be processed and the statement through YOYO within a certain time period, it can determine that the semantics expressed by the statement represent a processing intention for the image to be processed. If the time interval between receiving the image to be processed and the statement is relatively long, it indicates that the semantics expressed by the user's input statement may not represent a processing intention for the image to be processed, and therefore no processing is required, or the system can wait for the next statement or image to be sent.

[0160] For example, after receiving the image and statement to be processed, YOYO can, as follows: Figure 9 As shown in (a), the image to be processed (e.g., picture 1) and the statement (e.g., eliminate passersby) are displayed in the dialog box. Figure 9 Example (a) is given in text form using statements.

[0161] S702 and YOYO send the image and text to be processed to the image editing agent.

[0162] After receiving the image and statement to be processed, YOYO can send them to the image editing agent, which will then process the image according to the intent expressed in the user's statement.

[0163] Optionally, YOYO can also send the statement to the voice cloud via the voice engine to identify the user's intent. After receiving the intent returned by the cloud, it can call the image editing agent to process the intent accordingly.

[0164] S703: The image editing agent processes the image to be processed according to the intent expressed in the statement, and obtains the result image and the response.

[0165] Optionally, the image editing agent can also interact with a voice cloud (or other cloud-side servers) to send received statements from the voice cloud, identify the user's intent expressed in the statements, and process the image to be processed accordingly based on the user's intent.

[0166] During the processing of the image, YOYO's dialog box can also display the current processing progress, for example... Figure 9 The process of "intelligently identifying pedestrians" and "intelligently eliminating pedestrians" is shown in (b) and (c) in the figure; optionally, the processing progress can also be displayed, such as 10%, 50%, etc.

[0167] S704: The image editing agent returns the resulting image and response to YOYO.

[0168] After processing the image to be processed, the image editing agent obtains the processed image. For example, the processed image could be Image 2 after removing passersby, and the image editing agent could then send Image 2 to YOYO.

[0169] Optionally, the image editing agent can also return corresponding responses to YOYO. For example, when the image editing agent successfully removes the passerby from image 1, the response could be "Removal successful," or "Completed. You can save directly or perform custom editing."

[0170] S705 and YOYO will display the processed images and replies.

[0171] YOYO can display the processed image returned in a dialog box, and can also display a reply in the dialog box, such as... Figure 9 As shown in (d) in the figure.

[0172] Users can intuitively understand the processing process and the results through the content displayed by YOYO, realizing dialogue-style interaction and improving the user experience.

[0173] Optionally, the processed image may also display the source information, such as "Content provided by AI"; alternatively, the processed image may also display one or more of the following: a share control, a save control, and an edit control. When the user clicks the share control, the electronic device can share the processed image to other applications via YOYO, such as chat applications, email applications, etc., and can also share it to other devices, such as printing devices; when the user clicks the save control, the electronic device can save the processed image to the gallery; when the user clicks the edit control, the electronic device can edit the processed image based on the user's operation.

[0174] Figure 8 and Figure 9 The diagram illustrates the process and interface of image processing via a dialog-based image editing agent. However, when the user's input includes relationships between people, the image editing agent can also identify these relationships within the image, thus achieving precise image processing.

[0175] Next, combine Figure 10 The software architecture diagram shown Figure 11 and Figure 12 The flowchart shown illustrates the implementation process of an intelligent agent processing images when the statement contains relationships between people.

[0176] The electronic device's application layer includes YOYO and a gallery. Agents are also set up at the application framework layer; here, we'll illustrate this with examples of image editing and video editing agents. Optionally, other agents may also exist. For details on the architecture and interaction methods of the video editing and other agents, please refer to the section on the implementation principles of the image editing agent in this document; they will not be repeated here.

[0177] First combine Figure 10 This section provides an overview of the process for intelligent elimination based on dialogue. For example... Figure 10 As shown, after YOYO receives the image to be processed and the user's input statement, it interacts with the voice cloud through its voice engine to perform intent recognition on the statement, obtaining the intent and the identifier of the corresponding agent. YOYO sends the intent and agent identifier returned by the voice cloud to the agent management module, which then distributes different intents to different agents according to their corresponding agent identifiers. For example, the agent management module sends the intent to eliminate everyone except Zhang San to the image editing agent. YOYO also sends the image to be processed to the image editing agent through the agent management module. The image editing agent uses its own task engine to send the intent to the cloud-side server (e.g., the server deployed by the task decomposition model) for task decomposition, obtaining the corresponding task list. Figure 10The task decomposition model is deployed on a speech cloud as an example. In actual implementations, the task decomposition model can also be deployed on another server. The speech cloud sends the intent obtained from the recognized speech to the cloud-side server where the task decomposition model is deployed for task decomposition, resulting in a task list. The image editing agent can then, after obtaining the task list, call the corresponding processing tool based on the processing tool for each task in the list.

[0178] When the task list includes elimination tasks, the task engine can send the elimination task to the dialogue elimination module. If the user's input also contains pronouns, the character relationship clarification module is invoked to clarify the character indicated by the pronouns, thus obtaining the target character the user wants to indicate. During the character relationship clarification process, the learning and memory platform and media database can be queried to determine the target character. Detailed information on the character relationship clarification process can be found in the relevant embodiments below and will not be elaborated here.

[0179] After identifying the target person, the image editing agent can send the removal task to the visual cloud via the visual SDK interface and the visual platform. The visual cloud will then perform intelligent removal and retrieve the removed image. The image editing agent can also return the removed image to YOYO for display via the agent management module.

[0180] The image editing agent can also generate corresponding responses by calling the voice cloud based on the results of visual cloud elimination, and then return the responses to YOYO for display through the agent management module.

[0181] In addition, YOYO can compress and upload images to the storage cloud after receiving them. The voice cloud and vision cloud can access images stored on the storage cloud, eliminating the need to retrieve them from the edge each time. The storage cloud, or Object Storage Service (OBS) cloud, can store large amounts of data for provisioning to other cloud-side servers and edge-side devices.

[0182] Next, combined Figure 11 The flowchart shown describes in detail the process of intelligent elimination based on dialogue. Figure 11 Described from the perspective of the electronic device side (end side), the method includes:

[0183] S1001, YOYO obtains the image to be processed and the statement.

[0184] The method by which YOYO obtains the image and statement to be processed can be found in the relevant description of S701 above. As long as YOYO receives the image and statement to be processed, it is sufficient.

[0185] S1002, YOYO sends the statement to the speech engine.

[0186] After YOYO obtains the sentence, it can send the sentence to the speech engine. Optionally, when the sentence is text, YOYO can send the text to the speech engine; when the sentence is speech, YOYO can convert the speech into text and then send the text to the speech engine.

[0187] Optionally, YOYO can also remove duplicates from the input before sending the deduplicated version to the speech engine. For example, when a user inputs, "Eliminate pedestrians, um, that is, remove the pedestrians in the picture," YOYO determines that there are duplicates in the user's input. YOYO first removes duplicates from the user's input, obtaining the deduplicated version "Eliminate pedestrians in the picture," and then sends the deduplicated version to the speech engine.

[0188] S1003, The speech engine obtains at least one intent corresponding to the statement.

[0189] The voice engine may include an intent control center, which can interact with the voice cloud, send the sentence to the voice cloud, and then the voice cloud performs intent recognition on the sentence. After obtaining the intent corresponding to the sentence, the recognized intent is returned to the intent control center.

[0190] For example, a user enters the following statement in the YOYO dialog box: "Remove pedestrians, then beautify the portrait." The intent control center can then send this statement to the voice cloud. The voice cloud will recognize two intents corresponding to this statement: one is the intent to remove pedestrians, and the other is the intent to beautify the portrait. The voice cloud will then return these two recognized intents to the intent control center.

[0191] For example, if a user enters the following statement in the YOYO dialog box: "Remove passersby and generate a short video of dancing," the voice engine can call the intent control center to recognize the statement and obtain two intents: one is the intent to remove passersby, and the other is the intent to generate a dancing video. The voice cloud then returns these two recognized intents to the intent control center.

[0192] When a user's statement contains an intent, the intent controller can acquire that intent. For example, if the user inputs "Remove the passersby from the picture," the intent controller can acquire the intent to remove the passersby. As another example, if the user inputs "Keep only Zhang San in the picture," the intent controller can acquire the intent to remove everyone except Zhang San.

[0193] When a statement contains multiple user intents, the intent controller can retrieve multiple intents. For example, if a user inputs the statement "Remove passersby from the image and then beautify the portrait," the intent controller can retrieve two intents: the intent to remove passersby and the intent to beautify the portrait.

[0194] Optionally, intent control can also be implemented locally on the electronic device, calling a local intent recognition model to identify the intent corresponding to the statement. For example, if the electronic device has an intent recognition model deployed locally, intent control can send the statement to the local intent recognition model to identify the intent and obtain the recognition result fed back by the intent recognition model, i.e., at least one intent.

[0195] Alternatively, the electronic device may first call the local intent recognition model to identify the intent contained in the statement. If the local intent recognition model cannot identify the intent or the identified intent is incomplete, the statement is then sent to the voice cloud for recognition, thereby obtaining the intent corresponding to the statement.

[0196] S1004, the voice engine returns the first processing instruction to YOYO.

[0197] Optionally, when the speech engine obtains the intent of a statement through the intent control center, it can also obtain the identifier of the agent corresponding to each intent. It should be noted that the agent corresponding to an intent is an agent capable of performing the task expressed by that intent.

[0198] For example, when the two identified intentions are to remove passersby and to beautify the image, the voice engine can also obtain the identifier of the image editing agent corresponding to the intention to remove passersby, and can also determine the identifier of the image editing agent corresponding to the intention to beautify the image.

[0199] For example, when the two identified intentions are to remove passersby and to generate a dancing video, the speech engine can determine that the intention to remove passersby corresponds to the identifier of the image editing agent, and the intention to generate a dancing video corresponds to the identifier of the video editing agent.

[0200] After determining the obtained intent and the type or identifier of the agent corresponding to each intent, the voice engine can return a first processing instruction to YOYO, which carries the identified intent and the type or identifier of the agent corresponding to each intent.

[0201] S1005 and YOYO send the second processing instruction and the image to be processed to the agent management module.

[0202] The agent management module is used to manage various agents. For example, the agent management module can manage the registration, deregistration, running, and dormant states of multiple agents, realizing the lifecycle management of agents.

[0203] It should be noted that each agent needs to register with the agent management module before it can operate normally. Specifically, the agent registers its corresponding intents with the agent management module. Optionally, when registering with the agent management module, an image editing agent can register intents such as removing passersby or using beauty filters; when registering with the agent management module, a video editing agent can register intents such as generating dance videos, generating GIFs, and other video editing types, which will not be listed here. Based on this, when the agent management module receives the second processing instruction sent by YOYO and obtains the recognized intents, it can distribute the different intents to the corresponding agents.

[0204] Optionally, the intents acquired by the intent controller may also include intents that do not require the use of an agent. In this case, after receiving the intents, YOYO needs to distribute them according to whether the intents are to be processed by an agent.

[0205] Optionally, when the identified intent requires processing by an agent, YOYO sends a second processing instruction to the agent management module. This second processing instruction carries the intent requiring agent processing and the identifier of the agent corresponding to that intent. Optionally, the agent management module can distribute different intents to different agents corresponding to each intent.

[0206] Optionally, when the recognized intent includes an intent that does not require the use of an agent, YOYO does not need to send this intent to the agent management module, but can instead send it to the corresponding processing module. For example, if the user inputs the statement: "Send a picture to the watch via Bluetooth," then the intent obtained by the intent control center includes the intent to "turn on Bluetooth" (turn on Bluetooth connection). YOYO does not need to send the intent to turn on Bluetooth to the agent management module, but can instead send it to the system management module, which will then control the electronic device to turn on Bluetooth. For example, if the user inputs the statement: "Turn on music," then the intent recognized by the intent control center is the intent to open the music application. YOYO also does not need to send the intent to open the music application to the agent management module, but can instead send it to the system management module, which will then control the electronic device to open the music application and play music.

[0207] S1006, the agent management module distributes the images and intentions to be processed to the corresponding agents.

[0208] Specifically, the agent management module can send the image to be processed and the intent to the corresponding agent.

[0209] Taking an image editing agent as an example, when the agent management module receives the intent and identifier of the image editing agent, it can send the intent to the task engine of the image editing agent. The agent management module will also send the image to be processed to the image editing agent.

[0210] S1007. The agent uses the task engine to orchestrate the intent into tasks and obtains a task list.

[0211] Taking an image editing agent as an example, it includes a task engine, which interacts with the cloud testing server (the server where the task decomposition model is deployed). The received intent is sent to the cloud testing server, which then performs task decomposition and orchestration to obtain a task list. After obtaining the task list, the cloud testing server returns it to the task engine. Figure 11 Taking the task decomposition model deployed on a voice cloud as an example, it can also be deployed on other cloud-side servers. In that case, the voice cloud can send the recognized intents that require processing by the intelligent agent to the cloud-side server for task decomposition and to obtain a task list.

[0212] Optionally, taking the task decomposition model deployed in a voice cloud as an example, the task engine can send a task orchestration request to the voice cloud through the task orchestration interface. This task orchestration request carries an intent. Optionally, the task orchestration request can carry one or more intents.

[0213] When there are multiple intents, the voice cloud can break down the multiple intents into tasks, and then arrange the multiple tasks obtained from the breakdown into a task list.

[0214] For example, the task engine sends the intents of "removing pedestrians" and "generating ID photos" to the voice cloud, which can then process these two intents... Figure 1 The process is broken down into two tasks: removing passersby and generating ID photos. The voice cloud can then orchestrate these two tasks to create a task list that includes both. Furthermore, in the task list, the task of removing passersby is executed first, followed by the task of generating ID photos.

[0215] Optionally, the task list can be a list in the form of a DSL, where DSL stands for Domain-Specific Language. For example:

[0216]

[0217] This DSL-style task list indicates that the task of eliminating passersby should be performed first, followed by the task of generating a passport photo with a blue background.

[0218] Optionally, if the electronic device has a local model for task orchestration deployed locally, the task engine can also send intents to the local model for task breakdown and orchestration to obtain a task list.

[0219] S1008: The task engine calls the corresponding processing tools for different tasks in sequence according to the task list to process the tasks and obtain the processing results.

[0220] The task list can include multiple tasks, each corresponding to a processing tool. Different tasks within the same task list may use the same processing tool or different processing tools. This processing tool can be a system tool installed on the electronic device (such as a calculator) or a third-party tool (such as other image editing applications). It should be noted that the image editing agent can be configured with corresponding interfaces to invoke the appropriate processing tool to process and complete the corresponding task.

[0221] Optionally, the aforementioned processing tools can also be cloud-based servers. The image editing agent can be configured with interfaces to other cloud-based servers to call upon them for more complex task processing. For example, the image editing agent can be configured with a visual software development kit (SDK) interface, which interacts with the edge-side visual platform to perform complex image processing tasks, such as intelligent image removal. Optionally, the visual platform can also interact with the visual cloud to achieve more complex functions, such as removing pedestrians or objects selected by the user.

[0222] Optionally, when the user-inputted statement also includes information about relationships between people, the image editing agent can search its learning memory platform for this information before executing the task. This allows it to determine which person in the image being processed is referred to by the relationship information, and then perform the corresponding task for that person. Optionally, the relationship information can be pronouns, such as names and / or titles. The names can be the person's full name, code name, or nickname.

[0223] For example, consider the statement: "Keep only Zhang San." This statement contains the pronoun "Zhang San." The image editing agent can query the learning and memory platform (or learning and memory database) to check if a person identifier corresponding to this pronoun exists. If the person identifier exists in the learning and memory platform, it means that the media database contains a facial image corresponding to this pronoun, which can be used to match the target person corresponding to this pronoun.

[0224] If the learning and memory system does not contain a corresponding person icon for a pronoun, the system can interact with the user through dialogue and, based on user feedback, clarify the specific object indicated by the pronoun from the image to be processed. This process is called person relationship clarification. A detailed explanation of the person relationship clarification process can be found later in this text. Figure 12 The description of the relevant embodiments is omitted here.

[0225] Taking the task of removing passersby as an example, the task engine can call the corresponding visual SDK for the task, interact with the visual platform, and send the task to the visual cloud through the visual platform. The visual cloud then removes passersby from the image to be processed, except for the main subject, resulting in an image with passersby removed. The visual cloud can then send the image with passersby removed back to the visual platform. The visual platform then uses this image as the processing result and returns it to the task engine through the visual SDK.

[0226] Optionally, the task list may contain multiple tasks that can be executed in parallel, such as the task of removing pedestrians and the task of removing billboards. In this case, the task engine can call the tools for removing pedestrians and the tools for removing objects respectively to execute these two tasks in parallel.

[0227] When multiple intents correspond to multiple agents, the returned task list includes the order of the tasks, the identifiers of the corresponding agents, and the transmission relationships between different tasks. For example, after using an image editing agent to remove passersby, the result can be sent to a video editing agent for processing, such as generating a short video, GIF, AI actions, etc. Finally, the video editing agent returns the generated short video to YOYO for display through the agent management module.

[0228] S1009, The task engine retrieves the returned processing results.

[0229] Optionally, the processing result can be the result after the processing tool completes the task, such as the image after removing pedestrians; it can also be information representing the processing result, such as pedestrians being successfully removed; or it can be intermediate information generated during the process of the processing tool, such as: intelligently recognizing pedestrians, removing pedestrians, etc.

[0230] Taking pedestrian removal as an example, the visual cloud removes all other people except Zhang San from the image to be processed, and then sends the image after removing other people to the visual platform. The visual platform takes the image after removing other people as the processing result, returns it to the task engine through the visual SDK, and returns it to YOYO for display through the intelligent agent management module.

[0231] After receiving the processing result, the S1010 task engine sends a reply request to the voice cloud through the reply interface.

[0232] Optionally, after receiving the processing result or intermediate information, the task engine can also send a response request to the voice cloud through the response interface. This response request can carry result information representing the processing result. Optionally, the result information can indicate that the elimination was successful, the elimination failed, or the elimination is in progress.

[0233] In some situations, the image editing agent may malfunction and fail to complete its task. For example, when the electronic device is offline, the image editing agent cannot interact with the voice cloud, sending a reply request to the voice cloud will fail, and it will also be unable to obtain the reply message sent by the voice cloud. The image editing agent can then generate a reply message locally based on the reason for the failure. For example, when the electronic device is offline, the output reply message could be: "The network is down, please wait."

[0234] S1011 The task engine receives the reply through the reply interface and sends it to the agent management module.

[0235] The voice cloud can respond to reply requests and generate corresponding replies based on the result information. For example, if the result information indicates that a passerby has been successfully eliminated, the reply could be: "Congratulations, you have successfully eliminated a passerby." Similarly, if the result information indicates that text has been successfully eliminated, the reply could be: "Congratulations, you have successfully eliminated text."

[0236] The voice cloud can send the generated responses to the edge, for example, it can send them to the task engine of the image editing agent through the response interface.

[0237] The task orchestration interface and response interface mentioned above are interfaces (SDK) based on the Android Archive (AAR) framework.

[0238] S1012, the agent management module returns the processing results and / or response to YOYO.

[0239] Optionally, after receiving the processing result, the agent management module can send the processing result to YOYO. For example, if the processing result is an image after removing pedestrians, the agent management module can send the image after removing pedestrians to YOYO. Optionally, the agent management module can also obtain the response and send the response to YOYO as well.

[0240] S1013, YOYO will display the received result image processing results and / or reply text.

[0241] When YOYO receives a processing result from the agent management module, it can display the result in a dialog box. For example, if the result is an image after removing pedestrians, YOYO can display the image after removing pedestrians. When YOYO receives a reply from the agent management module, it can also display the reply in the dialog box, for example... Figure 9 As shown in (d) in the figure.

[0242] Next, combined Figure 12 The following describes in detail how to clarify interpersonal relationships, along with related implementation examples. Figure 12 A flowchart illustrating a method for clarifying interpersonal relationships is provided. Specifically, it includes:

[0243] S1101. Obtain the pronouns related to people in the sentence and query the learning memory bank to see if there is a target person tag (Tag ID) corresponding to the pronoun.

[0244] Optionally, the aforementioned personal pronouns can be personal names and / or titles. Among them, personal names can be the full names of people, such as Zhang San, Li Si, etc., or they can be pronouns for people, such as "baby"; titles can be: Dad, Mom, daughter, etc.

[0245] When the user's input contains pronouns that represent relationships between people, such as titles or names, the image editing agent can first check if the Tag ID corresponding to that pronoun exists in the learning memory database.

[0246] It should be noted that the learning memory bank can store the correspondence between multiple titles / names and multiple Tag IDs. Optionally, each Tag ID can correspond to one or more names or titles. For example, the names "son" and "baby" can correspond to the same Tag ID-1, and "dad" and "Zhang San" can correspond to the same Tag ID-2.

[0247] Optionally, the learning memory bank can also store the correspondence between different interpersonal relationships (names and / or titles) and Tag IDs, and can also include attribute information corresponding to the interpersonal relationships, such as the birthday and photo of the title "father". This attribute information can be obtained and stored from ID photos (headshots, facial feature information), text messages, notes, and chat logs stored on electronic devices.

[0248] The aforementioned learning and memory platform can access a preset database (i.e., a learning and memory bank) stored locally on the electronic device, or it can access a preset database on a cloud-side server to determine whether the TagID corresponding to the person's pronoun already exists.

[0249] If the learning memory bank contains a Tag ID corresponding to the person pronoun, then S1102A can be executed; if the learning memory bank does not contain a Tag ID corresponding to the person pronoun, then S1102B can be executed.

[0250] S1102A, Obtain the facial feature information corresponding to the target Tag ID.

[0251] Facial feature information can be used to distinguish different people. Optionally, the aforementioned facial feature information can be a person's facial feature vector; it can also be a person's head image instance box, that is, a rectangular box containing a person's head extracted from an image in a media database (referred to as a person's head image).

[0252] Optionally, during idle time, the media database can cluster multiple stored images according to different people, resulting in multiple image sets. Each image set contains the same person. The media database can extract the facial feature information of the corresponding person for each image set and establish a correspondence between it and the person's Tag ID. Taking facial feature information as a person's headshot as an example, based on this, the media database can obtain a one-to-one correspondence between multiple Tag IDs and multiple person headshots.

[0253] Optionally, the portrait corresponding to each image set can be one of the multiple images in the set, showing the face of the person in a clear and frontal view.

[0254] Optionally, the image intelligent editing body can query the correspondence between person avatars and tag IDs in the media database. If the correspondence includes a target tag ID, the target person avatar (or target avatar instance frame) corresponding to the target tag ID can be determined from it. Then, S1103 is executed. Optionally, the media database in this embodiment can also be replaced with other types of databases, as long as they can provide the corresponding data.

[0255] S1103. Based on the facial feature information corresponding to the target Tag ID, determine the target person in the image to be processed that matches the facial feature information corresponding to the target Tag ID. Then execute S1106.

[0256] The image editing agent can compare the facial image of each person in the image to be processed with the facial feature information corresponding to the target Tag ID, and select the person with the highest matching degree as the target person. The target person is the person referred to by the pronoun in the user's input statement.

[0257] Optionally, the image editing agent can also send the target person's headshot corresponding to the target Tag ID to the visual cloud through the visual platform, so that the visual cloud can determine the target person in the image to be processed stored on the cloud side based on the target person's headshot.

[0258] S1102B: Generate and display the first query based on the feature information of prominent figures in the image to be processed.

[0259] Electronic devices can employ visual positioning algorithms to locate visual targets in images and detect the visual features of salient individuals. Visual features may include, but are not limited to, the gender, clothing, color, and position of people in the image. Optionally, the electronic device can invoke a detection model to perform visual feature detection on the image to be processed, identifying the visual features of salient individuals.

[0260] Optionally, the detection model can be a subject detection model, capable of detecting the main subject in the image and its visual features, such as the observable visual features of a prominent person, such as their clothing, hairstyle, gender, and location. The main subject can include prominent people or objects. Prominent people or objects can be clearly visible to the human eye in the image, such as a person or object occupying the center of the image, or a person or object occupying the largest area of ​​the image.

[0261] Optionally, when the electronic device calls the detection model to detect features of salient persons, it can also send the image to be processed to a cloud-based server (e.g., a visual cloud) through a visual middleware platform, and call the large model on the cloud-based server for detection. This application embodiment does not limit the method of detecting features of salient persons. Afterwards, the electronic device can obtain the detection results from the cloud-based server.

[0262] Then, the image editing agent can generate the corresponding first query based on the feature information of prominent figures. For example, if the user inputs: "Only Zhang San should be kept in the photo, and the image to be processed can be..." Figure 14 Image 1 is shown in (a). In it, person A, located in the center and occupying the largest area, is wearing a red shirt. If the electronic device can detect person A as a prominent figure and the visual feature of person A wearing a red shirt, it can generate a first question, such as: "Is this the person wearing a red shirt?"

[0263] Optionally, the electronic device can also perform preliminary screening of people in the image to be processed based on the consistent features indicated in the statement. For example, if the reply statement includes "people wearing red hats", the electronic device can first screen out people wearing red hats, narrowing the search scope, which can effectively reduce the number of clarification rounds and thus improve clarification efficiency.

[0264] Optionally, the electronic device can also filter the people in the image to be processed based on the basic information indicated by the relationship between the people. For example, if the relationship between the people in the sentence is "father", the basic information corresponding to father is male, which can also correspond to an adult. Then, female people in the image to be processed can be excluded, and adult male people can be selected. Then, questions can be asked about the selected people, thereby narrowing the search scope and improving the clarification efficiency.

[0265] Next, the image editing agent can send the first query to YOYO, which will then display the query in a dialog box. Then, step S1104 is executed.

[0266] S1104, Receive the first response message input by the user.

[0267] After seeing the first question in YOYO's dialog box, users can first determine whether the person being asked in the first question is the person they want to instruct, and then reply accordingly with the first reply.

[0268] If the person being asked in the first question is the person the user wants to instruct, the user can enter a voice or text message in the YOYO dialog box as the first response, such as "Yes" or "That's right". Figure 13 Example (a) uses the first query to indicate the target person the user wants to point to. After the user replies with a confirmation, the image editing agent can perform corresponding removal tasks based on the clarified target person and return the removed image.

[0269] If the person addressed in the first query is not the person the user intends to refer to, the user can enter a denial in the YOYO dialog box as the first response, such as "No," or "It's someone else." The image editing agent can then continue to output other queries through the YOYO dialog box. Figure 13 Example of what YOYO's dialog box displays when the person asked in the first question is not the target person the user wants to indicate, as shown in (b) of the example.

[0270] S1105. Determine the target person based on the first reply entered by the user.

[0271] If the user's first response is a voice or text confirmation, YOYO can return the first response to the image editing agent. The image editing agent can then determine that the person in the first query is the person the user wants to instruct on, and designate that person as the target person.

[0272] Optionally, if the user's first response is a denial expressed by voice or text, YOYO can return the first response to the image editing agent. The image editing agent can then determine that the person being asked in the first question is not the person the user intends to point to, and can proceed to a second round of questioning.

[0273] During the second round of questioning, the electronic device can continue to detect the visual features of other people (such as a second person) and generate corresponding second questions, which are then displayed in YOYO's dialog box. For example, when a user replies with "no" or other denial to the first question, "Is it the person wearing the red shirt?", the electronic device can detect... Figure 13 In Figure (a), the image editing agent identifies person C on the left in picture 1 and generates and displays the second question: "Is it the person on the left wearing blue pants?" If the user is referring to person B wearing blue pants, they can enter a confirmation message in the YOYO dialog box; if the user is not referring to person B wearing blue pants, they can enter a denial message in the YOYO dialog box. Afterward, the image editing agent can determine that the person in question in the second question is not the person the user intends to point to, and can then proceed to the third round of questioning.

[0274] During the third round of questioning, the electronic device displays the third question in YOYO's dialog box, such as: "Is it the person on the right wearing a hat?" If the user refers to Zhang San as person C wearing a hat, the user can enter confirmation in YOYO's dialog box, and the image editing agent can then confirm that the target person, Zhang San, is person C. If the user's Zhang San is not person C wearing blue pants, the user can enter denial in YOYO's dialog box. If the image editing agent determines that the person referred to in the third question is not the person the user intends to indicate, it can continue to the next round of questioning.

[0275] Optionally, the image editing agent can use YOYO to conduct more rounds of queries, allowing the user to repeatedly confirm and clarify until the person the user wants to indicate is identified. It should be noted that the more people in the image to be processed, the more rounds of queries are required.

[0276] Optionally, the image editing agent can stop asking questions after a preset number of rounds, such as three times. At this point, it's possible the user's input cannot accurately identify the person being asked. The image editing agent can then output a prompt, such as displaying in YOYO's dialog box: "Unable to confirm who Zhang San is. Please check if Zhang San exists," or "Please re-enter the command," etc. This prompt indicates that the electronic device cannot identify a matching person based on the user's input and requires re-entry, thus avoiding multiple rounds of invalid questions. Figure 13 Example of what YOYO's dialog box displays when the person asked in the three rounds of questions is not the target person the user wants to indicate.

[0277] Optionally, during a round of questioning, the electronic device can also output a question containing multiple visual feature information. Each visual feature information in the question corresponds to one of the visual features of the same person. For example, if person A's visual features include: wearing a red top, white pants, and black shoes, then the output could be something like: Figure 13 The question shown in (d) is: Is it the person wearing a red top? Is it the person in the picture whose only pants are visible? Is it the person in the picture whose only shoes are visible? This question includes three visual features: wearing a red top, wearing white pants, and wearing black shoes, and all three visual features point to the same person, A. Using multiple features makes it easier to clearly describe the visual characteristics of the person being inquired about. At this point, the user can accurately understand which person is being inquired about based on the multiple visual features presented in the question, thus making an accurate response and improving the accuracy of clarifying relationships. For example, Figure 13 As shown in (d), the user can directly input: Yes, and the image editing agent can then determine that person A is the target person.

[0278] Optionally, during the inquiry process, the electronic device can also output multiple inquiries at once, each corresponding to a person's visual characteristics. Different inquiries correspond to different people's visual characteristics, reducing the number of interactions and improving interaction efficiency. In YOYO's dialog box, the inquiry corresponding to the most prominent person can be displayed in the first row, while inquiries corresponding to other people can be displayed in the second row and other rows. For example, displaying... Figure 14 The dialog box is shown in Figure (a). In the image to be processed, among the three figures, figure A, located in the middle and occupying the largest area, wears a red shirt; figure B, on the left, wears blue pants; and figure C, on the right, wears a hat. The electronic device can detect figure A and the fact that figure A wears a red shirt, figure B and the fact that figure B wears blue pants, and figure C and figure C wear hats. It can then generate multiple queries: Is this the person wearing the red shirt? Are only the people wearing blue pants in the image? (Is this the person wearing blue pants?) Are only the people wearing hats in the image? (Is this the person wearing a hat?) When the user sees these queries, they can select one from the queries to refer to the target figure and reply. For example, the user can input: "Is this the person wearing a red shirt?" Then, the electronic device can determine that the Zhang San the user wants to refer to is figure A.

[0279] Optionally, when faced with multiple inquiries, users can select one that does not refer to the target person and reply. For example, a user can input: "Not the person wearing the red shirt." The electronic device can then determine that the target person, Zhang San, is not person A, but possibly person B or person C. Further clarification can then be provided regarding person B or person C until the target person is identified.

[0280] Optionally, if more than one person in the image to be processed shares the same visual characteristics—for example, if the image contains five people, two of whom are wearing red shirts and the remaining three are wearing white shirts—YOYO can display the question: "Is this the person wearing the red shirt?" If the user replies "Yes," the image editing agent can then focus on the two people wearing red shirts. Then, it can ask the user a question about one of those two people. This effectively reduces the number of question rounds and improves interaction efficiency.

[0281] The above-mentioned inquiry based on the visual characteristics of a person can prompt the user to respond and describe the person according to the visual characteristics that the electronic device can recognize, and will not continue to use pronouns to ask questions, thus improving the efficiency of interaction.

[0282] Optionally, if the target tag ID corresponding to a person's pronoun does not exist in the learning memory bank, the electronic device can also display multiple avatars in the YOYO dialog box. These multiple avatars are person avatars corresponding to multiple image sets clustered in the media database, for example... Figure 14 As shown in (c) above. At this point, the user can click on the matched character avatar to determine the target character.

[0283] Optionally, each person's portrait can be accompanied by their name, allowing users to select a target person by entering a reply in YOYO's dialog box. For example, if the text identifier for Person 1's portrait is "Xiao Zhang," the user can reply with "Xiao Zhang," and the image editing agent can determine that Person 1, corresponding to the text identifier "Xiao Zhang," is the target person's portrait. Then, the image editing agent can use Person A's portrait to match among multiple people in the image to be processed, thereby identifying the target person in the image whose portrait matches Person A's.

[0284] Optionally, when displaying character avatars in YOYO's dialog box, a preset number of avatars can be displayed at once, such as six. If none of the six currently displayed avatars represent the user's desired person, Zhang San, the user can click an expansion control, such as the "More" control, to display other avatars for selection. Optionally, the user can also reply with a negative message such as "None of these" in YOYO's dialog box, at which point YOYO's dialog box will expand to display other avatars for selection. Through the user's selection, the image editing agent can determine the target person the user wants to represent.

[0285] Once the target person is identified, appropriate actions can be taken against them.

[0286] Optionally, after identifying the target person, the image editing agent can also establish a correspondence between the target person's portrait, tag ID, and pronouns, and store the target person's portrait. For example, the image editing agent establishes a correspondence between "Zhang San," "Zhang San's portrait," and "Zhang San's tag ID," and stores it in the learning memory. This allows the agent to quickly identify the target person by querying the learning memory when the user's input contains a person with that tag ID.

[0287] Optionally, the above-mentioned questions can also be tailored to user habits, selecting a style that the user prefers, making the questions more intelligent and enhancing the user experience.

[0288] After that, the corresponding processing tasks can be carried out on the target person. Details will not be elaborated here.

[0289] S1106. Perform intelligent elimination based on the target person to obtain the eliminated image.

[0290] If the user specifies that they want to remove a target person, the target person can be segmented and then filled in to obtain the processed image, i.e., the result image. If the user wants to remove people other than the target person, the people other than the target person can be segmented and then filled in to obtain the processed image. The process of image segmentation can be found in the relevant descriptions later in the text, and will not be repeated here.

[0291] Optionally, the above processing task can also be of other types, such as adding a yellow hat to the target person, changing the color of the flower basket behind the target person to blue, or making the target person's eyes open. After identifying the target person, the electronic device can call the corresponding image editing tool to perform processing operations such as adding a yellow hat to the target person, changing the color of the flower basket behind them to blue, and making their eyes open, to obtain the image desired by the user.

[0292] Optionally, when the statement includes multiple person pronouns, the electronic device can also use the target person determination method in this embodiment to determine the multiple person pronouns separately, and then perform corresponding processing. For example, adding person A to a photo, placing them between person B and person C. The electronic device can determine that person A does not exist in the image to be processed. In this case, the image of person A can be obtained from the results of media database clustering, and then added between person B and person C. Optionally, the image around person A can be blended with the image to be processed to avoid overly abrupt boundaries that result in poor image quality. Optionally, the positions of person B and person C can be appropriately moved to make the positional relationship of the people in the generated image more reasonable.

[0293] Next, Figure 15 The example shown uses the user-input statement "Keep only Zhang San" as an example to explain the overall elimination process when there are relationships between people in the statement.

[0294] S1401, Obtain the image to be processed and the statement input by the user.

[0295] Specifically, YOYO receives the image to be processed and the statements input by the user.

[0296] S1402. Perform intent recognition on the statement to obtain the corresponding intent.

[0297] S1403. Determine whether there are personal pronouns in the statement.

[0298] YOYO transmits the image to be processed and the user's input to the image editing agent. The agent interacts with the voice cloud to identify the intent of the statement. It also determines whether the statement contains information about relationships between people (e.g., pronouns). The voice cloud can further break down and orchestrate tasks based on the identified intent, generating a task list which is then returned to the image editing agent.

[0299] If the statement contains a pronoun referring to a person, the image editing agent executes S1404; if the statement does not contain a pronoun referring to a person, the image editing agent executes S1411.

[0300] For example, the image editing agent recognized the intention to remove the image and also identified the pronoun "Zhang San".

[0301] S1404. Query whether the target Tag ID corresponding to the person pronoun exists in the learning memory bank.

[0302] If the statement contains a person pronoun, the image editing agent can query the learning memory to see if the target tag ID corresponding to the person pronoun exists. If the target tag ID exists in the learning memory, then execute S1405A; if the target tag ID does not exist in the learning memory, then execute S1405B.

[0303] S1405A. Based on the target Tag ID, determine the person's avatar corresponding to the target Tag ID from the media database.

[0304] If the target tag ID exists in the learning memory, the image editing agent determines the person's portrait corresponding to the target tag ID from the media database. Then, S1406 is executed.

[0305] For example, if the learning memory bank contains the target TagID (Xiao Zhang) corresponding to Zhang San, then the person's portrait of the target TagID (Xiao Zhang) can be found in the media database.

[0306] S1406. Based on the person's portrait corresponding to the target TagID, determine the target person in the image to be processed that matches the portrait. Then execute S1410.

[0307] S1405B: Perform visual feature detection on the image to be processed to obtain visual feature information of prominent figures.

[0308] If the target Tag ID (Xiao Zhang) is not present in the learning memory, the image editing agent acquires the visual feature information of the salient person in the image to be processed. Optionally, the visual feature information of the salient person is obtained by the electronic device calling the visual detection model to perform visual feature detection on the image to be processed, which can characterize the visual features of the salient person. Then, S1407 is executed.

[0309] S1407. Output the query based on the visual characteristics of prominent figures.

[0310] The question can be directed at a single visual feature of a prominent person, or it can be directed at multiple visual features of a prominent person.

[0311] To see how YOYO displays a prompt in its dialog box, see: Figure 12 The various interfaces are shown in the image.

[0312] S1408. Receive the user's input response.

[0313] S1409A. If the reply indicates definite information, then the prominent person is identified as the target person.

[0314] After observing the inquiry, the system can determine whether the visual features represented by the inquiry correspond to the person the user is referring to, and then enter a reply in the dialog box. See details below. Figure 11 Flowcharts and Figure 12 , Figure 13 The interface diagram shown.

[0315] S1409B. If the response indicates a negative message, then the prominent person is determined not to be the target person.

[0316] If the user's response is "Yes," it means the visual features suggested by the query are those of a salient person, and therefore the salient person is the target person. If the user's response is "No," it means the visual features suggested by the query are not those of a salient person, and multiple rounds of questioning can be conducted with other individuals until the target person is identified.

[0317] S1410. Determine the position of the target person in the image to be processed.

[0318] Once the target person is identified, the image editing agent can use visual target localization to obtain the position of the target person in the image to be processed.

[0319] S1411. Based on the position of the target person in the image to be processed, segment other people besides the target person to obtain the segmentation result.

[0320] S1412. Based on the segmentation results, perform elimination and filling to obtain the result image.

[0321] The image editing agent interacts with a visual cloud and, based on the identified elimination task, segments other people in the image to be processed, according to their positions in the target person. This segmentation result can be a mask representing all people except Zhang San. The visual cloud can then segment the corresponding positions in the image to be processed using these masks, and then fill in the remaining images to obtain a final image where only Zhang San is retained after eliminating the other people.

[0322] S1413. Generate the correspondence between the person's pronouns and the target person's Tag ID, and store it in the learning memory bank.

[0323] Optionally, the image editing agent also generates and stores the correspondence between the target person's Tag ID and the person's avatar.

[0324] The execution order of S1413 can be before any of the steps S1410 to S1412, as long as the target person is identified.

[0325] Afterwards, the image editing agent can feed back the resulting image and the elimination results to YOYO for display, and can also return a response describing the elimination results.

[0326] Figure 16 This example illustrates the overall elimination process when a relationship exists between characters in another statement. For instance... Figure 16 The method includes:

[0327] S1401, Obtain the image to be processed and the statement input by the user.

[0328] S1402. Perform intent recognition on the statement to obtain the corresponding intent.

[0329] S1403. Determine whether there are personal pronouns in the statement.

[0330] If the statement contains a pronoun referring to a person, the image editing agent executes S1404; if the statement does not contain a pronoun referring to a person, the image editing agent executes S1411.

[0331] S1404. Query whether the target tag ID corresponding to the person pronoun exists in the learning memory bank.

[0332] If the target Tag ID exists in the learning memory, execute S1405A; if the target Tag ID does not exist in the learning memory, execute S1505.

[0333] S1405A. Based on the target Tag ID, determine the person's avatar corresponding to the target Tag ID from the media database.

[0334] If the target tag ID exists in the learning memory, the image editing agent determines the person's avatar corresponding to the target tag ID from the media database. Then, S1406 is executed.

[0335] For example, if the learning memory bank contains the target Tag ID (Xiao Zhang) corresponding to Zhang San, then the person's portrait with the target Tag ID (Xiao Zhang) can be found in the media database.

[0336] S1406. Based on the person's portrait corresponding to the target Tag ID, determine the target person in the image to be processed that matches the portrait. Then execute S1410.

[0337] S1505: Display multiple character portraits clustered from the media database.

[0338] In YOYO's dialog box, multiple character avatars can be displayed as candidate avatars. These avatars are the result of clustering multiple images from a media database.

[0339] S1506. Receive selection information input by the user.

[0340] Users can select one of multiple character avatars by clicking on it; the selection information is generated in response to the user's click action. Users can also enter the name of the desired avatar in the YOYO dialog box; the selection information is the user's input. For displaying multiple avatars in the YOYO dialog box, please refer to... Figure 13 As shown in (b) and (c) in the figure.

[0341] S1507. Based on the user's input selection information, determine the target person. Then, execute S1410.

[0342] S1410. Determine the position of the target person in the image to be processed.

[0343] Once the target person is identified, the image editing agent can use visual target localization to obtain the position of the target person in the image to be processed.

[0344] S1411. Based on the position of the target person in the image to be processed, segment other people besides the target person to obtain the segmentation result.

[0345] S1412. Based on the segmentation results, perform elimination and filling to obtain the result image.

[0346] S1413. Generate the correspondence between the person's pronouns and the target person's Tag ID, and store it in the learning memory bank.

[0347] Optionally, the image editing agent also generates and stores the correspondence between the target person's Tag ID and the person's avatar.

[0348] The execution order of S1413 can be before any of the steps S1410 to S1412, as long as it is after the target person has been identified.

[0349] Optionally, before clarifying the aforementioned relationships between individuals, or before querying the media database for a corresponding tagID, the name of the person indicated in the statement can be queried to determine if they are a widely known individual. If so, the cloud server can be used to query the network data to see if an image of the person indicated by that name exists. A large-scale facial recognition model can then be used to match the image of the person to be processed based on the image of the person indicated by that name. If the match is successful, processing can proceed for the target person indicated by that name; if the match fails, the target person determination method described in this embodiment can be executed based on that name to determine the target person.

[0350] Optionally, if the character's name corresponds to a character that is not allowed to be processed, no other task processing will be performed on that character.

[0351] It should be noted that the timing for stopping task processing on individuals who are not permitted to be processed can be either when the user's speech is uploaded to the voice cloud and verified by the voice cloud. If this is the case, the image processing flow can be stopped and a prompt message can be returned to YOYO; or it can be when the visual cloud is processing the image of the target person and determines that the individual is not permitted to be processed, in which case the image processing flow can be stopped and a prompt message can be returned to YOYO.

[0352] Figure 16 The implementation principles and technical effects of each step can also be found in [reference needed]. Figure 15 The relevant descriptions of other embodiments in this document will not be repeated here.

[0353] Figure 17A A portion of the interaction diagram for an example of a dialogue elimination method provided in this application, such as Figure 17A As shown, the method includes:

[0354] S1601 and YOYO receive the image to be processed and the statement input by the user.

[0355] Users can drag and drop images to be processed into YOYO's dialog box, and can also enter statements in the dialog box. For example, a user might enter: "Keep only Zhang San in the image."

[0356] S1602, YOYO obtains the local path of the image to be processed.

[0357] When an image to be processed is stored locally on an electronic device, its storage location can be represented by a local path. Applications can read the image using this local path. During image transfer, the local path can also be sent to transfer the image.

[0358] S1603 and YOYO send the image to be processed to the storage cloud (i.e., OBS cloud).

[0359] YOYO compresses the image to be processed and then sends it to the OBS cloud via the network.

[0360] S1604. OBS cloud storage of the image to be processed, and generation of the URL address of the image to be processed stored on OBS cloud.

[0361] OBS cloud storage for images to be processed also generates a storage address for these images. This address is a Uniform Resource Locator (URL), which is a network address. Other devices can access the images to be processed through this URL. Different servers or terminal devices can also transfer the images to be processed by transmitting this URL, thus avoiding the significant network resource consumption caused by transmitting the images themselves.

[0362] S1605, OBS Cloud will return the URL to YOYO.

[0363] After receiving the URL, YOYO allows electronic devices to send images to be processed to other cloud-side servers without actually sending the image itself, thus conserving network resources. Other cloud-side servers can then retrieve the image from the URL.

[0364] S1606, YOYO sends the user's input to the speech engine.

[0365] After receiving the user's input, YOYO can send the input to the speech engine. The speech engine can then interact with the speech cloud to recognize the intent expressed in the input.

[0366] It should be noted that step S1606 can be executed before or after any of steps S1602 to S1605, and this application embodiment does not limit this.

[0367] S1607: The voice engine calls the voice cloud through the voice cloud platform to identify the intent corresponding to the sentence and the identifier of the required intelligent agent: image editing intelligent agent.

[0368] The speech engine interacts with the speech cloud platform and sends sentences to the speech cloud. The speech cloud uses a cloud-based Large Language Model (LLM) to perform intent recognition on the sentences, obtaining the user's intended meaning. Simultaneously, the speech cloud can also identify the identifier of the agent required for each intent. Optionally, the agent's identifier can be its name, type, or code name, as long as it represents which agent it is.

[0369] For example, a user inputs the statement: "Keep only Zhang San in the image, and remove all passersby except my daughter." The voice cloud system recognizes the user's intention to remove everyone except Zhang San, and identifies the corresponding agent as an image editing agent.

[0370] Optionally, when certain intents do not require processing by the intelligent agent, the voice cloud can return the identifiers of other modules that are not intelligent agents corresponding to these intents. For example, if the intent to turn on Bluetooth does not require processing by the intelligent agent, the voice cloud can return the intent to turn on Bluetooth and the corresponding processing module as: the system settings module.

[0371] S1608, the voice cloud platform returns the recognized intent and the identifier of the image editing agent to YOYO.

[0372] S1609, YOYO sends the local path, URL, and intent of the image to be processed to the agent corresponding to the agent's identifier.

[0373] YOYO can also send the local path and URL of the image to be processed to the corresponding agent according to the agent's identifier.

[0374] Afterwards, the image editing agent can perform actions such as... Figure 17B The interactive flow shown includes:

[0375] S1610, the image editing agent sends the URL of the image to be processed and the corresponding statement to the voice cloud.

[0376] Optionally, the image editing agent can send the URL of the image to be processed and the corresponding statement to the voice cloud through its own task engine and task orchestration interface.

[0377] Optionally, the URL and statement of the image to be processed can be sent together or separately. This application embodiment does not limit the order in which these three are sent.

[0378] S1611. The voice cloud performs task decomposition and task arrangement on the sentences to obtain a task list and information on the relationships between characters.

[0379] Optionally, the speech cloud can perform intent recognition on the speech and perform task decomposition and task orchestration based on the recognized intent; this is not limited here.

[0380] S1612, the voice cloud returns the task list and character relationship information to the image editing agent.

[0381] Optionally, the voice cloud can receive task lists and person relationship information through the task orchestration interface and task engine of the image editing agent.

[0382] S1613, The image editing agent sends the first query request to the learning and memory platform.

[0383] When the image editing agent determines that the user's input contains information about relationships between people, it can send a first query request to the learning and memory platform after receiving the task list. This first query request carries information about relationships between people, such as pronouns, titles, and / or names.

[0384] S1614. The learning and memory platform queries the learning and memory database to determine whether the TagID of the person indicated by the person pronoun exists.

[0385] This learning and memory database stores the correspondence between multiple titles / personal names and multiple tag IDs. In response to the first query request, the learning and memory platform can query the local database to check if the tag ID of the person referred to by the pronoun in the sentence exists in the correspondence between the multiple titles / personal names and multiple tag IDs.

[0386] If the TagID exists in the learning memory, proceed to step S1615 and thereafter. If the TagID does not exist in the learning memory, proceed to step S1623 and thereafter.

[0387] S1615. Return the TagID to the image editing agent.

[0388] If the learning and memory platform stores the tag IDs corresponding to pronouns in a sentence, it can directly retrieve the tag IDs and return them to the image editing agent. For example, the learning and memory platform returns the tag ID of Zhang San to the image editing agent.

[0389] S1616, The image editing agent sends a second query request to the media database.

[0390] After receiving the Tag ID, the image editing agent can also send a second query request to the media database. This second query request carries the Tag ID corresponding to the person's pronoun.

[0391] S1617. Query the media database for the TagID to obtain the facial coordinates of the person corresponding to the TagID: Zhang San in the preset image.

[0392] The preset image is an image from a media database that includes the face corresponding to Zhang San's Tag ID. This preset image can be a clearly defined, frontal image of Zhang San selected from Zhang San's image set.

[0393] S1618, The media database returns the coordinates of Zhang San's face to the image editing agent.

[0394] In response to the second query request, the media database retrieves the image corresponding to the Tag ID and obtains the facial coordinates of the person corresponding to the Tag ID in the image. Here, the image is a preset image in the media database containing Zhang San. Optionally, the media database can also obtain the facial entropy information of the person in the image based on the facial coordinates, such as obtaining a bounding box containing the face. The facial entropy information is used to characterize the facial features of a person and can distinguish different people. Afterward, the media database returns the facial coordinates of Zhang San in the preset image found to the image editing agent.

[0395] S1619, The image editing agent sends the URL of the image to be processed and Zhang San's avatar frame to the visual middleware platform.

[0396] The image editing AI agent can extract Zhang San's avatar frame from a preset image based on Zhang San's facial coordinates, and then send Zhang San's avatar frame and URL to the visual platform.

[0397] S1620, the visual platform sends Zhang San's avatar frame and URL to the visual cloud.

[0398] The visual platform sends the corresponding avatar frame and URL from the image to be processed to the visual cloud.

[0399] S1621. Visual Cloud segments out the human body instance bounding box of Zhang San in the image to be processed based on Zhang San's avatar frame and the image to be processed stored in the URL.

[0400] Visual Cloud can compare Zhang San's avatar frame with the image to be processed stored in the URL, find Zhang San's avatar in the image to be processed, and then segment Zhang San's human body instance frame based on Zhang San's avatar frame.

[0401] Optionally, the image editing agent can also compare the facial features represented by the headshot frame of Zhang San in the preset image with those features in the image to be processed locally to find the facial position (coordinates) of Zhang San in the image to be processed. Then, the visual platform converts the facial position (coordinates) of Zhang San in the image to be processed into the facial position (coordinates) of Zhang San in the image to be processed on the cloud side, and then sends it to the visual cloud. The visual cloud then segments the human body instance bounding box of Zhang San based on Zhang San's headshot.

[0402] S1622. Return the human body instance frame of Zhang San to the image editing agent through the visual middleware platform.

[0403] Steps S1615 to S1622 above are the process of learning the Tag IDs corresponding to the person pronouns in the memory bank.

[0404] If the tag ID corresponding to the person pronoun does not exist in the learning memory bank, then proceed with steps S1623 and thereafter. For details, please refer to [link to relevant documentation]. Figure 17CAs shown, it includes:

[0405] S1623, The learning and memory platform returns a message to the image editing agent indicating that the TagID does not exist.

[0406] The returned information is used to represent the TagID corresponding to a person's pronoun that does not exist in the learning memory bank.

[0407] S1624, the image editing agent sends the URL, returned information, and conversation history to the voice cloud.

[0408] The image editing agent then responds to the returned information and executes a process to clarify the relationships between the individuals. Specifically, the image editing agent sends the URL of the image to be processed, the returned information, and the conversation history to the voice cloud.

[0409] S1625. The voice cloud generates a query based on the conversation history and returns it to the image editing agent.

[0410] The voice cloud responds to the returned information by calling a multimodal large model on the cloud side to generate corresponding queries based on the conversation history, and then returns the queries to the image editing agent. For example, if the conversation history includes the user's input statement: "Only Zhang San is in the image," then the query could be: "Is Zhang San the person wearing the red shirt?"

[0411] S1626. After receiving the query, the image editing agent returns the query to YOYO.

[0412] Optionally, when the query is directed at a single person, such as a prominent figure, it can be referred to as the main query object statement. Optionally, when the query includes multiple visual features of a single person, or inquires about multiple visual features of multiple people, it can be displayed in the form of a list of message prompts, with each message prompting one visual feature.

[0413] S1627, YOYO displays a prompt to the user.

[0414] YOYO displays the query in a dialog box for the user to see.

[0415] S1628, YOYO receives the user's response to the query and sends it to the image editing agent.

[0416] Users can reply based on the query, entering their response. YOYO receives the user's response and sends it to the image editing agent.

[0417] For information on how to ask questions and how users should respond, please refer to the previous text. Figure 11 , Figure 12 and Figure 13The process and interface shown are not described in detail here.

[0418] S1629. The image editing agent determines the human body instance frame of Zhang San based on the reply.

[0419] The image editing agent can determine which person in the image Zhang San is based on the reply and obtain the bounding box of Zhang San's human body in the image.

[0420] Optionally, after the image editing agent identifies Zhang San as the target person, when further obtaining Zhang San's human body instance bounding box, it can use the visual platform to call the visual cloud to segment Zhang San's human body instance bounding box, which is not shown in the figure.

[0421] Regardless of whether the learning memory bank contains a Tag ID corresponding to the pronoun "Zhang San," after the image editing agent obtains the human body instance frame, it can execute S1630 and subsequent steps. For details, please refer to [link to relevant documentation]. Figure 17D As shown, it includes:

[0422] S1630, the image editing intelligent agent sends the human body instance frame and URL of Zhang San, along with the task of keeping Zhang San and removing passersby, to the visual cloud through the visual middle platform.

[0423] S1631, Visual Cloud segments the characters other than Zhang San to obtain the first mask image.

[0424] Specifically, the visual cloud calls the segmentation model to segment all characters except Zhang San, and obtains the first mask image.

[0425] S1632, Visual Cloud sends the first mask image to the image editing intelligent agent through the visual middleware platform.

[0426] S1633, The image editing agent sends the first mask image and the original image of the image to be processed to the visual middleware platform.

[0427] S1634. The visual platform will segment the original image of the image to be processed based on the first mask image to obtain a rectangular image of the part that needs to be eliminated.

[0428] Optionally, the visual platform can select the pedestrians that need to be eliminated in the original image based on the pedestrians indicated by the first mask image, and obtain a rectangular image containing the part that needs to be segmented. Optionally, the rectangle of the rectangular image can be the smallest bounding rectangle of the pedestrians to be eliminated.

[0429] S1635, The visual platform sends the rectangular image to the visual cloud.

[0430] S1636. Visual Cloud performs elimination and filling based on the rectangular image to obtain a partial result image.

[0431] This rectangular image represents a portion of the image to be processed. Visual Cloud uses its cloud-based segmentation model to remove pedestrians from this portion of the image and fills in the removal area, resulting in a partial image with all pedestrians except Zhang San removed.

[0432] S1637, Visual Cloud returns some of the resulting images to the Visual Platform.

[0433] S1638. The visual platform stitches together the original images of some of the result images and the images to be processed to obtain the final result image.

[0434] The visual processing platform can cut out the rectangular portion of the original image to be processed, and then stitch the partial result image and the cut image to form the final result image.

[0435] S1639. The visual platform sends the resulting image to the image editing agent.

[0436] S1640, the image editing agent obtains the task execution results and the result image.

[0437] After receiving the result image, the image editing agent also generates the corresponding task execution result.

[0438] For example, when the image editing agent receives the result image of the completed deletion task, it indicates that the task execution result is successful.

[0439] If the image editing agent does not receive the result image of the completed removal task, it means that the task execution result is a failure.

[0440] S1641, the image editing agent sends the task execution result to the voice cloud.

[0441] S1642. Voice Cloud generates a response based on the task execution result.

[0442] S1643, Voice Cloud will return the reply to the image editing agent.

[0443] S1644 The image editing agent sends the resulting image and / or reply to YOYO for display.

[0444] Users can observe the image of Zhang San retained, and also see the reply indicating the elimination result.

[0445] The above Figures 17A to 17D The implementation principles and technical effects can be found in the relevant descriptions above, and will not be repeated here.

[0446] The foregoing has detailed examples of the methods provided in this application. It is understood that the corresponding apparatus, in order to achieve the above functions, includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0447] This application can divide the target person determination device into functional modules based on the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0448] Figure 18 A schematic diagram of a target person identification device provided in this application is shown. The device 1800 includes:

[0449] The first determining module 1801 is used to determine that the statement input by the user contains a first person pronoun, which includes at least the person's name and / or title.

[0450] The first processing module 1802 is used to display first prompt information based on the first person pronoun, the first prompt information being used to prompt the visual features of at least one person in the image to be processed;

[0451] The receiving module 1803 is used to receive first indication information input by the user, the first indication information being used to indicate whether the first prompt information matches the first person pronoun;

[0452] The second determining module 1804 is used to determine the target person indicated by the first person pronoun in the image to be processed based on the first indication information.

[0453] In some implementations, the first processing module 1802 is specifically used to determine whether there is a target person identifier corresponding to the first person pronoun in the preset database. The preset database includes a first correspondence between at least one person pronoun and at least one person identifier. The at least one person pronoun includes the first person pronoun, and the at least one person identifier includes the target person identifier. If it exists, the target person indicated by the first person pronoun in the image to be processed is determined according to the target person identifier. If it does not exist, a first prompt message is displayed according to the first person pronoun.

[0454] In some implementations, the first processing module 1802 is specifically used to determine the target facial feature information corresponding to the target person identifier based on the target person identifier; and to determine the target person matching the target facial feature information from the image to be processed based on the target facial feature information.

[0455] In some implementations, the target facial feature information is the target person's headshot. Based on the target person's identifier, the first processing module 1802 is specifically used to obtain the clustering results of the media database. The clustering results include a second correspondence between multiple person identifiers and multiple person headshots. Among the multiple person identifiers, there is a target person identifier, and among the multiple person headshots, there is a target person's headshot. From the second correspondence, the person headshot corresponding to the target person identifier is determined to be the target person's headshot. From at least one person in the image to be processed, the target person who is the same person as the target person's headshot is matched.

[0456] In some embodiments, the first processing module 1802 is specifically used to detect the image to be processed, obtain at least one visual feature information, the at least one visual feature information representing at least one visual feature of a person in the image to be processed; and display first prompt information, the first prompt information being used to prompt at least one visual feature, the at least one visual feature and the at least one visual feature information corresponding one-to-one.

[0457] In some implementations, the first person is a prominent person in the image to be processed.

[0458] In some implementations, the first prompt information includes at least a first question, and the first processing module 1802 is specifically used to display the first question. The first question includes the first visual feature information of the first person, and the first visual feature information is one of at least one visual feature information.

[0459] The receiving module 1803 is used to receive the first response input by the user;

[0460] The first processing module 1802 is further configured to determine the first person as the target person if the first response indicates that the first visual feature information matches the first person pronoun.

[0461] In some implementations, the first processing module 1802 is further configured to display a second query if the first response indicates that the first visual feature information does not match the first person pronoun. The second query is used to characterize the second visual feature information of the second person in the image to be processed, and the second person is different from the first person.

[0462] The receiving module 1803 is also used to receive a second response input by the user;

[0463] The first processing module 1802 is further configured to determine the second person as the target person if the second response indicates that the second visual feature information matches the first person's pronoun.

[0464] In some implementations, the first processing module 1802 is further configured to display a third query if the second response indicates that the second visual feature information does not match the first person pronoun. The third query is used to characterize the third visual feature information of a third person in the image to be processed, and the third person is different from both the first person and the second person.

[0465] The receiving module 1803 is also used to receive a third response input by the user;

[0466] The first processing module 1802 is further configured to determine the third person as the target person if the third response indicates that the third visual feature information matches the first person's pronoun;

[0467] The first processing module 1802 is further configured to output a second prompt message if the third response indicates that the third visual feature information does not match the first person pronoun. The second prompt message is used to prompt the user to re-enter the statement.

[0468] In some implementations, the number of at least one visual feature information is multiple, and the first processing module 1802 is also used to display a fourth question, the fourth question including multiple visual feature information, the multiple visual feature information being used to characterize multiple visual features of the first person.

[0469] The receiving module 1803 is also used to receive a fourth response input by the user;

[0470] The first processing module 1802 is further used to determine the first person as the target person if the fourth response indicates that multiple visual feature information matches the first person's pronoun.

[0471] In some implementations, the number of at least one visual feature information is multiple. The first processing module 1802 is specifically used to display a fifth query. The fifth query includes multiple visual feature information. The multiple visual feature information is used to characterize the visual features of multiple people in the image to be processed. The multiple visual feature information and the multiple people correspond one-to-one.

[0472] The receiving module 1803 is also used to receive a fifth response input by the user, which is used to indicate the fourth visual feature information among multiple visual feature information.

[0473] The first processing module 1802 is further used to determine the person corresponding to the fourth visual feature information as the target person if the fourth visual feature information indicated by the fifth reply matches the first person pronoun.

[0474] In some implementations, the first processing module 1802 is further configured to output a third prompt message if the fifth response indicates that any one of the multiple visual feature information does not match the first person pronoun. The third prompt message is used to prompt the user to re-enter the statement.

[0475] In some embodiments, the device 1800 further includes a mapping module for establishing a correspondence between the target person identifier and the first person pronoun; and adding the correspondence between the target person identifier and the first person pronoun to the first correspondence.

[0476] In some embodiments, the apparatus 1800 further includes an acquisition module for acquiring the image to be processed and the user-inputted statement before determining that the statement input by the user contains a first person pronoun; acquiring the intent represented by the statement, the intent corresponding to the first person pronoun, the intent being an intention to eliminate;

[0477] The device 1800 also includes an acquisition module execution module, which is used to perform an elimination task corresponding to the intent of the image to be processed based on the target person, obtain a result image, and display the result image.

[0478] The specific manner in which the device 1800 executes the target identification method and the beneficial effects thereof can be found in the relevant descriptions in the method embodiments, and will not be repeated here.

[0479] This application also provides an electronic device, including the processor described above. The electronic device provided in this embodiment may be... Figure 1 The terminal device 100 shown is used to execute the aforementioned target identification method. When using integrated units, the terminal device may include a processing module, a storage module, and a communication module. The processing module can be used to control and manage the actions of the terminal device; for example, it can support the terminal device in executing the steps performed by the display unit, detection unit, and processing unit. The storage module can support the terminal device in storing program code and data. The communication module can support communication between the terminal device and other devices.

[0480] The processing module can be a processor or a controller. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination of functions that implement computing capabilities, such as a combination of one or more microprocessors, a digital signal processor (DSP), and a microprocessor, etc. The storage module can be a memory. The communication module can specifically be a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, or other devices that interact with other terminal devices.

[0481] In one embodiment, when the processing module is a processor and the storage module is a memory, the terminal device involved in this embodiment can be a device having... Figure 1 The device with the structure shown.

[0482] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the target person determination method described in any of the above embodiments.

[0483] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the target person determination method described above.

[0484] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0485] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units. The replaced units may or may not be physically separate. The component shown as a unit may be one physical unit or multiple physical units, that is, it may be located in one place or distributed in multiple different places. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0486] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0487] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0488] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for identifying a target person, characterized in that, include: Determine that the statement entered by the user contains a first person pronoun, which includes at least the person's name and / or title; Based on the first person pronoun, display the first prompt information, which is used to prompt the visual features of at least one person in the image to be processed; Receive first instruction information input by the user, the first instruction information being used to indicate whether the first prompt information matches the first person pronoun; Based on the first indication information, the target person indicated by the first person pronoun in the image to be processed is determined.

2. The method according to claim 1, characterized in that, The step of displaying the first prompt information based on the first person pronoun includes: Determine whether a target person identifier corresponding to the first person pronoun exists in a preset database. The preset database includes a first correspondence between at least one person pronoun and at least one person identifier. The at least one person pronoun includes the first person pronoun, and the at least one person identifier includes the target person identifier. If it exists, then based on the target person identifier, determine the target person indicated by the first person pronoun in the image to be processed; If it does not exist, then display the first prompt message based on the first person pronoun.

3. The method according to claim 2, characterized in that, The step of determining the target person indicated by the first person pronoun in the image to be processed based on the target person identifier includes: Based on the target person identifier, determine the target facial feature information corresponding to the target person identifier; Based on the target facial feature information, the target person matching the target facial feature information is determined from the image to be processed.

4. The method according to claim 3, characterized in that, The target facial feature information is a target person's portrait. The step of determining the target facial feature information corresponding to the target person's identifier based on the target person's identifier includes: Obtain the clustering results of the media database, wherein the clustering results include a second correspondence between multiple character identifiers and multiple character avatars, wherein the multiple character identifiers include the target character identifier, and the multiple character avatars include the target character avatar; From the second correspondence, the avatar corresponding to the target person identifier is determined to be the target person avatar; From at least one person in the image to be processed, match the target person whose headshot is the same as the target person's headshot.

5. The method according to claim 2, characterized in that, The step of displaying the first prompt information based on the first person pronoun includes: The image to be processed is detected to obtain at least one visual feature information, wherein the at least one visual feature information represents at least one visual feature of a person in the image to be processed; The first prompt information is displayed, which is used to prompt the at least one visual feature, and the at least one visual feature and the at least one visual feature information correspond one-to-one.

6. The method according to claim 5, characterized in that, The first person is a prominent figure in the image to be processed.

7. The method according to claim 6, characterized in that, The first prompt information includes at least a first question, and displaying the first prompt information includes: The first question is displayed, and the first question includes the first visual feature information of the first person, wherein the first visual feature information is one of the at least one visual feature information; The first instruction information received from user input includes: Receive the user's first response; The step of determining the target person indicated by the first person pronoun in the image to be processed based on the first indication information includes: If the first response indicates that the first visual feature information matches the first person pronoun, then the first person is determined to be the target person.

8. The method according to claim 7, characterized in that, The method further includes: If the first response indicates that the first visual feature information does not match the first person pronoun, then a second query is displayed. The second query is used to characterize the second visual feature information of the second person in the image to be processed, and the second person is different from the first person. Receive the second response message entered by the user; If the second response indicates that the second visual feature information matches the first person's pronoun, then the second person is determined to be the target person.

9. The method according to claim 8, characterized in that, The method further includes: If the second response indicates that the second visual feature information does not match the first person pronoun; Then a third query is displayed, which is used to characterize the third visual feature information of a third person in the image to be processed. The third person is different from the first person and the second person. Receive a third response from the user; If the third response indicates that the third visual feature information matches the first person pronoun, then the third person is determined to be the target person; If the third response indicates that the third visual feature information does not match the first person pronoun, then a second prompt message is output, which prompts the user to re-enter the statement.

10. The method according to claim 6, characterized in that, The number of the at least one visual feature information is multiple, and the display of the first prompt information includes: The fourth question is displayed, which includes multiple visual feature information, which is used to characterize multiple visual features of the first person. The first instruction information received from user input includes: Receive the fourth response message input by the user; The step of determining the target person indicated by the first person pronoun in the image to be processed based on the first indication information includes: If the fourth response indicates that the multiple visual feature information matches the first person's pronoun, then the first person is determined to be the target person.

11. The method according to claim 5, characterized in that, The number of the at least one visual feature information is multiple, and the display of the first prompt information includes: The fifth query is displayed. The fifth query includes multiple visual feature information. The multiple visual feature information is used to characterize the visual features of multiple people in the image to be processed. The multiple visual feature information corresponds one-to-one with the multiple people. The first instruction information received from user input includes: The system receives a fifth response from the user, which is used to indicate the fourth visual feature information among the plurality of visual feature information. The step of determining the target person indicated by the first person pronoun in the image to be processed based on the first indication information includes: If the fourth visual feature information indicated by the fifth response matches the first person pronoun, the person corresponding to the fourth visual feature information is determined to be the target person.

12. The method according to any one of claims 11, characterized in that, The method further includes: If the fifth response indicates that any one of the multiple visual feature information does not match the first person pronoun, then a third prompt message is output, which prompts the user to re-enter the statement.

13. The method according to any one of claims 5 to 12, characterized in that, The method further includes: Establish a correspondence between the target person identifier and the first person pronoun; Add the correspondence between the target person identifier and the first person pronoun to the first correspondence.

14. The method according to any one of claims 1 to 13, characterized in that, Before determining that the user-input statement contains a first-person pronoun, the process also includes: Obtain the image to be processed and the statement input by the user; The method further includes: Obtain the intent represented by the statement, the intent corresponding to the first person pronoun, and the intent being the intention to eliminate; After determining the target person indicated by the first person pronoun in the image to be processed based on the first indication information, the method further includes: Based on the target person, perform the elimination task corresponding to the intent on the image to be processed to obtain the result image; The resulting image is displayed.

15. An electronic device, characterized in that, include: Processor, memory, and interface; The processor, the memory, and the interface cooperate with each other to enable the electronic device to perform the method as described in any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1 to 14.

Images