Photo processing method and electronic equipment

Abstract

The invention provides a photo processing method and electronic equipment, and relates to the technical field of terminals, the method comprises the steps that the electronic equipment receives a first voice input by a user, a first text corresponding to the first voice is displayed on a first interaction interface, and the first text comprises information indicating an owner; the electronic device displays a first target photo and a first control on the first interaction interface in response to the first voice, the electronic device displays a thumbnail of a target video in response to a first operation input by a user to the first control, and the target video is generated by the electronic device based on the first target photo including the first face in the gallery application. By adopting the scheme, the electronic equipment can determine the instruction for indicating the owner based on the voice input by the user, and responds to the instruction to generate the video based on the picture of the owner in the picture library application, so that the man-machine interaction efficiency is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of terminal technology, and in particular to a photo processing method and an electronic device. Background Art

[0002] With the development of terminal technology, voice recognition technology has become increasingly mature, and voice input has become increasingly important due to its high naturalness and effectiveness in interaction. Voice interaction applications in electronic devices are also known as a function that users often use. Users can interact with electronic devices (such as mobile phones, tablets, smart watches, etc.) through voice to complete various operations such as command input, information query, and voice chat. Summary of the invention

[0003] The present application provides a photo processing method and an electronic device, which can determine an instruction for instructing an owner based on a voice input by a user, and generate a video based on photos of the owner included in a gallery application in response to the instruction, thereby improving the efficiency of human-computer interaction.

[0004] The embodiments of the present application adopt the following technical solutions:

[0005] In a first aspect, a photo processing method is provided, which is applied to an electronic device, and includes: the electronic device receives a first voice input by a user, and displays a first text corresponding to the first voice on a first interactive interface. That is, the first interactive interface of the electronic device can display the text content corresponding to the voice input by the user.

[0006] On this basis, the first text includes information indicating the owner, that is, the first voice includes information for indicating the owner. In response to the first voice, the electronic device displays the first target photo and the first control on the first interactive interface. The first target photo includes the first face, and the first face is set with the owner tag in the gallery application of the electronic device.

[0007] In response to a first operation input by a user to a first control, the electronic device displays a thumbnail of a target video; wherein the target video is generated by the electronic device based on a first target photo including a first face in a gallery application. In this way, by recognizing a first voice input by a user, information indicating the owner of the device is determined; and then, in response to the first voice, a video can be generated from the photo including the owner of the device in the gallery application of the electronic device, thereby improving the efficiency of human-computer interaction.

[0008] Optionally, the electronic device receives the first voice input by the user and may not display the first interactive interface. The first voice includes information for indicating the owner, and in response to the first voice, a photo including the owner is obtained from the gallery application and a video is generated.

[0009] In a possible implementation of the first aspect, the method also includes: the electronic device receives a second operation of the user deleting the owner tag in the gallery application; in response to a second voice input by the user, the electronic device displays a second text corresponding to the second voice on the first interactive interface, and the second text includes information indicating the owner.

[0010] In response to the second voice, the electronic device displays the second target photo, the third target photo and the first control in the first interactive interface; wherein the second target photo includes the second face, and the third target photo includes the third face. In response to the user's third operation, the second target photo is unchecked. In response to the user's fourth operation on the first control, the electronic device displays a thumbnail of the target video, which is generated by the electronic device based on the third target photo including the third face in the gallery application. In this way, after the user deletes the owner tag in the gallery application, the electronic device can display photos of multiple owners in the first interactive interface and check all the owners' photos by default. On this basis, in response to the user's unchecking operation, the owner's photo selected by the user (i.e., the third target photo) can be retained, and a video can be generated based on the owner's photo selected by the user, which improves the efficiency of human-computer interaction while also enhancing the user's sense of participation.

[0011] In a possible implementation manner of the first aspect, the second target photo is determined by the electronic device based on a front-facing photo containing the second person's face among all photos included in a gallery application.

[0012] In a possible implementation of the first aspect, the number of front-camera photos containing the second person's face is the largest number among all front-camera photos containing faces; or, the number of front-camera photos containing the second person's face is greater than or equal to a selfie count threshold.

[0013] In a possible implementation of the first aspect, the method also includes: the electronic device displays a second interactive interface, the second interactive interface includes a first location and a second location; wherein the first location is a home location learned by the electronic device, the second location is a company location learned by the electronic device, and the photo containing the third person's face has appeared in both the first location and the second location.

[0014] In a possible implementation of the first aspect, a second control is further displayed on the first interactive interface; the method further includes: in response to a fifth operation input by the user on the second control, the electronic device expands and displays multiple photos including the first person's face. In this way, the user can see the multiple photos including the first person's face obtained by the electronic device from the gallery application, thereby improving the user's sense of participation.

[0015] In a possible implementation of the first aspect, before displaying the first target photo, the method further includes: the electronic device determines that the first voice includes an instruction to instruct the owner, and queries the owner's face ID; the electronic device determines that the owner's face ID is a first face according to the confidences corresponding to different face IDs; the confidence of the first face is a first confidence, and the first confidence is greater than the confidences of other faces; the electronic device screens photos including the first face and determines the first target photo. In this way, the owner's face ID is queried by confidence, so as to screen out photos of the first face, thereby improving the reliability of screening photos of the first face.

[0016] In a possible implementation of the first aspect, before the second target photo and the third target photo are displayed on the first interactive interface, the method also includes: the electronic device determines that the second voice includes an instruction to instruct the owner, and queries the face ID of the owner; the electronic device determines that the face ID of the owner is the second face and the third face according to the confidence levels corresponding to different face IDs; the electronic device screens the photos including the second face and the third face, and determines the second target photo and the third target photo.

[0017] In a possible implementation of the first aspect, before the electronic device receives a first voice input by a user, the method also includes: the electronic device generates multiple photo collections of different persons based on different face ID analysis results; the electronic device sets an owner tag for the photo collection containing the first face; and the electronic device determines that a first confidence level of the first face is a first preset value.

[0018] In a possible implementation of the first aspect, before the electronic device receives the second voice input by the user, the method also includes: the electronic device obtains the gender and age corresponding to each face in the multiple faces included in all the front-facing photos in the gallery application; the electronic device selects a first group of faces according to the gender and age corresponding to each face in the multiple faces; the gender corresponding to each face in the first group of faces is the same as the gender corresponding to the second face, and the difference between the age corresponding to each face and the age corresponding to the second face is within a preset range; the electronic device uses the ratio of the number of front-facing photos of the second face to the number of front-facing photos of the first group of faces as the second confidence level; the second confidence level is less than the first preset value; the electronic device reads the photos in the gallery application and obtains the shooting location and shooting time of the photos corresponding to each face ID in all the face IDs; the electronic device determines the target face ID corresponding to the photos taken at the first location and the second location at the same time according to the shooting location and shooting time of the photos corresponding to each face ID in all the face IDs; the target face ID includes a third face; the electronic device sets the third confidence level corresponding to the third face to the second preset value, and the second preset value is less than the first preset value.

[0019] In a possible implementation manner of the first aspect, the method also includes: the electronic device determines whether the value of the highest confidence among the confidences corresponding to different face IDs is a first preset value; when the value of the highest confidence is the first preset value, returns the face ID with the confidence value of the first preset value as the face ID of the owner; when the value of the highest confidence is not the first preset value, returns all face IDs with confidence values ​​exceeding a preset threshold as the face ID of the owner.

[0020] In a possible implementation of the first aspect, the electronic device determines the target face ID corresponding to the photos taken at the first location and the second location based on the shooting location and shooting time of the photos corresponding to each face ID in all face IDs, including: the electronic device determines that the distance between the shooting location and the first location is less than or equal to the first preset distance, and determines that the distance between the shooting location and the second location is less than or equal to the second preset distance based on the shooting location of the photos corresponding to each face ID in all face IDs; the electronic device determines the frequency of taking photos for each face ID in all face IDs at the first location, and determines the frequency of taking photos at the second location; the electronic device screens out the target face ID corresponding to the photos taken at the first location and the second location based on the shooting time; wherein the frequency of the target face ID taking photos at the first location is greater than the preset frequency, and the frequency of taking photos at the second location is greater than the preset frequency.

[0021] In a possible implementation of the first aspect, before the electronic device determines the target face ID corresponding to the photos taken at the first location and the second location based on the shooting location and shooting time of the photos corresponding to each face ID in all face IDs, the method further includes: the electronic device determines that the distance between the first location and the second location is greater than or equal to a preset distance. In this way, when the distance between the first location and the second location is greater than or equal to the preset distance, it can be considered that the first location and the second location are relatively accurate, so as to facilitate the subsequent execution of the process of determining the owner.

[0022] In a possible implementation of the first aspect, before the electronic device receives the second voice input by the user, the method further includes: the electronic device determines the age range of the owner according to the user's usage habits. The method further includes: if the number of target face IDs is less than or equal to a preset number, the electronic device uses the corresponding face ID in the target face ID within the age range as the face ID of the owner.

[0023] Optionally, if the number of target face IDs is greater than a preset number, the electronic device ends the process of determining the owner to ensure the accuracy of the determined owner.

[0024] In a second aspect, an electronic device is provided, which has the function of implementing any one of the functions in the first aspect, and the function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0025] In a third aspect, an electronic device is provided, comprising: a memory and one or more processors, and a display screen; the memory stores computer program code, and the computer program code includes computer instructions; when the computer instructions are executed by the processor, the electronic device executes the above-mentioned first aspect or any one of the methods in the first aspect.

[0026] In a fourth aspect, a chip system is provided, comprising: at least one processor and an interface, the interface being used to receive instructions and transmit them to at least one processor; at least one processor executes the instructions so that the electronic device executes any one of the methods described in the first aspect.

[0027] In a fifth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer can execute any of the methods described in the first aspect.

[0028] In a sixth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the methods described in the first aspect.

[0029] Among them, the technical effects brought about by any implementation method in the above-mentioned second to sixth aspects can refer to the technical effects brought about by different implementation methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0031] Figure 2A A scenario diagram of waking up a mobile phone provided in an embodiment of the present application Figure 1 ;

[0032] Figure 2B A second schematic diagram of a scenario for waking up a mobile phone provided in an embodiment of the present application;

[0033] Figure 2C A scenario diagram of waking up a mobile phone provided in an embodiment of the present application Figure 3 ;

[0034] Figure 3 A voice interaction interface provided in the embodiment of the present application Figure 1 ;

[0035] Figure 4 Schematic diagram 2 of a voice interaction interface provided in an embodiment of the present application;

[0036] Figure 5 A schematic diagram of a plurality of photo collections included in a gallery application provided in an embodiment of the present application;

[0037] Figure 6 A schematic diagram of adding names to a collection of portrait photos in a gallery application provided in an embodiment of the present application;

[0038] Figure 7 A schematic diagram of a process for determining the owner of a device provided in an embodiment of the present application Figure 1 ;

[0039] Figure 8 A second schematic diagram of a process for determining a machine owner provided in an embodiment of the present application;

[0040] Fig. 9 A schematic diagram of a user trajectory provided in an embodiment of the present application;

[0041] Fig.10 A schematic diagram of an interactive interface including a home location and a company location provided in an embodiment of the present application;

[0042] Fig.11 A voice interaction interface provided in the embodiment of the present application Figure 3 ;

[0043] Fig.12 A schematic diagram of a software framework of an electronic device provided in an embodiment of the present application;

[0044] Fig.13 A schematic diagram of a photo processing process provided in an embodiment of the present application;

[0045] Fig.14 A schematic diagram of the structure of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The embodiment of the present application provides a photo processing method, which is applied to electronic devices supporting voice interaction functions such as mobile phones, tablet computers, and smart watches. Among them, the electronic device can provide a voice interaction interface to the user, and the voice interaction interface can receive the voice input by the user. In response to the voice input by the user, the electronic device determines an instruction for instructing the owner. Then, in response to the instruction for instructing the owner, the electronic device obtains photos including the owner from the gallery application and generates a video.

[0047] To summarize, by adopting the solution of the embodiments of the present application, the electronic device can determine the instructions for instructing the owner based on the voice input by the user, and thus generate a video based on the photos of the owner included in the gallery application in response to the instructions for instructing the owner, thereby improving the efficiency of human-computer interaction.

[0048] For example, the electronic device in the embodiments of the present application may be a mobile phone, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA). The electronic devices supporting voice interaction functions such as augmented reality (AR) / virtual reality (VR) devices, etc., are not particularly limited to the specific form of the electronic device in the embodiments of the present application.

[0049] refer to Figure 1 , is a hardware structure diagram of an electronic device 100 provided in an embodiment of the present application. Figure 1 As shown, the electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 1~N195, etc.

[0050] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0051] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0052] It is understandable that the interface connection relationship between the modules shown in this embodiment is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0053] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device 100 through the power management module 141.

[0054] The power management module 140 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0055] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0056] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0057] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0058] The electronic device 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen and the application processor. The ISP is used to process the data fed back by the camera 193. The camera 193 is used to capture a still image or a video. In some embodiments, the electronic device may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0059] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0060] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0061] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170 , the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0062] The buttons 190 include a power button, a volume button, etc. The buttons 190 may be mechanical buttons or touch buttons. The electronic device 100 may receive button inputs and generate key signal inputs related to user settings and function controls of the electronic device 100. The motor 191 may generate vibration prompts, and the motor 191 may be used for incoming call vibration prompts or for touch vibration feedback. The indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc. The SIM card interface 195 is used to connect a SIM card. The SIM card may be brought into contact with or separated from the electronic device 100 by inserting the SIM card interface 195 or pulling the SIM card interface 195 out. The electronic device 100 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1.

[0063] Of course, the electronic device 100 provided in the embodiment of the present application may also include one or more components such as a positioning module 181, a button 190, a motor 191, an indicator 192 and a SIM card interface 195, without limitation.

[0064] The methods in the following embodiments can all be implemented in an electronic device 100 having the above hardware structure. It is understandable that the photo processing method provided in the embodiments of the present application can be executed by electronic devices that support voice interaction, such as mobile phones, tablet computers, smart watches, etc., or by chips, chip systems or processors that can implement the photo processing method, or by logic modules or software that can implement all or part of the functions of electronic devices, without limitation. The following embodiments use mobile phones as the execution subject to introduce the solutions provided in the embodiments of the present application in detail.

[0065] In some scenarios, many mobile phones support voice control functions. Specifically, the mobile phone collects user voice through a microphone, analyzes and recognizes the user voice, and executes instructions corresponding to the user voice, so that the user can control the mobile phone through voice.

[0066] Generally speaking, in order to save power consumption of the mobile phone and avoid false triggering, the voice control function needs to be turned on before it can be used. For example, the user inputs a preset word (or wake-up word) into the mobile phone through voice to wake up the mobile phone. After the mobile phone is awakened, it can execute the instructions corresponding to the user's voice, that is, the voice control function is turned on. For example, the user can turn on or off the voice control function by turning on or off the preset switch in the human-computer interaction interface of the mobile phone.

[0067] In different mobile phones, the voice control function may have different names, such as "smart assistant", "smart application", "voice assistant", etc., and the specific implementation of voice control functions with different names may also be different.

[0068] Taking "Smart Assistant" as an example, the following introduces several different implementation forms of the voice control function.

[0069] Before using the smart assistant to control the phone, the user needs to input the wake-up word into the phone to wake up the phone. Generally speaking, before the phone is awakened, the phone's microphone works in a power-saving mode (such as searching for signals at a lower function) to pick up sounds from the surrounding environment. The voice collected by the microphone is only detected for wake-up words at the kernel layer, and the corresponding recording channel of the smart assistant is not started in the phone's system and driver.

[0070] In response to user operations (such as receiving a wake-up word input by the user's voice), the phone is awakened, and the corresponding recording channel of the smart assistant is started in the system and driver. After the phone is awakened, the voice (audio stream) collected by the microphone is sent to the smart assistant application for processing through the recording channel corresponding to the smart assistant. In this way, the phone can execute the instructions corresponding to the user's voice and enable the user to control the phone through voice. Optionally, the phone can also realize functions such as dialogue with the user.

[0071] refer to Figure 2A , which shows a schematic diagram of a scenario for waking up a mobile phone. Figure 2A As shown, the mobile phone displays the desktop interface, and the mobile phone receives the voice wake-up word (such as "Hello, YOYO") input by the user. In response to the wake-up word, the mobile phone is awakened. Exemplarily, after the mobile phone is awakened, a voice interaction interface such as interface 101 (or referred to as the first interaction interface) is displayed. Among them, interface 101 is used to display the conversation content when the mobile phone and the user have a voice conversation.

[0072] refer to Figure 2B , which shows another scenario schematic diagram of waking up the mobile phone. Figure 2B As shown, the mobile phone displays the desktop interface, and the mobile phone receives a long press operation of the power button (or power button) by the user. In response to the operation, the mobile phone is awakened. Exemplarily, after the mobile phone is awakened, a voice interaction interface such as interface 101 is displayed.

[0073] In some embodiments of the present application, the smart assistant has a corresponding function entrance in the mobile phone, such as a control center, a setting application, etc. The user can enter the corresponding function entrance and turn on or off the function of the smart assistant by turning on or off a preset switch in the human-computer interaction interface of the mobile phone.

[0074] refer to Figure 2C , which shows another scenario schematic diagram of waking up the mobile phone. Figure 2C As shown, in response to the user opening the setting function of the mobile phone (for example, clicking the icon of the setting application on the desktop), the mobile phone displays the setting interface 102, and the setting interface 102 includes the smart assistant option 103, and the smart assistant option 103 is used to turn on or off the voice control function. Figure 2C As shown, in response to the user's activation operation of the smart assistant option 103, the mobile phone is awakened. Exemplarily, after the mobile phone is awakened, a voice interaction interface such as interface 101 is displayed.

[0075] Optionally, after the phone is awakened, the user can give a command to the phone by inputting voice into the phone, and the phone executes the command corresponding to the voice. After the phone executes a command, or if the phone does not receive a command from the user through voice within a period of time (for example, within 8 seconds) after it is awakened, the phone will no longer respond to the user's voice command. For example, the phone will turn off the recording channel corresponding to the smart assistant. The user needs to wake up the phone again before he can give commands to the phone by inputting voice into the phone again. In other words, after the phone is awakened, it enters a "short reception" state and can respond to the user's voice commands within a short period of time (for example, within 8 seconds).

[0076] Optionally, when the mobile phone is connected to the network, it supports entering a continuous conversation scenario after being awakened, and the user can have a continuous conversation with the mobile phone. After each broadcast, the mobile phone will continue to receive the sound without repeated awakening until the user exits the continuous conversation through commands such as "exit".

[0077] Exemplarily, when the mobile phone is awakened and enters the continuous conversation scene, the user can have a continuous conversation with the mobile phone in the interface 101. The interface 101 is used to display the content of the continuous conversation between the user and the mobile phone. Optionally, the interface 101 includes prompt information, which is used to prompt the user that the mobile phone enters the continuous conversation scene. For example, the prompt information may be "Welcome".

[0078] Optionally, the interface 101 further includes a prompt icon 10a, which is used to prompt the user that the mobile phone enters a continuous sound receiving state. In actual implementation, the prompt icon 10a can be a microphone sound receiving icon, or a "robot sound receiving icon", etc., without limitation.

[0079] After the phone is awakened, it enters a continuous sound receiving state. The user can input voice into the phone, and the phone parses and recognizes the voice input by the user and performs operations corresponding to the voice. For example, the phone can receive voice input by the user to indicate the owner of the phone. In response to the voice input by the user, the phone can obtain photos including the owner of the phone from the gallery application and generate a video. Among them, the "owner" can be understood as the main user of the phone, that is, the owner of the phone.

[0080] For example, Figure 3 As shown in (a) of FIG. 1 , during the process of the user inputting voice, the mobile phone displays an interface 104, and the interface 104 includes a prompt icon 10b, which is used to prompt the user that the microphone is recording the sound. For example, Figure 3 As shown in (a) of FIG. 1 , the prompt icon 10b is a transparent circle displayed on the basis of the prompt icon 10a. On this basis, the mobile phone receives a voice 1 input by the user, and the voice 1 includes instructions for instructing the owner. For example, Figure 3 As shown in (a) of FIG. 1 , the voice 1 (or the first voice) may be “create a video from my photo”; wherein the “I” in the voice 1 is an instruction to the owner of the device. Figure 3 As shown in (b) of FIG. 1 , in response to voice 1, the mobile phone displays interface 105, and interface 105 includes text content 1 (or referred to as first text), and the text content 1 includes text indicating the owner. For example, the text content 1 may be "create a video from my photo". For example, "I" in the above text content 1 is text indicating the owner.

[0081] Exemplarily, the mobile phone can recognize the voice 1 received from the user input, thereby recognizing "I" as an instruction to instruct the owner of the device, and execute an operation corresponding to the instruction. Optionally, when the voice 1 is matched to the database, the mobile phone recognizes "I" as an instruction to instruct the owner of the device. For example, the voice 1 includes "I" and the database includes "I", so the mobile phone can recognize "I" as an instruction to instruct the owner of the device. It should be noted that the mobile phone can perform semantic understanding of the voice 1, and the voice 1 may not be completely consistent with the database, as long as the semantics are the same or close, that is, the voice 1 matches the database, but it does not have to be completely consistent. Exemplarily, in Figure 3 In the scenario shown, the voice 1 input by the user includes "I", and the database includes "myself", "myself", and "myself", and "I" has the same semantics as "myself", "myself", and "myself", that is, the two match, so the mobile phone can recognize that "I" is an instruction used to instruct the owner, thereby determining that "I" is used to instruct the owner.

[0082] Alternatively, the mobile phone may recognize and analyze the voice 1 to identify that "I" is a command for indicating the owner, thereby determining that "I" is used to indicate the owner. Exemplarily, after the mobile phone receives the voice 1 input by the user, it uses automatic speech recognition (ASR) technology to convert the user's voice into text, and then uses natural language understanding (NLU) technology to perform intent recognition on the converted text, thereby recognizing that "I" is a command for indicating the owner, thereby determining that "I" is used to indicate the owner.

[0083] After the mobile phone displays the text content 1, Figure 4 As shown in (a) in the figure, the mobile phone displays an interface 106, which includes text content 2, and the text content 2 is used to indicate the response of the voice 1. For example, the text content 2 can be "OK, the following materials have been found for you." On this basis, the mobile phone also displays a message 1 in the interface 106, and the message 1 includes a photo of the owner (or a first target photo including the first face). Optionally, the interface 106 also includes a control 1 (or a second control) and a control 2 (or a first control), and the control 1 is used to trigger the mobile phone to display all photos related to the owner, and the control 2 is used to trigger the mobile phone to generate a video including the photos of the owner. Exemplary. Control 1 can be called a "view all" control, for example, and control 2 can be called a "generate video" control, for example.

[0084] Exemplarily, in response to the user's operation on control 1 (or the fifth operation), the mobile phone expands to display multiple photos including the owner of the phone (for example, multiple photos including the first person's face are displayed).

[0085] Exemplarily, in response to a user operation on control 2 (or a first operation, such as a click operation, a long press operation, a sliding operation, etc.), Figure 4 As shown in (b) of FIG. 1 , the mobile phone displays an interface 107, which includes a thumbnail of video 1 (or target video). Video 1 is a video of the mobile phone based on Figure 4 (a) in the figure includes a photo of the owner. Optionally, the interface 107 includes a control 3 for triggering the mobile phone to play the video 1. For example, the mobile phone can receive a click operation, a long press operation, a sliding operation, etc. of the user on the control 3. The following mainly uses the click operation as an example for explanation. In response to the user's click operation on the control 3, the mobile phone plays the video 1. The content of the video 1 is not shown, and the actual implementation shall prevail.

[0086] In summary, by adopting the solution of the embodiment of the present application, after the mobile phone is awakened, the mobile phone can enter a continuous conversation scene, and the user can have a continuous conversation with the mobile phone. On this basis, in response to the voice 1 input by the user, the mobile phone determines that "I" is used to indicate the owner, and obtains photos including the owner from the gallery application. Then, the mobile phone generates a video related to the owner based on the photos including the owner, thereby improving the efficiency of human-computer interaction.

[0087] In some embodiments of the present application, the mobile phone may first identify the owner of the device based on the photos included in the gallery application. On this basis, when the mobile phone recognizes that the user's voice includes an instruction for indicating the owner of the device, the mobile phone responds to the instruction for indicating the owner, obtains photos including the owner of the device from the gallery application, and generates a video based on the photos including the owner of the device.

[0088] Optionally, the mobile phone can identify the owner based on photos in the gallery application when the phone is charging and the screen is off; or, the mobile phone can identify the owner based on photos in the gallery application when the screen is black, etc., without restriction.

[0089] As an example, the mobile phone can identify the owner based on the photos in the gallery application in the following three ways. It should be noted that the following three ways are only some examples of this application and do not constitute a limitation of this application. Other ways of identifying the owner's photos from the gallery application should also fall within the scope of protection of this application.

[0090] Method 1: The phone determines the owner based on the name set in the gallery application to indicate the owner.

[0091] For example, Figure 5As shown, the gallery application can divide the photos saved in the gallery application into multiple photo collections according to different photo types or themes (such as time, portraits, landscapes, food, pets, etc.), and each photo collection in the multiple photo collections includes photos of a type or theme. For example, the portrait photo collection includes photos containing portraits in the gallery application; the landscape photo collection includes photos containing natural landscapes in the gallery application; the food photo collection includes photos containing food in the gallery application; and the pet photo collection includes photos containing pets in the gallery application. It is understandable that the types or themes of different photos include but are not limited to the above-mentioned time, portraits, landscapes, food, pets, etc., and may also include other types or themes, such as architecture, etc., without limitation.

[0092] In the above embodiment, the photos in the gallery application are divided into different photo collections according to the types or themes of different photos, which can facilitate users to view multiple photos of different types or different themes without having to flip through all the photos, making user operations more convenient and improving user experience.

[0093] Taking a collection of portrait photos as an example, for example, Figure 6 As shown, by performing face clustering algorithm and other analyses on all the photos in the gallery, photo collections of different persons can be generated based on the analysis results of different face IDs, such as a photo collection of person 1, a photo collection of person 2, a photo collection of person 3, etc., which will not be elaborated here.

[0094] For example, the mobile phone can accept the user's input operation to set corresponding nouns for different character photo collections, and the mobile phone can also set corresponding names for different character photo collections after using a character relationship analysis algorithm. Figure 6 As shown, the name corresponding to the photo collection of person 1 set in the mobile phone is "Me", the name corresponding to the photo collection of person 2 set in the mobile phone is "Son", and the name corresponding to the photo collection of person 3 set in the mobile phone is "Wife".

[0095] It should be noted that Figure 6 The names corresponding to the photo collections of different characters shown are only examples and do not constitute a limitation of the present application. Of course, the mobile phone can also set other names corresponding to the photo collections of different characters. For example, taking the photo collection of character 1 as an example, the mobile phone can also set the name corresponding to the photo collection of character 1 as "myself", "myself", etc., without limitation.

[0096] It is understandable that the pre-configured names "I", "myself", "myself", etc. are all names used to indicate the owner. Therefore, based on the names "I", "myself", "myself", etc. set in the gallery application, the mobile phone can determine the photo collection of the owner of the photo collection corresponding to the name, that is, the face ID corresponding to the photo collection is the owner.

[0097] Optionally, after the mobile phone determines the owner through the above method 1, the mobile phone can also record the relevant information of the owner. Exemplarily, the relevant information of the owner includes: owner A, face ID-A, confidence A (or understood as the first confidence), photo source A. Among them, confidence can be understood as reliability, that is, the accuracy of the determined owner (or it can also be understood as the priority corresponding to method one; the higher the confidence, the higher the priority). Exemplarily, in the above method 1, since the owner is determined by the mobile phone based on the name set in the gallery application to indicate the owner, the above confidence A can be set to a value that is guaranteed to be greater than the confidence determined by other methods, such as the confidence determined by other methods must be less than 100%, that is, the confidence is set to 1 (that is, the mobile phone has the highest priority in determining the owner through method 1). In addition, the above photo source A can be expressed as: the name set in the gallery application to indicate the owner.

[0098] Method 2: The mobile phone determines the owner based on the selfie photos (or front-facing photos) in the gallery application. The selfie photos refer to photos taken with the front-facing camera of the mobile phone.

[0099] In some embodiments of the present application, if the number of self-portraits corresponding to face ID-1 exceeds a preset threshold (or self-portrait threshold), the face corresponding to face ID-1 is determined as the owner of the device. Alternatively, if the number of self-portraits corresponding to face ID-1 is the largest number of self-portraits included in all faces, the face corresponding to face ID-1 is determined as the owner of the device.

[0100] The preset threshold can be adjusted according to the number of photos in the gallery application, such as taking X% of the number of photos in the gallery application. The preset threshold can also be set according to other actual conditions without limitation. For example, the preset threshold can be set to other suitable values ​​such as 5, 10, etc. without limitation.

[0101] Alternatively, in other embodiments of the present application, if the gallery application includes selfie photos of multiple faces, in order to improve the accuracy of determining the owner of the device, for selfie photos of multiple faces, the mobile phone can determine the owner based on the facial information in the selfie photos.

[0102] Exemplarily, facial information may include selfie frequency, selfie time, face ID (or tag ID, or face identification), face gender, face age, etc.; or, facial information may also include selfie time, etc., without limitation.

[0103] Optional, such as Figure 7As shown, the mobile phone may include the following steps to determine the owner based on facial information: Step 1, the mobile phone queries the selfie photos in the gallery application, and queries the facial information corresponding to each face in the selfie photo from the selfie table. Exemplarily, the facial information of each face in the selfie photo is stored in the selfie table, and the selfie table is obtained after the gallery application of the mobile phone analyzes the photos when the mobile phone is off. Step 2, the mobile phone determines the selfie frequency of each face ID based on the face ID in the facial information. Among them, the selfie frequency includes the number of selfies and / or the frequency of selfies. Optionally, after executing steps 1 and 2, it may also include step 3, the mobile phone filters out the face IDs that do not meet the conditions (i.e., filters out the first group of faces) based on the face gender and face age in the face information.

[0104] Exemplarily, the mobile phone reads the gender and age of the user from the user profiling table. The user profiling table is learned by the mobile phone based on the records of the user using the application (APP) installed in the mobile phone, and the user profiling table stores the gender and age or / age range of the user predicted by the mobile phone. In some embodiments of the present application, if the gender of the face in the above-mentioned face information does not match the gender in the user profiling table, the mobile phone filters out the corresponding face ID. Alternatively, the mobile phone calculates the average age of each face ID, and if the difference between the average age and the age in the user profiling table exceeds a preset value, the mobile phone filters out the corresponding face ID. There is no specific limitation on the preset value, which is subject to the actual setting. Exemplarily, the preset value can be 20, etc., without limitation.

[0105] Optionally, assuming that in the gallery application, face ID1 includes age 1, age 2, age 3...age n in selfie photos, the average age of face ID1 can be expressed as (age 1+age 2+age 3+...+age n) / n, where n is a positive integer.

[0106] Alternatively, in other embodiments of the present application, if the age difference between the face corresponding to the face ID and the age in the user habit table exceeds a preset threshold or / is not within the age range in the user habit table, the mobile phone filters out the face ID.

[0107] Step 4: The mobile phone determines the owner based on the selfie frequency of the face ID. For example, assume that the face ID includes face ID1 and face ID2. The mobile phone calculates the frequency ratio of face ID1 and face ID2 respectively, and uses the frequency ratio as the confidence level (or as the second confidence level). If the confidence level is greater than the confidence level threshold, the face ID corresponding to the confidence level is determined as the owner.

[0108] Exemplarily, in the above step 2, the mobile phone determines the selfie frequency of each face ID. For example, the selfie frequency of face ID1 is 10, and the selfie frequency of face ID2 is 20, then the frequency ratio of face ID1 is 10 / (10+20)=0.33, that is, the confidence of face ID1 is 0.33; correspondingly, the frequency ratio of face ID2 is 20 / (10+20)=0.67, that is, the confidence of face ID2 is 0.67.

[0109] Optionally, if the confidence levels of face ID1 and face ID2 are both greater than the confidence threshold, the mobile phone determines face ID1 and face ID2 as the owner.

[0110] In order to ensure the accuracy of the owner determined by the mobile phone, the confidence level can be set higher, for example, the confidence level can be set to 0.4. Of course, the confidence level can also be a value greater than 0.4 without limitation.

[0111] Optionally, after the mobile phone determines the owner through the above method 2, the mobile phone can also record the relevant information of the owner. Exemplarily, the relevant information of the owner includes: owner B, face ID-B, confidence B, photo source B. Among them, in the above method 2, owner B can be the user corresponding to face ID2, face ID-B can be face ID2, confidence B can be 0.67, and photo source B can be "selfie".

[0112] It can be understood that if the mobile phone determines two owners through the above method 2, the relevant information of the owners recorded by the mobile phone includes: owner B, face ID-B, confidence B, photo source B; owner C, face ID-C, confidence C, photo source C.

[0113] Method 3: The phone determines the owner based on the location and time the photos were taken in the gallery app, as well as the location of the user's home and company.

[0114] For example, Figure 8 As shown, the mobile phone may include the following steps to determine the owner based on method 3: Step 1, the mobile phone reads the photos in the gallery application and obtains the shooting location and shooting time of the photos corresponding to each face ID in all face IDs. Step 2, the mobile phone reads the first location (which can be understood as the location of the home) and the second location (which can be understood as the location of the company) determined by the mobile phone. The first location and the second location are learned by the mobile phone based on the user's movement trajectory.

[0115] Step 3. The mobile phone calculates the distance between the first location and the second location, and determines whether the distance between the first location and the second location is greater than the preset distance. Exemplarily, if the distance between the first location and the second location is greater than the preset distance, the mobile phone continues to execute step 4. Correspondingly, if the distance between the first location and the second location is less than the preset distance, the mobile phone ends the process of determining the owner of the phone. Among them, there is no specific limitation on the preset distance, and it can be set according to actual conditions. Exemplarily, the preset distance can be 3 kilometers (km), or the preset distance can be set to M times the radius of the corresponding range of the first location; or the preset distance can be set to M times the radius of the corresponding range of the second location, without limitation. Among them, M is a positive integer greater than 1.

[0116] It should be noted that, under normal circumstances, the distance between the first location and the second location is relatively far, generally greater than 3 km. Therefore, in order to avoid errors in the first location and the second location determined by the mobile phone, the mobile phone can calculate the distance between the first location and the second location. If the distance is greater than the preset distance, it means that the first location and the second location determined by the mobile phone are relatively accurate, and the mobile phone can continue to perform the following steps. Correspondingly, if the distance is less than the preset distance, it means that there is an error in the first location and the second location determined by the mobile phone, so the mobile phone can end the process of determining the owner.

[0117] Step 4, the mobile phone determines the target face ID (or target face identification) corresponding to the photos taken at the first location and the second location at the same time. Exemplarily, the mobile phone determines the face ID whose shooting location is at the first location based on the shooting location of the photos corresponding to each face ID. On this basis, the mobile phone determines whether to shoot at the second location based on the face ID shot at the first location. If the face ID shot at the first location is also shot at the second location, the mobile phone uses the face ID as the target face ID (which may include multiple face IDs), that is, the target face ID has taken photos at both the first location and the second location. In other words, the mobile phone can find the face ID (i.e., the target face ID) corresponding to the photos taken at the first location and the second location at the same time through the above step 4.

[0118] Optionally, if the distance between the shooting location and the first location is less than the first preset distance, the mobile phone determines that the shooting location is at the first location. Correspondingly, if the distance between the shooting location and the second location is less than the second preset distance, the mobile phone determines that the shooting location is at the second location. The first preset distance and the second preset distance are not limited, and are subject to actual settings. The first preset distance and the second preset distance may be the same or different. For example, the first preset distance may be 200 meters, and the second preset distance may be 500 meters.

[0119] Step 5: The mobile phone determines the first frequency of the target face ID taking photos at the first location, and the second frequency of the target face ID taking photos at the second location. If the first frequency and the second frequency are both greater than the preset frequency, the face ID that meets the above conditions is determined as the face ID of the owner. The preset frequency is not specifically limited and is subject to the actual setting. For example, the preset frequency can be 2.

[0120] Exemplarily, assuming that the target face ID includes face ID1, the first frequency of face ID1 taking photos at the first location is 10, and the second frequency of taking photos at the second location is 2. It can be seen that the first frequency and the second frequency of face ID1 are both greater than the preset frequencies, so the mobile phone determines that face ID1 is the owner.

[0121] As another example, assuming that the target face ID includes face ID1 and face ID2, the first frequency of face ID1 taking photos at the first location is 10, and the second frequency of face ID2 taking photos at the second location is 2; the first frequency of face ID2 taking photos at the first location is 5, and the second frequency of face ID2 taking photos at the second location is 5. It can be seen that the first frequency and second frequency corresponding to face ID1 and face ID2 are both greater than the preset frequencies, so the mobile phone determines the owner based on face ID1 and face ID2.

[0122] In some embodiments of the present application, when the mobile phone determines that the number of face IDs that have taken photos at both the first location and the second location is less than or equal to threshold 1, the mobile phone determines the owner based on the gender of the face. Exemplarily, if the gender of the face corresponding to face ID1 is the same as the gender of the user read by the mobile phone from the user habit table, the mobile phone determines that face ID1 is the owner. There is no limitation on threshold 1, which is subject to the actual setting. Exemplarily, threshold 1 can be 2.

[0123] Alternatively, in some other embodiments of the present application, when the mobile phone determines that the number of face IDs that have taken photos at both the first location and the second location is less than or equal to threshold 1, the mobile phone determines the owner based on the face age. Exemplarily, if the face age corresponding to face ID1 is close to the age of the user read by the mobile phone from the user habit table, the mobile phone determines that face ID1 is the owner. Among them, similar age can be understood as: the difference between the face age corresponding to face ID1 and the user's age is less than a preset difference; or, the user's age is within a preset range, and the face age corresponding to face ID1 is also within the preset range.

[0124] Optionally, if the mobile phone determines that the number of face IDs that have taken photos at both the first location and the second location is greater than threshold 2, the mobile phone may give up determining the owner. In this embodiment, if the mobile phone screens out a large number of face IDs that have taken photos at both the first location and the second location, it can be said that the accuracy of the owner determined by the mobile phone is insufficient. Therefore, in this case, the mobile phone may give up determining the owner through the above method 3, thereby ensuring the accuracy of the mobile phone in determining the owner. There is no limitation on threshold 2, which is subject to the actual setting. Exemplarily, threshold 2 can be 3.

[0125] Optionally, after the mobile phone determines the owner through the above-mentioned method three, the mobile phone can also record the relevant information of the owner. Exemplarily, the relevant information of the owner includes: owner D, face ID-D, confidence D, photo source D. Exemplarily, in this method, the confidence D (or understood as the third confidence) can be defaulted to a value less than 1 (or the first preset value) (such as 0.96), and the specific value of the confidence is not limited. The photo source D can be "home and company". In some embodiments, the mobile phone sets the third confidence corresponding to the owner determined by method three to a second preset value, and the second preset value is less than the first preset value.

[0126] In summary, the mobile phone combines the location of the user's home and company to determine the owner, which can make the coverage of the owner identification technology wider.

[0127] The following provides an implementation method for a mobile phone to determine the location of a user's home and company. It is to be understood that the following embodiments are merely some examples of the embodiments of the present application and do not constitute a limitation on the present application.

[0128] For example, the mobile phone can calculate the user's stop point data along the moving trajectory based on the signaling data. Then, the mobile phone marks the user's daily residence and workplace based on the user's stop point data. Then, the mobile phone determines the location of the user's home and company based on the user's daily residence and workplace.

[0129] In some embodiments of the present application, the mobile phone can calculate the stop point data based on the trajectory point data of the user when moving along the moving trajectory. Among them, the trajectory point data is data with time information and spatial information. The user carries a mobile phone when traveling, and the user's trajectory point data is essentially the trajectory point data of the mobile phone. The two are equivalent. The user's trajectory point data can be determined by the spatial position corresponding to the mobile phone at different time points.

[0130] Exemplarily, the track point data can be obtained through global positioning system (GPS) positioning, or through information exchange between a mobile phone and a base station; for example, a mobile phone will record track point data when it turns on and off, makes calls, sends text messages, updates location, and switches base stations. It can also be obtained through other methods without limitation.

[0131] Fig. 9 A schematic diagram of a moving trajectory is shown. For example, the moving trajectory can be Fig. 9 The route between T1-T22 shown in the figure; T1-T22 in the moving trajectory can be understood as trajectory points. For example, taking the mobile phone obtaining trajectory point data through GPS as an example, the mobile phone periodically obtains the latitude and longitude information according to the time interval. For example, the time interval is 1 minute, and the mobile phone obtains the latitude and longitude of its current geographical location every 1 minute to obtain the user's daily trajectory point data.

[0132] Exemplarily, the track point data may be as shown in the following Table 1. Wherein, one track point data includes a time point information and a space point information (such as longitude and latitude information).

[0133] Table 1

[0134] Track Points Time Space Point T1 2022.07.01 00:00 116.671°，39.856° T2 2022.07.01 00:01 116.672°，39.857° T3 2022.07.01 00:02 116.672°，39.857° … … … T22 2022.07.01 23:59 116.78°，39.157°

[0135] Exemplarily, the mobile phone determines the stay point data based on the acquired trajectory point data (as shown in Table 1). The stay point is used to indicate the user's stay in a certain time period and the location of the stay in the time period. In other words, the stay point includes a time period and a spatial area (also referred to as a spatial range). The spatial area can be understood as an area formed by taking the longitude and latitude information corresponding to the target trajectory point as the center and a certain distance as the spatial radius. The above-mentioned target trajectory point can be a trajectory point located at the center position among multiple trajectory points. Exemplarily, as Figure 3 As shown, trajectory points T3-T8 constitute a stop point (which can be expressed as A1), and trajectory points T12-T19 constitute a stop point (which can be expressed as A2).

[0136] Exemplarily, the stop point data determined by the mobile phone may be as shown in the following Table 2. Wherein, one stop point data includes a time period and a spatial area (the latitude and longitude information and the spatial radius of the target track point).

[0137] Table 2

[0138] Stopover Time period Space area A1 2022.07.01 00:00-2022.07.01 06:00 (116.600°, 40°), radius: 100 meters A2 2022.07.01 07:00-2022.07.01 18:00 (116.700°, 40°), radius: 130 meters A3 2022.07.01 13:00-2022.07.01 14:00 (116.601°, 40°), radius: 100 meters … … … An 2022.07.01 18:30-2022.07.01 23:59 (116.602°, 40°), radius: 90 meters

[0139] Based on the above Table 2, it can be seen that the user stayed at the stop point A1 during the time period of 2022.07.01 00:00-2022.07.01 06:00, so the mobile phone can mark the spatial area corresponding to the stop point A1 as the residence. Correspondingly, the user stayed at the stop point A2 during the time period of 2022.07.01 07:00-2022.07.01 18:00, so the mobile phone can mark the spatial area corresponding to the stop point A2 as the workplace.

[0140] In the embodiment of the present application, the mobile phone can mark the user's residence and workplace every day through the above method, and determine the location of the user's home and company based on the residence and workplace marked every day. Optionally, the mobile phone can save the location of the user's home and company determined each time, so as to determine the owner of the phone based on the saved home and company locations later.

[0141] For example, Fig.10 As shown, the mobile phone displays an interface 201 (or a second interactive interface), which includes the location of the user's home and the location of the company. The location of the user's home and the location of the company are learned by the mobile phone through automatic learning. For example, the location of the user's home and the location of the company are learned by the mobile phone through the above method.

[0142] Based on the above, in the embodiment of the present application, the mobile phone can determine the owner through the above-mentioned method 1, method 2, and method 3, and after determining the owner in each method, record the relevant information of the owner. Exemplarily, the mobile phone can save the corresponding relationship between the owner determined in different ways and the relevant information of the owner in the form of a table or array (or a preset storage table). For example, the corresponding relationship can be shown in Table 3.

[0143] Table 3

[0144]

[0145] In the embodiment of the present application, since there are errors in determining the owner in different ways, the owners determined by the mobile phone in different ways may be the same or different. Optionally, since the first way is a way for indicating the name of the owner in the gallery application of the mobile phone, it can be considered that the owner determined by the mobile phone in the first way is more accurate, that is, the priority of the mobile phone in determining the owner in the first way is the highest.

[0146] In some embodiments of the present application, after receiving the voice input by the user, the mobile phone determines the instruction for indicating the owner in response to the voice. Then, in response to the instruction for indicating the owner, the mobile phone obtains photos including the owner from the gallery application according to the confidence level. Since the mobile phone determines that the owner A has the highest priority through method 1 and sets the confidence level to 1 (the first preset value), the gallery application of the mobile phone returns the face ID of the owner A determined as the face ID of the owner. Exemplary, in combination with the above Figure 4 As shown in (a) in the figure, in the message 1 displayed on the interface 106 of the mobile phone, the photo of the owner is determined by the mobile phone based on the name set in the gallery application to indicate the owner.

[0147] In some other embodiments of the present application, upon receiving the user's operation (or second operation), the mobile phone deletes the name (i.e., the owner tag) set in the gallery application to indicate the owner. It should be understood that since the owner tag does not exist, the next time the owner is identified from photos in the gallery application, the owner cannot be determined by method 1, and therefore there is no owner with a confidence level of 1. On this basis, when the mobile phone obtains photos including the owner from the gallery application based on the confidence level, the photos of the owner with a confidence level exceeding the threshold can be displayed for the user to select.

[0148] The mobile phone receives a voice (or second voice) input by the user, and in response to the voice, the mobile phone determines an instruction for indicating the owner. Then, in response to the instruction for indicating the owner, the mobile phone obtains photos including the owner from the gallery application. Referring to the above Table 3, it is assumed that the mobile phone identifies the owners (determined by the second and third methods) through the photos in the gallery application. The owners include: owner B, owner C and owner D, and the confidence levels are 0.33, 0.67 and 0.96 respectively. Optionally, the mobile phone can display owner B, owner C and owner D as the identified owners. As another embodiment, the mobile phone can also display owner B and owner D whose confidence levels exceed the confidence threshold (or exceed the preset threshold, such as 0.4) as the identified owners. Exemplarily, the mobile phone receives voice 2 (or second voice) input by the user, and voice 2 includes an instruction for indicating the owner. For example, the voice 2 can be, for example, "generate a video from my photos"; wherein "I" in voice 2 is an instruction for indicating the owner. In response to speech 2, such as Fig.11 As shown in (a) of FIG. 1 , the mobile phone displays an interface 202 (or is understood as a first interactive interface), and the interface 202 includes text content 3 (or is called a second text), and the text content 3 includes text indicating the owner. For example, the text content 3 may be "Make a video from my photo", for example, "I" in the text content 3 is text indicating the owner.

[0149] After the mobile phone displays the text content 2, Fig.11As shown in (a) in the figure, the interface 202 also includes text content 4, which is used to indicate the response of voice 2. For example, text content 4 can be "OK, the following materials have been found for you". On this basis, the mobile phone also displays the photo of the owner B (or understood as the second target photo including the second person's face) and the photo of the owner D (or understood as the third target photo including the third person's face) in the interface 202. It should be understood that the number of photos of the owner displayed on the interface 202 is related to the number of owners determined according to the second and third methods and the confidence setting, and this solution does not limit this.

[0150] In some embodiments, Fig.11 As shown in (a) of FIG. 1 , the mobile phone selects the photos of the owner B and the photos of the owner D by default. Exemplarily, in response to the user's operation (or the third operation), as Fig.11 As shown in (b) in the figure, the mobile phone unchecks the photo of the owner B. Then, in response to the user clicking the "Generate Video" control (or the first control) (or the fourth operation), the mobile phone displays a thumbnail of Video 2, where Video 2 is a video generated by the mobile phone based on the photo of the owner D (i.e., the photo including the face of the third person) in the gallery application.

[0151] It should be noted that for Fig.11 For examples of other contents of each interface shown, please refer to the above Figure 3 and Figure 4 As shown, no further details will be given here.

[0152] The above embodiments mainly describe the scheme of the embodiments of the present application in detail in combination with the hardware structure of the mobile phone (ie, the electronic device 100). The following describes the specific implementation process of the embodiments of the present application in detail in combination with the software framework of the electronic device 100.

[0153] Still taking the electronic device 100 as a mobile phone as an example, the software system of the mobile phone can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture or a cloud architecture. The embodiment of the present application takes the layered architecture as an example to illustrate the software structure of the mobile phone.

[0154] Fig.12 : is a software structure diagram of a mobile phone provided in an embodiment of the present application. The layered architecture can divide the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the layered architecture is, from top to bottom, an application layer (referred to as application layer, application), an application framework layer (referred to as framework layer, framework), a hardware abstraction layer (hardware abstraction layer, HAL) and a kernel layer (kernel, also known as a driver layer).

[0155] Application layer: can include a series of application packages. For example, the application layer can include gallery applications and voice interaction applications. Of course, the application layer can also include camera, calendar, call, map and other applications, which will not be elaborated here.

[0156] The gallery application is used to store photos or videos taken by the mobile phone. After the voice interaction application is started, the user can interact with the mobile phone through voice to complete various operations such as command input, information query and voice chat.

[0157] The framework layer provides a programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Fig.13 As shown, the framework layer may include input management services, window management services, activity management services, etc. Optionally, in the embodiment of the present application, the framework layer also includes a basic judgment module for determining the owner of the device according to the photos in the gallery application.

[0158] Exemplarily, the input management service is mainly responsible for input event monitoring, input event parsing, input event distribution, etc.

[0159] The window management service is used to manage window programs. The window management service can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc. The activity management service is mainly responsible for managing activities, starting, switching, and scheduling various components in the system, and managing and scheduling applications.

[0160] The hardware abstraction layer may include multiple library modules, which may be hardware synthesizer (HWcomposer, HWC), camera library module, etc. The mobile phone operating system may load the corresponding library module for the device hardware, thereby enabling the application framework layer to access the device hardware.

[0161] The kernel layer is the layer between hardware and software. The kernel layer may include camera drivers, display drivers, audio drivers, etc., without limitation.

[0162] Combination Fig.12 ,like Fig.13 As shown, it shows a schematic flow chart of a photo processing method. Exemplarily, the method may include the following steps.

[0163] Step 301: The gallery application of the mobile phone determines the face ID corresponding to the photo set to indicate the name of the owner of the phone.

[0164] Step 302: The gallery application of the mobile phone sends the corresponding face ID to the basic judgment module.

[0165] Step 303: The basic judgment module of the mobile phone determines the owner of the phone according to the corresponding face ID.

[0166] Exemplarily, the above steps 301 to 303 are for the mobile phone to determine the owner through method 1, and the specific implementation method can refer to the above embodiment and will not be described in detail.

[0167] Step 304: The gallery application of the mobile phone obtains the facial information corresponding to the selfie photo.

[0168] Among them, facial information can include selfie frequency, selfie time, facial ID, facial gender, facial age, etc., without restriction.

[0169] For example, the gallery application of the mobile phone can analyze the photos saved in the gallery application when the mobile phone is off (or the screen is off when charging) to generate a selfie table. On this basis, the mobile phone can obtain face information from the selfie table.

[0170] Step 305: The gallery application of the mobile phone sends the facial information corresponding to the selfie photo to the basic judgment module.

[0171] Step 306: The basic judgment module of the mobile phone determines the owner of the phone based on the facial information corresponding to the selfie photo.

[0172] Exemplarily, the above steps 304 to 306 are for the mobile phone to determine the owner through method 2, and the specific implementation method thereof can refer to the above embodiment and will not be described in detail.

[0173] Step 307: The gallery application of the mobile phone reads and obtains the photos in the gallery application, and obtains the shooting location and shooting time corresponding to each face ID in all face IDs.

[0174] Step 308: The gallery application of the mobile phone sends the shooting location and shooting time corresponding to each face ID to the basic judgment module.

[0175] Step 309: The basic determination module of the mobile phone reads the first location and the second location.

[0176] Step 310: The basic judgment module of the mobile phone selects the face IDs of people who have taken photos at both the first location and the second location according to the shooting location and shooting time corresponding to each face ID to determine the owner of the phone.

[0177] Exemplarily, the above steps 307 to 310 are for the mobile phone to determine the owner through method three, and the specific implementation method thereof can refer to the above embodiment and will not be described in detail.

[0178] It should be noted that the above-mentioned methods 1 to 3 are three parallel implementation methods. The embodiment of the present application does not limit the order of methods 1 to 3, which is subject to the actual implementation.

[0179] Step 311, after the voice interaction application of the mobile phone determines the instruction for instructing the owner, it queries the basic judgment module for the face ID of the owner. In this step, the face ID of the owner can be determined based on the face ID and its confidence. Specifically, the basic judgment module returns the face ID with the highest confidence as the face ID of the owner, and can also return the face ID with a confidence exceeding a preset threshold as the face ID of the owner. Optionally, the basic judgment module can judge the way to return the face ID of the owner for the value of the highest confidence. Specifically, it is judged whether the value of the highest confidence is a first preset value (such as the face ID confidence determined in method one is 1). When the value of the highest confidence is the first preset value, only the face ID with a confidence value of the first preset value can be returned as the face ID of the owner; when the value of the highest confidence is not the first preset value, all face IDs with a confidence exceeding the preset threshold are returned as the face ID of the owner. Step 312: The voice interaction application of the mobile phone sends an instruction message to the gallery application. The instruction message includes the face ID of the owner, which is used to instruct the gallery application to filter photos including the owner according to the face ID of the owner.

[0180] Step 313: The gallery application of the mobile phone returns photos including the owner's photos to the voice interaction application.

[0181] Step 314: The voice interaction application displays photos including the owner, and generates a video based on the photos of the owner. Specifically, the voice interaction application selects a photo from the photos including the owner according to a preset rule as a representative photo to be displayed (the first target photo). In response to the user's operation of viewing all photos, multiple photos of the owner may be displayed. The preset rule may be based on the position of the face corresponding to the owner in the photo, the aesthetic score of the photo, the time when the photo was taken, etc., which are not limited in this solution again.

[0182] To sum up, by adopting the solution of the embodiment of the present application, the basic judgment module of the mobile phone can determine the owner in three ways. The voice interaction application of the mobile phone can query the face ID of the owner from the basic judgment module, and obtain the owner's photo from the gallery application based on the owner's face ID to generate a video, thereby improving the efficiency of human-computer interaction.

[0183] It should be noted that the contents recorded in each embodiment of the embodiments of the present application can explain the technical solutions in other embodiments of the embodiments of the present application, and the technical features recorded in each embodiment can also be applied in other embodiments to form new solutions by combining the technical features in other embodiments. The present application only exemplarily lists several embodiments for illustration, and does not mean that the present application is limited to this.

[0184] The present application embodiment provides an electronic device, which may include a memory and one or more processors; the memory stores computer program code, which includes computer instructions. When the computer instructions are executed by the processor, the electronic device executes the functions or steps executed by the mobile phone in the above embodiment. The structure of the electronic device can refer to the above Figure 1 The structure of the electronic device 100 is shown.

[0185] The present application also provides a chip system for use in electronic devices. Fig.14 As shown, the chip system 1100 includes at least one processor 1101 and at least one interface circuit 1102. The processor 1101 may be the processor of the above embodiment. Figure 1 The processor 110 is shown. On this basis, the interface circuit 1102 may be, for example, an interface circuit between the processor 110 and an external memory; or an interface circuit between the processor 110 and an internal memory.

[0186] The processor 1101 and the interface circuit 1102 can be interconnected via lines. For example, the interface circuit 1102 can be used to receive signals from other devices (such as the memory of the electronic device 100). For another example, the interface circuit 1102 can be used to send signals to other devices (such as the processor 1101). Exemplarily, the interface circuit 1102 can read the instructions stored in the memory and send the instructions to the processor 1101. When the instructions are executed by the processor 1101, the electronic device can execute the various functions or steps executed by the mobile phone in the above embodiment. Of course, the chip system can also include other discrete devices, which is not specifically limited in the embodiments of the present application.

[0187] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes each function or step executed by the mobile phone in the above method embodiment.

[0188] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute each function or step executed by the mobile phone in the above method embodiment.

[0189] It should be noted that the terms "first" and "second" in the embodiments of the present application and the accompanying drawings are used to distinguish different objects rather than to describe a specific order. "First" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "multiple" means two or more.

[0190] In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0191] It should be understood that in the present application, "at least one (item)" refers to one or more. "Multiple" refers to two or more. "At least two (items)" refers to two or three and more than three. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple. “When” and “if” both mean that corresponding measures will be taken under certain objective circumstances. It does not limit the time, nor does it require any judgment when it is implemented, nor does it mean that there are other limitations.

[0192] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way for easy understanding.

[0193] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules according to the system, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0194] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0195] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0196] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0197] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0198] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A photo processing method, characterized in that: Used in electronic equipment, including: The electronic device receives a first voice input by a user, and displays a first text corresponding to the first voice on a first interactive interface; the first text includes information indicating the owner of the device; In response to the first voice, displaying a first target photo and a first control on the first interactive interface; wherein the first target photo includes a first face, and the first face is set with an owner tag in the gallery application of the electronic device; In response to a first operation input by a user on a first control, the electronic device displays a thumbnail of a target video; the target video is generated by the electronic device based on a first target photo including the first face in a gallery application.

2. The method according to claim 1, characterized in that The method further comprises: The electronic device receives a second operation by a user to delete an owner tag in the gallery application; In response to a second voice input by a user, the electronic device displays a second text corresponding to the second voice on the first interaction interface; the second text includes information indicating the owner; In response to the second voice, the electronic device displays a second target photo, a third target photo and the first control on the first interactive interface; wherein the second target photo includes a second face, and the third target photo includes a third face; In response to a third operation by the user, the electronic device cancels the selection of the second target photo; In response to a fourth operation of the user on the first control, the electronic device displays a thumbnail of a target video; the target video is generated by the electronic device based on a third target photo in a gallery application that includes the third face.

3. The method according to claim 2, characterized in that The second target photo is determined by the electronic device based on a front-facing photo containing the second face among all photos included in the gallery application.

4. The method according to claim 3, characterized in that The number of front-camera photos containing the second face is the largest number among all front-camera photos containing faces; or, the number of front-camera photos containing the second face is greater than or equal to a selfie count threshold.

5. The method according to any one of claims 2 to 4, characterized in that: The method further comprises: The electronic device displays a second interactive interface, wherein the second interactive interface includes a first location and a second location; The first location is a home location learned by the electronic device, the second location is a company location learned by the electronic device, and the photo containing the third person's face has appeared in both the first location and the second location.

6. The method according to any one of claims 1 to 5, characterized in that The first interactive interface also displays a second control; the method further includes: In response to a fifth operation input by the user to the second control, the electronic device expands and displays multiple photos including the first face.

7. The method according to claim 2, characterized in that Before displaying the first target photo, the method further includes: The electronic device determines that the first voice includes a command to instruct the owner, and queries the face ID of the owner; The electronic device determines, according to the confidences corresponding to the different face IDs, that the face ID of the machine owner is the first face; the confidence of the first face is a first confidence, and the first confidence is greater than the confidences of the other faces; The electronic device screens photos including the first face and determines a first target photo.

8. The method according to claim 7, characterized in that Before the first interactive interface displays the second target photo and the third target photo, the method further includes: The electronic device determines that the second voice includes an instruction to instruct the owner, and queries the face ID of the owner; The electronic device determines, according to the confidence levels corresponding to the different face IDs, that the face IDs of the device owner are the second face and the third face; The electronic device screens photos including the second face and the third face to determine the second target photo and the third target photo.

9. The method according to claim 7 or 8, characterized in that: Before the electronic device receives a first voice input by a user, the method further includes: The electronic device generates a plurality of photo collections of different persons based on different face ID analysis results; The electronic device sets an owner tag for the photo collection including the first person's face; The electronic device determines that the first confidence level of the first face is the first preset value.

10. The method according to claim 8, characterized in that Before the electronic device receives a second voice input by a user, the method further includes: The electronic device obtains the gender and age corresponding to each of the multiple faces included in all the front-facing photos in the photo gallery application; The electronic device selects a first group of faces according to the gender and age corresponding to each face in the plurality of faces; the gender corresponding to each face in the first group of faces is the same as the gender corresponding to the second face, and the difference between the age corresponding to each face and the age corresponding to the second face is within a preset range; The electronic device uses the ratio of the number of front-mounted photos of the second face to the number of front-mounted photos of the first group of faces as a second confidence level; the second confidence level is less than the first preset value; The electronic device reads the photos in the gallery application and obtains the shooting location and shooting time of the photos corresponding to each face ID in all face IDs; The electronic device determines, according to the shooting location and shooting time of the photos corresponding to each face ID in all face IDs, a target face ID corresponding to the photos taken at the first location and the second location at the same time; the target face ID includes the third face; The electronic device sets a third confidence level corresponding to the third face to a second preset value, where the second preset value is smaller than the first preset value.

11. The method according to claim 10, characterized in that The method further comprises: The electronic device determines whether the value of the highest confidence level among the confidence levels corresponding to different face IDs is the first preset value; When the value of the highest confidence is the first preset value, returning the face ID whose confidence value is the first preset value as the face ID of the owner; When the value of the highest confidence is not the first preset value, all face IDs whose confidence exceeds the preset threshold are returned as the face ID of the owner.

12. The method according to claim 10, characterized in that The electronic device determines the target face ID corresponding to the photos taken at the first location and the second location at the same time according to the shooting location and shooting time of the photos corresponding to each face ID in all face IDs, including: The electronic device determines, based on the shooting location of the photo corresponding to each face ID in all face IDs, that the distance between the shooting location and the first location is less than or equal to a first preset distance, and determines that the distance between the shooting location and the second location is less than or equal to a second preset distance; The electronic device determines a frequency of taking photos of each face ID in all face IDs at the first location, and determines a frequency of taking photos at the second location; The electronic device selects, according to the shooting time, target face IDs corresponding to the photos taken at the first location and the second location at the same time; The target face ID takes photos at the first location at a frequency greater than a preset frequency, and takes photos at the second location at a frequency greater than the preset frequency.

13. The method according to claim 10 or 12, characterized in that: Before the electronic device determines the target face ID corresponding to the photos taken at the first location and the second location at the same time according to the shooting location and shooting time of the photos corresponding to each face ID in all face IDs, the method further includes: The electronic device determines that a distance between the first location and the second location is greater than or equal to a preset distance.

14. The method according to claim 12 or 13, characterized in that Before the electronic device receives a second voice input by a user, the method further includes: The electronic device determines the age range of the owner according to the user's usage habits; Wherein, the method further comprises: If the number of the target face IDs is less than or equal to a preset number, the electronic device uses the corresponding face IDs within the age range among the target face IDs as the face IDs of the owner.

15. An electronic device, characterized in that: include: memory and one or more processors, a display screen; The memory stores computer program code, which includes computer instructions. When the computer instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 14.

16. A chip system, characterized in that: include: at least one processor and an interface; The interface is used to receive instructions and transmit them to the at least one processor; The at least one processor executes the instructions so that the electronic device performs the method according to any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 1 to 14.

18. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 14.

Images