A photograph processing method and electronic device

Abstract

The application provides a photo processing method and an electronic device, and relates to the technical field of terminals. The method comprises the following steps: an electronic device receives a first voice input by a user, and displays a first photo and a first control in a first interaction interface in response to the first voice. The first photo contains a user corresponding to a target user ID, the user corresponding to the target user ID has a first social relationship with a machine owner, the target user ID is determined based on the first social relationship from a plurality of social relationships stored in the electronic device, and the plurality of social relationships are obtained by cyclic iteration of the electronic device on the social relationship between the machine owner and other people. In response to a first operation of the first control input by the user, the electronic device displays a thumbnail of a target video, so that the problems of poor logic, weak relevance, poor human-computer interaction efficiency and the like of the generated video in the process of generating the video by the electronic device can be solved.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a photo processing method and an electronic device. Background Technology

[0002] With the development of terminal technology and the increasing maturity of speech recognition technology, voice input has become increasingly important due to its high naturalness and effectiveness in interaction. Voice interaction applications within electronic devices have also become a frequently used function. Users can interact with electronic devices (such as mobile phones, tablets, and smartwatches) via voice to complete various operations such as command input, information retrieval, and voice chat.

[0003] For example, electronic devices can respond to a user's voice and generate videos from photos in their gallery applications. However, in the process of generating videos, electronic devices often suffer from problems such as poor video logic, weak relevance, and inefficient human-computer interaction. Summary of the Invention

[0004] This application provides a photo processing method and an electronic device to solve problems such as poor logical consistency, weak relevance, and poor human-computer interaction efficiency in the process of generating videos by electronic devices.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] In a first aspect, a photo processing method is provided, the method comprising: an electronic device receiving a first voice input by a user, and displaying a first text corresponding to the first voice on a first interactive interface; the first text including information indicating the owner of the device, and information indicating a first social relationship with the owner.

[0007] In response to the first voice command, a first photo and a first control are displayed on the first interactive interface. The first photo contains a user corresponding to a target user ID, and the user corresponding to the target user ID has a first social relationship with the device owner. The target user ID is determined from multiple social relationships stored in the electronic device based on the first social relationship indicated by the first text. The multiple social relationships are obtained by the electronic device iteratively analyzing the social relationships between the device owner and others.

[0008] In response to the user's first input to the first control, the electronic device displays a thumbnail of the target video. The target video is generated by the electronic device based on a first photo of the user corresponding to the target user ID in a photo library application. In this way, the electronic device can determine the first social relationship indicated by the first text from multiple social relationships stored in the electronic device, obtain the target user ID corresponding to the first social relationship from the photo library application, and obtain the first photo containing the user corresponding to the target user ID, and then generate the video. This solves the problems of poor logical consistency, weak relevance, and poor human-computer interaction efficiency in the video generation process of electronic devices. Furthermore, since multiple social relationships are obtained by the electronic device through iterative analysis of the user's social relationships with others, the stability of the multiple social relationships stored by the electronic device is relatively high.

[0009] In one design approach, the electronic device includes multiple photos, each photo corresponding to a multiple user ID, and each user ID corresponding to a face; the electronic device stores multiple social relationships, specifically: the electronic device stores the social relationships between the device owner and the user ID corresponding to the same face.

[0010] The method further includes: the electronic device inputs the user information corresponding to the first user ID and the second user ID from multiple user IDs into a same person prediction model to obtain a first prediction result, the first prediction result indicating whether the faces corresponding to the first user ID and the second user ID are the faces of the same person; if the first prediction result indicates that the faces corresponding to the first user ID and the second user ID are the faces of the same person, the electronic device assigns a first fusion ID and fuses the user information corresponding to the first user ID and the second user ID to obtain the user information corresponding to the first fusion ID; the electronic device inputs the user information corresponding to the first fusion ID and the user information of the device owner into a relationship prediction model to obtain the social relationship between the device owner and the user corresponding to the first fusion ID.

[0011] In this design, the number of iterations is reduced by first predicting whether the faces corresponding to the first user ID and the second user ID are the same person's faces. If so, the social relationship between the device owner and the user corresponding to the first fused ID (i.e., the ID formed by merging the first user ID and the second user ID) is then predicted, thereby reducing the power consumption.

[0012] In one design approach, the method further includes: the electronic device inputs user information corresponding to a first fused ID and a third user ID from a plurality of user IDs into a same-person prediction model to obtain a second prediction result, the second prediction result indicating whether the faces corresponding to the first fused ID and the third user ID are the faces of the same person; if the second prediction result indicates that the faces corresponding to the first fused ID and the third user ID are the faces of the same person, the electronic device assigns a second fused ID and merges the user information corresponding to the first fused ID and the third user ID to obtain user information corresponding to the second fused ID; the electronic device inputs the user information corresponding to the second fused ID and the user information of the device owner into a relationship prediction model to obtain the social relationship between the device owner and the user corresponding to the second fused ID.

[0013] In this design approach, electronic devices predict the social relationships between the user and others through iterative cycles, thereby making the predicted social relationships more stable.

[0014] In one design approach, the method further includes: if the second prediction result indicates that the faces corresponding to the first fusion ID and the third user ID are not the same person's faces, the electronic device saves the social relationship between the device owner and the user corresponding to the first fusion ID, and no longer updates the social relationship between the device owner and the user corresponding to the first fusion ID. Thus, it can be understood that the social relationship predicted by the electronic device between the device owner and the user corresponding to the first fusion ID is stable, ending the loop iteration, thereby reducing power consumption.

[0015] In one design approach, the method further includes: for a user ID corresponding to the same face, the electronic device updates the social relationships of the user ID once, incrementing the social relationship iteration count of the user ID by 1; when the social relationship iteration count of the user ID reaches a preset number, the electronic device stops updating the social relationships of the user ID. Thus, by setting the number of iterations, power consumption can be reduced while ensuring the stability of the predicted social relationships.

[0016] In one design approach, the user information corresponding to the first user ID and the second user ID includes at least: facial features and co-occurrence frequency; wherein, the facial features include at least the average age, gender difference, and facial similarity of the users corresponding to the first user ID and the second user ID; and the co-occurrence frequency includes at least the frequency of the users corresponding to the first user ID and the second user ID appearing together in the same photo.

[0017] In one design approach, the co-occurrence frequency also includes the frequency of the first user ID and the second user ID appearing together in the same image set; wherein, the same image set includes M photos, the shooting time interval between any two photos in the M photos does not exceed a preset interval, and the distance difference between the shooting positions of any two photos does not exceed a preset distance; M is a positive integer.

[0018] In one design approach, the user information corresponding to the first fusion ID and the third user ID includes at least: facial features, co-occurrence frequency, and relationship features; wherein, the facial features include at least the average age, gender difference, and facial similarity of the users corresponding to the first fusion ID and the third user ID; the co-occurrence frequency includes at least the frequency of the users corresponding to the first fusion ID and the third user ID appearing together in the same photo; and the relationship features include the social relationship between the device owner and the user corresponding to the first fusion ID.

[0019] In one design, an electronic device inputs user information corresponding to a first user ID and a second user ID from multiple user IDs into a same-person prediction model to obtain a first prediction result. This includes: the electronic device inputting user information corresponding to a first user ID and a second user ID from multiple user IDs into the same-person prediction model to obtain the same-person probability; the electronic device obtaining the first prediction result based on the same-person probability; wherein, if the same-person probability is greater than or equal to a probability threshold, the first prediction result indicates that the faces corresponding to the first user ID and the second user ID are the faces of the same person; if the same-person probability is less than the probability threshold, the first prediction result indicates that the faces corresponding to the first user ID and the second user ID are not the faces of the same person.

[0020] In one design approach, the user information corresponding to the first fusion ID includes at least: the number of photos containing the first fusion ID, the number of image sets containing the first fusion ID, and the image set location type corresponding to the photos containing the first fusion ID; the user information of the device owner includes: the number of photos containing the device owner ID, the number of image sets containing the device owner ID, and the image set location type corresponding to the photos containing the device owner ID; wherein, the image set location type includes: home, hometown, indoor or outdoor.

[0021] In one design approach, the user information corresponding to the first fused ID and the user information corresponding to the device owner also include: the visual relationship between the user corresponding to the first fused ID and the device owner; the visual relationship includes: close, back to back, near, or hugging.

[0022] In a second aspect, an electronic device is provided, which has the functions described in any one of the first aspects above. These functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned functions.

[0023] Thirdly, an electronic device is provided, comprising: a memory and one or more processors, and a display screen; the memory stores computer program code, the computer program code including computer instructions; when the computer instructions are executed by the processor, the electronic device performs the method described in the first aspect or any one of the first aspects.

[0024] Fourthly, a chip system is provided, the chip system comprising: at least one processor and an interface for receiving instructions and transmitting them to the at least one processor; the at least one processor executes instructions to cause an electronic device to perform the method described in any one of the first aspects.

[0025] Fifthly, a computer-readable storage medium is provided that stores instructions which, when executed on a computer, cause the computer to perform the method described in any one of the first aspects.

[0026] In a sixth aspect, a computer program product containing instructions is provided, which, when run on a computer, enables the computer to perform the method described in any one of the first aspects above.

[0027] The technical effects of any of the implementation methods in the second to sixth aspects mentioned above can be referred to the technical effects of different implementation methods in the first aspect, and will not be elaborated here. Attached Figure Description

[0028] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0029] Figure 2A This application provides an illustration of a scenario for waking up a mobile phone. Figure 1 ;

[0030] Figure 2B A second schematic diagram illustrating a scenario for waking up a mobile phone, provided as an embodiment of this application;

[0031] Figure 2C This application provides an illustration of a scenario for waking up a mobile phone. Figure 3 ;

[0032] Figure 3 A schematic diagram of a voice interaction interface provided in this application embodiment. Figure 1 ;

[0033] Figure 4 A second schematic diagram of a voice interaction interface provided in an embodiment of this application;

[0034] Figure 5 A schematic diagram illustrating a cyclical iteration of social relationships provided in this application embodiment;

[0035] Figure 6 A flowchart illustrating a partitioning atlas provided in an embodiment of this application;

[0036] Figure 7 A schematic diagram illustrating a process for generating an atlas, provided as an embodiment of this application;

[0037] Figure 8 This application provides a flowchart illustrating the prediction of indoor and outdoor map types in an embodiment.

[0038] Figure 9 A schematic diagram illustrating a process for updating an atlas, provided as an embodiment of this application;

[0039] Figure 10 A flowchart illustrating the prediction of the same person is provided for an embodiment of this application;

[0040] Figure 11 A flowchart illustrating the prediction of social relationships is provided for an embodiment of this application;

[0041] Figure 12 A schematic diagram of another process for predicting the same person provided in an embodiment of this application;

[0042] Figure 13 A schematic diagram illustrating a photo processing method provided in an embodiment of this application;

[0043] Figure 14 A schematic diagram illustrating another photo processing method provided in an embodiment of this application;

[0044] Figure 15 This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation

[0045] This application provides a photo processing method applicable to electronic devices such as mobile phones, tablets, and smartwatches that support voice interaction. The electronic device provides a voice interaction interface to the user, which can receive voice input from the user. In response to the user's voice input, the electronic device determines an instruction indicating a social relationship with the device owner. Then, in response to the instruction indicating a social relationship with the device owner, the electronic device retrieves photos from a photo library application, including photos of users corresponding to the target user ID with whom the user has a social relationship, and generates a video.

[0046] In summary, by adopting the solution of this application embodiment, the electronic device can determine the instruction indicating a social relationship with the device owner based on the user's voice input, and then generate a video in response to the instruction indicating a social relationship with the device owner, based on the photos of the target user ID corresponding to the user with a social relationship with the device owner included in the photo library application. This can solve the problems of poor logic, weak relevance, and poor human-computer interaction efficiency of the generated video.

[0047] For example, in the embodiments of this application, the aforementioned social relationship can also be understood as an "intimate relationship," which may include lovers, wives, husbands, children, fathers, mothers, etc., without limitation. It should be understood that the names of the social relationships shown in the embodiments of this application are merely examples, and social relationships may have different names, as long as they can represent the social relationship with the device owner, they are all within the protection scope of the embodiments of this application.

[0048] For example, the term "wife" can also be called "wife"; the term "husband" can also be called "husband"; the term "father" can also be called "dad" or "Dad"; and the term "mother" can also be called "mom" or "Mom," etc., without any restrictions.

[0049] For example, the electronic devices in this application embodiment may be mobile phones, tablets, desktops, laptops, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, cellular phones, and personal digital assistants (PDAs). Augmented reality (AR) / virtual reality (VR) devices and other electronic devices that support voice interaction functions are also included. This application embodiment does not impose special limitations on the specific form of the electronic devices.

[0050] refer to Figure 1 This is a hardware structure diagram of an electronic device 100 provided in an embodiment of this application. Figure 1 As shown, the electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and subscriber identification module (SIM) card interfaces 1~N 195, etc.

[0051] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0052] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0053] It is understood that the interface connection relationships between the modules illustrated in this embodiment are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0054] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via a USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device 100 via the power management module 141.

[0055] Power management module 140 is used to connect battery 142, charging management module 140, and processor 110. Power management module 141 receives input from battery 142 and / or charging management module 140, and supplies power to processor 110, internal memory 121, external memory, display 194, camera 191, and wireless communication module 160, etc. Power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, power management module 141 may also be located in processor 110. In other embodiments, power management module 141 and charging management module 140 may also be located in the same device.

[0056] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0057] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0058] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0059] The electronic device 100 can perform shooting functions through an ISP, a camera 193, a video codec, a GPU, a display screen, and an application processor. The ISP is used to process data fed back by the camera 193. The camera 193 is used to capture still images or videos. In some embodiments, the electronic device may include one or N cameras 193, where N is a positive integer greater than 1.

[0060] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0061] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0062] Electronic device 100 can implement audio functions through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor, such as music playback and recording.

[0063] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch buttons. Electronic device 100 can receive button inputs and generate key signal inputs related to user settings and function control. Motor 191 can generate vibration alerts; motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. Indicator 192 can be an indicator light, used to indicate charging status, battery level changes, messages, missed calls, notifications, etc. SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with electronic device 100. Electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1.

[0064] Of course, the electronic device 100 provided in this application embodiment may also include one or more devices such as a positioning module 181, a button 190, a motor 191, an indicator 192, and a SIM card interface 195, without limitation.

[0065] The methods described in the following embodiments can all be implemented in the electronic device 100 with the above-described hardware structure. It is understood that the photo processing method provided in this application can be executed by electronic devices supporting voice interaction, such as mobile phones, tablets, and smartwatches; it can also be executed by a chip, chip system, or processor capable of implementing the photo processing method; or it can be executed by a logic module or software capable of implementing all or part of the functions of the electronic device, without limitation. The following embodiments use a mobile phone as the execution subject to provide a detailed description of the solution provided in this application.

[0066] In some scenarios, many mobile phones support voice control. Specifically, the phone uses a microphone to capture the user's voice, analyzes and recognizes the voice, and executes the corresponding commands, allowing the user to control the phone via voice.

[0067] Generally, to conserve power and prevent accidental triggering, voice control needs to be enabled before use. For example, a user can wake up the phone by inputting a preset word (or wake-up word) through voice. Once awakened, the phone can execute the corresponding voice command, thus activating voice control. Users can also enable or disable voice control by toggling preset switches on or off through the phone's user interface.

[0068] Voice control functions may have different names on different mobile phones, such as "smart assistant", "smart application", "voice assistant", etc. The specific implementation of voice control functions with different names may also be somewhat different.

[0069] The following section uses "Smart Assistant" as an example to illustrate several different implementations of voice control functions.

[0070] Before using the smart assistant to control their phone, users need to wake up the phone by inputting a wake word via voice. Normally, before the phone is woken up, the microphone operates in a power-saving mode (e.g., searching for signals at lower power levels) to pick up ambient sounds. The microphone only detects the wake word at the kernel level; the phone's system and drivers do not activate the corresponding recording channel for the smart assistant.

[0071] In response to user actions (such as receiving a wake-up word from the user's voice), the phone is woken up, and the corresponding recording channel for the smart assistant is activated in the system and drivers. After the phone is woken up, the voice (audio stream) captured by the microphone is sent to the smart assistant application for processing through the corresponding recording channel. In this way, the phone can execute the user's voice commands, enabling the user to control the phone via voice. Optionally, the phone can also perform functions such as conversing with the user.

[0072] refer to Figure 2A It illustrates a scenario for waking up a mobile phone. For example... Figure 2A As shown, the mobile phone displays a desktop interface. The phone receives a voice wake-up word input by the user (e.g., "Hello, YOYO"), and wakes up in response to this wake-up word. For example, after being woken up, the phone displays a voice interaction interface, such as interface 101. Interface 101 is used to display the dialogue content between the phone and the user during a voice conversation.

[0073] refer to Figure 2B This illustrates another scenario for waking up a phone. For example... Figure 2B As shown, the phone displays a desktop interface. The phone receives a long press operation from the user on the power button (or power-on button), and in response to this operation, the phone is woken up. For example, after the phone is woken up, it displays a voice interaction interface, such as interface 101.

[0074] In some embodiments of this application, the smart assistant has corresponding function entry points on the mobile phone, such as the control center and settings application. Users can access the corresponding function entry points and turn the smart assistant's functions on or off by turning preset switches on or off in the mobile phone's human-computer interaction interface.

[0075] refer to Figure 2C This illustrates another scenario for waking up a phone. For example... Figure 2C As shown, in response to a user opening the phone's settings function (e.g., clicking the settings app icon on the desktop), the phone displays a settings interface 102. The settings interface 102 includes a smart assistant option 103, which is used to enable or disable voice control functionality. For example, as... Figure 2C As shown, in response to the user's activation of the smart assistant option 103, the phone is woken up. For example, after the phone is woken up, a voice interaction interface is displayed, such as interface 101.

[0076] Optionally, after the phone is woken up, the user can issue a command by inputting voice, and the phone will execute the command. Once the phone has executed a command, or if it does not receive a voice command from the user within a certain period after being woken up (e.g., within 8 seconds), the phone will no longer respond to voice commands. For example, the phone will close the recording channel corresponding to the smart assistant. The user needs to wake up the phone again to issue commands by inputting voice again. In other words, after the phone is woken up, it enters a "short-reception" state, and can respond to voice commands from the user for a short period of time (e.g., within 8 seconds).

[0077] Optionally, when connected to the network, the phone supports entering a continuous conversation scenario after being woken up, allowing users to engage in ongoing dialogue. After each broadcast, the phone will resume receiving audio without needing to be woken up again, until the user exits the continuous conversation using a command such as "exit."

[0078] For example, when the mobile phone is woken up and enters a continuous conversation scenario, the user can engage in a continuous conversation with the phone on interface 101. Interface 101 displays the content of the continuous conversation between the user and the phone. Optionally, interface 101 includes a prompt message to notify the user that the phone has entered a continuous conversation scenario. For example, the prompt message could be "Welcome."

[0079] Optionally, interface 101 also includes a prompt icon 10a, which is used to notify the user that the phone has entered continuous recording mode. In actual implementation, the prompt icon 10a can be a microphone recording icon, or a "robot recording icon," etc., and is not limited thereto.

[0080] After the phone is woken up, it enters continuous audio reception mode. The user can input voice commands, which the phone parses and recognizes, and then executes the corresponding action. For example, the phone can receive voice input from the user indicating a first social relationship with the phone's owner. In response to this voice input, the phone can retrieve a first photo from the gallery app. This first photo contains a user with a target user ID, and this user has a first social relationship with the phone's owner. Based on this, the phone retrieves the first photo containing the target user ID from the gallery app and generates a video. Here, "phone owner" can be understood as the primary user of the phone, i.e., the owner of the phone. The first social relationship with the phone owner can include, but is not limited to, "the phone owner's wife," "the phone owner's father," "the phone owner's mother," "the phone owner's child," etc.

[0081] For example, such as Figure 3As shown in (a), during the user's voice input, the phone displays interface 104, which includes a notification icon 10b to inform the user that the microphone is recording sound. For example, as shown in (a)... Figure 3 As shown in (a), the prompt icon 10b is a transparent circle displayed on top of the prompt icon 10a. Based on this, the phone receives voice input 1 from the user, which includes instructions for the owner and instructions for indicating the existence of a first social relationship with the owner. For example, as... Figure 3 As shown in (a), the voice command 1 can be "Generate a video from my dad's photo." In this voice command 1, "I" is the instruction given to the device owner, and "dad" is the instruction establishing a primary social relationship with the device owner (i.e., "I"). Figure 3 As shown in (b), in response to voice 1, the mobile phone displays interface 105, which includes text content 1. This text content 1 includes text instructing the phone owner and text that has a first social relationship with the phone owner. For example, the text content 1 could be "Generate a video from my dad's photo." In this text content 1, "I" is the text instructing the phone owner, and "dad" is the text that has a first social relationship with the phone owner (i.e., "I").

[0082] For example, the mobile phone can recognize the received user input voice 1, thereby identifying "I" as a command instructing the phone owner, and "Dad" as a command indicating a first social relationship with the phone owner. Optionally, when voice 1 matches the database, the mobile phone recognizes "I" as a command instructing the phone owner. For example, if voice 1 includes "I" and the database includes "I," the mobile phone can recognize "I" as a command instructing the phone owner. Similarly, when voice 1 matches the database, the mobile phone recognizes "Dad" as a command indicating a first social relationship with the phone owner. For example, if voice 1 includes "Dad" and the database includes "Dad," the mobile phone can recognize "Dad" as a command indicating a first social relationship with the phone owner.

[0083] It should be noted that the mobile phone can perform semantic understanding on voice 1. Voice 1 does not need to be completely identical to the database, as long as the semantics are the same or close. That is, voice 1 matches the database, but does not have to be completely identical. For example, in... Figure 3 In the scenario shown, the user's voice input 1 includes "I", and the database includes "myself", "myself", and "myself". Since "I" has the same semantic meaning as "myself", "myself", and "myself", the two match. Therefore, the mobile phone can recognize that "I" is a command used to instruct the owner, thus determining that "I" is used to instruct the owner.

[0084] Correspondingly, in Figure 3In the scenario shown, the user's voice input 1 includes "Dad", and the database includes "father" and "Dad". Since "Dad" has the same meaning as "father" and "Dad", the two match. Therefore, the phone can recognize "Dad" as the command that has the first social relationship with the owner, and thus determine that the first social relationship with the owner is "Dad".

[0085] Alternatively, the phone can recognize and analyze Voice 1 to identify "I" as an instruction to the phone owner and "Dad" as the primary social relationship with the phone owner. For example, after receiving Voice 1 from the user, the phone uses Automatic Speech Recognition (ASR) technology to convert the user's speech into text, and then uses Natural Language Understanding (NLU) technology to perform intent recognition on the converted text, thereby identifying "I" as an instruction to the phone owner and "Dad" as the primary social relationship with the phone owner.

[0086] After the phone displays text content 1, as follows Figure 4 As shown in (a), the mobile phone displays interface 106, which includes text content 2, used to indicate the response of voice 1. For example, text content 2 could be "Okay, the following materials have been found for you." In addition, the mobile phone displays a first photo on interface 106, which contains the user corresponding to the target user ID. It is understood that the user corresponding to the target user ID has a first social relationship with the phone owner, such as "father" as shown above, meaning the user corresponding to the target user ID is the phone owner's father. Optionally, interface 106 also includes control 1 and control 2. Control 1 is used to trigger the mobile phone to display all photos related to the user corresponding to the target user ID, and control 2 is used to trigger the mobile phone to generate a video from the first photo containing the user corresponding to the target user ID. For example, control 1 could be called a "View All" control, and control 2 could be called a "Generate Video" control.

[0087] For example, in response to user actions on control 2, such as Figure 4 As shown in (b), the mobile phone displays interface 107, which includes a thumbnail of video 1. Video 1 is a thumbnail of video 1 displayed on the mobile phone. Figure 4(a) shows the generation of the first photo of the user corresponding to the target user ID. Optionally, interface 107 includes control 3 for triggering the mobile phone to play video 1. For example, the mobile phone can receive user click operations, long press operations, swipe operations, etc. on control 3. The following description will mainly use click operations as an example. In response to the user's click operation on control 3, the mobile phone plays video 1. The content of video 1 is not shown and is subject to the actual implementation.

[0088] In summary, the solution adopted in this application allows the phone to enter a continuous dialogue scenario after being woken up, enabling the user to engage in continuous conversation. Based on this, in response to the user's voice input (voice 1), the phone identifies the owner and the first social relationship with that owner. Then, the phone retrieves a first photo containing the user corresponding to the target user ID from the gallery application—that is, a photo of the user with the first social relationship with the owner. The phone generates a video based on this first photo, thereby resolving issues such as poor video logic, weak relevance, and poor human-computer interaction efficiency.

[0089] In some embodiments of this application, the mobile phone can first identify the owner based on photos included in the gallery application; correspondingly, the mobile phone can identify the first social relationship between different faces and the owner based on the photos included in the gallery application. Based on this, when the mobile phone recognizes that the user's voice includes instructions for instructing the owner and instructions for indicating the first social relationship with the owner, the mobile phone responds to the instructions for indicating the first social relationship with the owner by retrieving a first photo from the gallery application including the user corresponding to the target user ID with the first social relationship with the owner, and generates a video based on the first photo including the user corresponding to the target user ID.

[0090] Optionally, the phone can identify the owner based on photos in the Gallery app while charging and the screen is off, as well as identify multiple social relationships between the owner and others; or, the phone can identify the owner based on photos in the Gallery app while the screen is off, as well as identify multiple social relationships between the owner and others, etc., without limitation.

[0091] It should be noted that, in the embodiments of this application, the specific implementation method for mobile phone owner identification can refer to relevant technologies, and will not be repeated here.

[0092] The following detailed description, in conjunction with the accompanying drawings, provides a specific technical solution for identifying the social relationships between the phone owner and others based on photos in the gallery application. It should be understood that the solutions described in the following embodiments are merely illustrative examples and do not constitute a limitation of this application.

[0093] refer to Figure 5It illustrates a flowchart for identifying the social relationships between the device owner and others, such as... Figure 5 As shown, the method may include the following steps.

[0094] Step 1: The phone reads multiple user IDs corresponding to multiple photos in the gallery application, and obtains the facial information corresponding to each user ID based on the multiple user IDs.

[0095] In this system, each user ID corresponds to one face, and each face corresponds to one set of facial information. For example, facial information may include at least: face identifier (such as tag_id), gender, age, facial features, and yaw from face. Of course, facial information may also include other parameters, which will not be elaborated upon here.

[0096] For example, a mobile phone can read the hash value of each photo from a gallery data table, and then read the user ID corresponding to the hash value of each photo to obtain the facial information corresponding to the user ID.

[0097] Optionally, the gallery data table stores the mapping between the hash value of each photo and the user ID and facial information. This gallery data table is obtained by the phone's gallery application analyzing photos when the phone screen is off. For an example illustrating how the phone's gallery application analyzes photos to obtain the gallery data table, please refer to relevant technical documentation; details will not be elaborated here.

[0098] As an example, a mobile phone can save a gallery data table in the form of a table or an array. For example, a gallery data table saved by a mobile phone can be as shown in Table 1 below.

[0099] Table 1

[0100]

[0101] It should be noted that each photo corresponds to a hash value. If the photo is of a single person, the hash value corresponds to a user ID and the facial information corresponding to that user ID. If the photo is of multiple people, the hash value corresponds to multiple user IDs, and each user ID corresponds to one facial information. The above example illustrates one hash value corresponding to one user ID.

[0102] Step 2: The phone reads the map set attributes corresponding to each map set in multiple map sets.

[0103] The atlas attributes may include, for example, one or more of the following: atlas date type (day Type), atlas location type (poiType), atlas start time (day hour begin), atlas end time (day hour end), atlas indoor / outdoor type (indoor / outdoor), and atlas hash value.

[0104] The start time of an image set can be understood as the earliest shooting time among all the photos included in the set; the end time of an image set can be understood as the latest shooting time among all the photos included in the set. The image set hash value includes the IDs (or tags) of all photos in the set.

[0105] For example, the same image set includes M photos, the time interval between any two photos in the M photos does not exceed a preset interval, and the distance difference between the shooting positions of any two photos does not exceed a preset distance; M is a positive integer.

[0106] As an example, a mobile phone can divide photos stored in a photo library application into multiple photo sets based on the shooting time and location of different photos. In this embodiment of the application, dividing different photos into different photo sets according to shooting time and location can facilitate data statistics and improve the efficiency of subsequent prediction of social relationships.

[0107] refer to Figure 6 This illustrates a flowchart of the process of dividing different photos into different albums in a gallery application. For example, such as... Figure 6 As shown, the process may include the following steps.

[0108] Step a: The phone reads the photo data. The photo data includes the shooting time (date token) and shooting location for each photo in the gallery application.

[0109] For example, the phone can read photo data from a media data table (gallery_media). This media data table is generated by the phone's gallery app analyzing each photo while the screen is off. Optionally, the photo data may also include a hash value corresponding to each photo, without limitation.

[0110] For example, the shooting location may include latitude and longitude information, such as the longitude (lat) and latitude (lon) information obtained by the mobile phone through the global positioning system (GPS) when taking a photo.

[0111] In some embodiments of this application, the multiple photos included in the gallery application may be taken by the mobile phone through the camera; or, they may also include photos downloaded by the mobile phone through a third-party application, without limitation.

[0112] Step b: The phone generates multiple image sets based on the photo data.

[0113] For example, such as Figure 7 As shown, for every two photos, the phone sorts them according to the shooting time, determining the shooting time interval between each pair of photos. If the shooting time interval does not exceed a preset interval, the phone assigns the two photos to the same image set. Conversely, if the shooting time interval exceeds the preset interval, the phone assigns the two photos to different image sets.

[0114] Based on this, after grouping the two photos into the same image set according to the shooting time interval, the phone determines the distance difference between the shooting locations of the two photos. If the distance difference does not exceed a preset distance, the phone keeps the two photos in the same image set. Conversely, if the distance difference exceeds a preset distance, the phone groups the two photos into different image sets.

[0115] Assuming the two photos above are the first photo and the second photo, through... Figure 7 In this manner, the first and second photos are grouped into the same image set. Based on this, the phone obtains the shooting time and location of the third photo. For the first, second, and third photos, the phone further determines the shooting time interval and shooting location difference between each pair of photos. If the shooting time interval between the third photo and the first photo does not exceed a preset interval, and the shooting location difference does not exceed a preset difference; and if the shooting time interval between the third photo and the second photo does not exceed a preset interval, and the shooting location difference does not exceed a preset difference, then the phone groups the third photo into the image set corresponding to the first and second photos.

[0116] Accordingly, in one example, if the time interval between the shooting of the third photo and the first photo exceeds a preset interval, or if the difference in shooting positions exceeds a preset difference, the phone will assign the third photo to another image set. Alternatively, in another example, if the time interval between the shooting of the third photo and the second photo exceeds a preset interval, or if the difference in shooting positions exceeds a preset difference, the phone will assign the third photo to another image set.

[0117] In other words, through Figure 7 The method shown involves M photos within the same image set, where the time interval between any two photos does not exceed a preset interval, and the distance difference between the shooting locations of any two photos does not exceed a preset distance. The preset interval and preset difference are not specifically limited and are subject to actual implementation. For example, the preset interval could be 3 hours, and the preset distance could be 2 kilometers.

[0118] Step c: The phone generates image set attributes for each of the multiple image sets.

[0119] Optionally, the phone can generate the image set date type based on the shooting time corresponding to each image set. For example, the image set date type can include: weekend, statutory short holiday, statutory long holiday, weekday, etc., without restriction.

[0120] Optionally, the phone generates the location type of the image atlas using the "home tag" and "company tag" saved on the phone. Optionally, if the distance between the shooting location (e.g., latitude and longitude information) corresponding to the image atlas and the location corresponding to the "home tag" is less than a threshold of 1, the location type of the image atlas generated by the phone will be "home" or "hometown," etc., without restriction. Among them, "hometown" can be obtained by querying the city or city trajectory based on the same smallest administrative region (county / city).

[0121] Optionally, the mobile phone can generate indoor / outdoor atlas types using a classification model. For example, such as... Figure 8 As shown, for a given image set, the mobile phone reads the theme tag corresponding to each photo in the set. This theme tag can be anything from food and scenery to green grass, sports and fitness, parties, luxury cars and cruise ships, etc., without restriction. Then, the mobile phone inputs the theme tag corresponding to each photo into a classification model for prediction, outputting the proportion of photos taken indoors (or outdoors). Based on the proportion of photos taken indoors (or outdoors) output by the classification model, the mobile phone predicts the indoor / outdoor type of the image set.

[0122] Understandably, mobile phones can... Figure 8 The process shown predicts the indoor / outdoor type of each atlas. The name of the classification model is not specifically defined and is subject to change based on the actual implementation.

[0123] In the above embodiments, due to GPS accuracy limitations, mobile phones cannot accurately distinguish between indoor and outdoor locations based on the photo's shooting location (i.e., latitude and longitude information). Therefore, predicting the indoor / outdoor type of the atlas using a classification model can improve the accuracy of the generated atlas attributes, facilitating subsequent improvements in the efficiency of predicting social relationships.

[0124] Step d: The phone saves the image set attributes corresponding to each image set in multiple image sets.

[0125] For example, the mobile phone can store the correspondence between the atlas and the atlas attributes in the form of an array or a table. Optionally, the correspondence between the atlas and the atlas attributes can be as shown in Table 2 below.

[0126] Table 2

[0127]

[0128] For example, in step 2 above, the mobile phone can read the map attribute corresponding to each map set from the correspondence shown in Table 2. For instance, the mobile phone can read the map attribute corresponding to map set 1, such as map set date type 1, map set location type 1, map set start time 1, map set end time 1, map set indoor / outdoor type 1, and map set hash value 1.

[0129] In some embodiments, when a new photo is added to the phone's gallery app, the phone updates the gallery based on the new photo. For example, the phone detects whether a new photo has been added while charging and the screen is off. Optionally, the phone may detect whether a new photo has been added daily while charging and the screen is off. Alternatively, the phone may periodically detect whether a new photo has been added. The period is not specifically limited and can be determined by actual settings. For example, the period could be 7 days, 10 days, etc., without limitation.

[0130] refer to Figure 9 This illustrates a schematic diagram of an atlas update process. For example, as shown below... Figure 9 As shown, the phone detects if any new photos have been added. If a new photo is detected, the phone updates the photo set. For example, the phone iterates through each photo set, comparing the shooting time and location of the new photo with the corresponding photo set attributes. If the shooting time or location of the new photo matches the photo set attributes of an existing photo set, the phone adds the new photo to the existing photo set. If the shooting time and location of the new photo do not match the photo set attributes of an existing photo set, the phone generates a new photo set based on the new photo.

[0131] For example, if the shooting time of a newly added photo falls within the range of the start time 1 to the end time 1 of the corresponding image set 1, then the newly added photo will be assigned to image set 1. Alternatively, if the shooting location of a newly added photo matches the image set location type 1 corresponding to image set 1, then the newly added photo will be assigned to image set 1.

[0132] As an example, such as Figure 9 As shown, if the phone does not detect any new photos, the process will end. Figure 9 The diagram shows the process of updating the atlas.

[0133] Step 3: The phone filters out the first group of user IDs based on the facial information corresponding to each user ID among multiple user IDs.

[0134] The first group of user IDs includes multiple user IDs. Each user ID corresponds to a user whose age is within a preset age range and whose face deflection angle is less than or equal to a face deflection angle threshold.

[0135] In some embodiments, due to errors in facial recognition performed by the mobile phone, the facial information corresponding to each user ID may include multiple ages and multiple facial angles. Based on this, for the facial information corresponding to the same face, the mobile phone determines the average (or standard deviation) of multiple ages. If the average (or standard deviation) of multiple ages is greater than threshold 1, the mobile phone excludes the corresponding user ID. Correspondingly, the mobile phone determines the average and standard deviation of multiple facial angles. If the average of multiple facial angles is greater than threshold 2, and the standard deviation of the facial angles is greater than threshold 3, the mobile phone excludes the corresponding user ID. That is, in step 3, in the first group of user IDs filtered by the mobile phone, the age difference between users corresponding to each user ID is small, and the facial angle is small. In other words, user IDs with large age differences between users corresponding to the same user ID and severe facial angles are excluded.

[0136] It should be noted that the embodiments of this application do not specifically limit the above-mentioned thresholds 1, 2, and 3, and the actual implementation shall prevail. For example, the above-mentioned threshold 1 can be 5, the above-mentioned threshold 2 can be 20, and the above-mentioned threshold 3 can be 25.

[0137] In addition, the aforementioned preset age range and face deflection angle threshold can be set according to actual implementation and are not restricted.

[0138] In this embodiment of the application, steps 1 to 3 are preprocessing steps before the mobile phone identifies the social relationship between the owner and others. This preprocessing can exclude user IDs corresponding to some erroneous facial information. At the same time, the preprocessing can also divide photos that meet the conditions into the same image set, which can improve the efficiency and accuracy of subsequent social relationship identification.

[0139] For example, the first group of user IDs determined in step 3 above may include multiple user IDs, such as a first user ID, a second user ID, a third user ID, a fourth user ID, ..., an Nth user ID, etc., without limitation. The following example illustrates the specific implementation process of the social relationship between the device owner and the users corresponding to the above multiple user IDs.

[0140] It should be noted that, in this embodiment of the application, the social relationships between the users corresponding to the multiple user IDs and the phone owner can be obtained by the phone iteratively analyzing the social relationships between the phone owner and others. The iterative process is illustrated below with an example.

[0141] Step 4: The mobile phone inputs the user information corresponding to the first user ID and the second user ID from multiple user IDs into the same person prediction model to obtain the first prediction result; the first prediction result indicates whether the face corresponding to the first user ID and the second user ID is the face of the same person.

[0142] For example, the user information corresponding to the first user ID and the second user ID includes at least: facial features and co-occurrence frequency. The facial features include at least the mean age, gender difference, and facial similarity of the faces corresponding to the first user ID and the second user ID. Optionally, the facial features may also include the standard deviation of age, the mean face angle, and the standard deviation of face angle of the faces corresponding to the first user ID and the second user ID, etc., without limitation.

[0143] For example, gender differences may include: the users corresponding to the first user ID and the second user ID are both male or both female; or, the user corresponding to the first user ID is male and the user corresponding to the second user ID is female; or, the user corresponding to the first user ID is female and the user corresponding to the second user ID is male, etc., without limitation.

[0144] The co-occurrence frequency includes at least the frequency of the first user ID and the second user ID appearing together in the same photo (or photo co-occurrence frequency). Optionally, the co-occurrence frequency may also include the frequency of the first user ID and the second user ID appearing together in the same image set (or image set co-occurrence frequency).

[0145] In some embodiments, the user information corresponding to the first user ID and the second user ID can be obtained by the mobile phone based on the facial information corresponding to the first user ID and the second user ID, as well as the determined image set attributes, which will not be elaborated further. For example, the mobile phone calculates the aforementioned facial features based on the facial information of the first user ID and the second user ID. Correspondingly, the mobile phone queries the co-occurrence frequency from multiple photos and multiple image sets based on the first user ID and the second user ID.

[0146] As an example, such as Figure 10 As shown, the mobile phone inputs the user information corresponding to the first user ID and the second user ID, such as facial features and co-occurrence frequency, into the same person prediction model and outputs the first prediction result.

[0147] Optionally, the mobile phone inputs the user information corresponding to the first user ID and the second user ID from multiple user IDs into the same person prediction model to obtain the probability of the same person. The mobile phone then obtains a first prediction result based on the probability of the same person. For example, if the probability of the same person is greater than or equal to a probability threshold, the first prediction result indicates that the faces corresponding to the first user ID and the second user ID are the faces of the same person; if the probability of the same person is less than the probability threshold, the first prediction result indicates that the faces corresponding to the first user ID and the second user ID are not the faces of the same person.

[0148] It should be noted that the probability threshold is not specifically limited in this embodiment, and the actual implementation shall prevail. For example, the probability threshold may be 0.85, or other values, without limitation.

[0149] Additionally, it should be noted that in some cases, after a mobile phone performs facial recognition, it assigns a user ID to each face. Due to potential errors in facial recognition, the phone may assign different user IDs to the same face. Therefore, in this embodiment, by predicting whether two user IDs belong to the same person, the number of subsequent iterations can be reduced, thereby lowering power consumption.

[0150] Step 5: If the first prediction result indicates that the face corresponding to the first user ID and the second user ID is the same person's face, the mobile phone assigns a first fusion ID to it and merges the user information corresponding to the first user ID and the second user ID to obtain the user information corresponding to the first fusion ID.

[0151] For example, assuming the first user ID is represented as tag_id1 and the second user ID as tag_id2, then the first merged ID can be represented as people_id1 or person_id1. Of course, the first user ID, the second user ID, and the first merged ID can also be represented in other ways, without restriction.

[0152] For example, the user information corresponding to the first fusion ID includes at least: the number of photos containing the first fusion ID, the number of image sets containing the first fusion ID, and the image set location type corresponding to the photos containing the first fusion ID. It is understood that the image set location type may include, for example, home, hometown, indoor, or outdoor, etc., without limitation.

[0153] It should be noted that, since the phone assigns a user ID to each user (or each face, one user corresponding to one face) in multiple photos within the gallery application, and the first fused ID includes both the first user ID and the second user ID, the aforementioned number of photos containing the first fused ID can be understood as: the number of photos containing both the first user ID and the second user ID. Similarly, the aforementioned number of image sets containing the first fused ID can be understood as: the number of image sets containing both the first user ID and the second user ID. Finally, the aforementioned image set location type corresponding to photos containing the first fused ID can be understood as: the image set location type corresponding to photos containing both the first user ID and the second user ID.

[0154] Optionally, if the first prediction result indicates that the faces corresponding to the first user ID and the second user ID are not the same person's faces, the mobile phone inputs the user information corresponding to the first user ID and the third user ID from among multiple user IDs into the same person prediction model to predict whether the faces corresponding to the first user ID and the third user ID are the same person's faces. That is, in this embodiment of the application, for multiple user IDs corresponding to multiple photos, the mobile phone predicts whether the faces corresponding to every two user IDs are the same person's faces. If they are the same person's faces, the mobile phone continues to execute the following steps (such as step 6); if they are not the same person's faces, the mobile phone re-predicts whether the faces corresponding to the other two user IDs are the same person's faces, and so on iteratively.

[0155] Step 6: The mobile phone inputs the user information corresponding to the first fused ID and the user information of the phone owner into the relationship prediction model to obtain the social relationship between the phone owner and the user corresponding to the first fused ID.

[0156] For example, the user information of the device owner may include: the number of photos containing the device owner ID, the number of photo sets containing the device owner ID, and the location type of the photo set corresponding to the photos containing the device owner ID.

[0157] Optionally, the user information corresponding to the first fused ID and the user information corresponding to the device owner may also include: the visual relationship between the user corresponding to the first fused ID and the device owner; such visual relationship may include, for example, closeness, back to back, proximity, or hugging, etc., without limitation.

[0158] In the above embodiments, when the mobile phone predicts the social relationship between the owner and the user corresponding to the first fused ID, it can improve the accuracy of the predicted social relationship by inputting the visual relationship between the user corresponding to the first fused ID and the owner.

[0159] As an example, such as Figure 11 As shown, the mobile phone inputs the user information corresponding to the first fused ID and the user information of the phone owner into the relationship prediction model, runs the relationship prediction model, and outputs the social relationship between the user and the phone owner corresponding to the first fused ID. For example, this social relationship may include: child (son or daughter), lover (wife or husband), parents (father or mother); or other relationships (such as colleagues, friends, best friends, etc.), without limitation.

[0160] It should be noted that, in the embodiments of this application, the aforementioned "same person prediction model" and "relationship prediction model" can be the same model, meaning that the model simultaneously has the function of predicting the same person and predicting social relationships. Alternatively, the aforementioned "same person prediction model" and "relationship prediction model" can be two different models, without limitation. For example, when the "same person prediction model" and "relationship prediction model" are the same model, the model can be, for example, the lightGBM model, or other suitable models, without limitation. Models that primarily have the ability to predict the same person and predict social relationships are all within the protection scope of the embodiments of this application.

[0161] In some embodiments of this application, in order to make the social relationships between the phone owner and others more accurate, for the user ID corresponding to the face of the same person, the phone can iterate the social relationships of the user ID multiple times. When the social relationships predicted by the phone are stable, the phone ends the prediction of the social relationships of the user ID.

[0162] For example, as Figure 5 As shown, the method also includes the following steps.

[0163] Step 7: The mobile phone inputs the user information corresponding to the first fusion ID and the third user ID among multiple user IDs into the same person prediction model to obtain the second prediction result. The second prediction result indicates whether the face corresponding to the first fusion ID and the third user ID is the face of the same person.

[0164] For example, the user information corresponding to the first fusion ID and the third user ID includes at least: facial features, co-occurrence frequency, and relationship features. The facial features include, but are not limited to, the mean age, gender difference, and facial similarity of the users corresponding to the first fusion ID and the third user ID. Optionally, the facial features may also include, but are not limited to, the standard deviation of age, the mean facial angle, and the standard deviation of facial angle of the users corresponding to the first fusion ID and the third user ID.

[0165] For example, the co-occurrence frequency includes at least the frequency of users corresponding to the first fusion ID and the third user ID appearing together in the same photo. Optionally, the co-occurrence frequency may also include the frequency of users corresponding to the first fusion ID and the third user ID appearing together in the same image set, etc., without limitation.

[0166] For example, the relationship features include the social relationship between the device owner and the user corresponding to the first merged ID.

[0167] It should be noted that, in the embodiments of this application, as Figure 5As shown, when the phone first predicts whether two user IDs belong to the same person, the user information corresponding to those two user IDs does not include relationship features. When the phone makes the nth prediction, the user information corresponding to those two user IDs includes relationship features, specifically the social relationship between the phone's previous prediction of the phone owner and the merged ID. n is a positive integer greater than or equal to 2.

[0168] For example, such as Figure 12 As shown, the mobile phone inputs the user information corresponding to the first fused ID and the third user ID among multiple user IDs, such as facial features, co-occurrence frequency, and relationship features, into the same person prediction model, and outputs the second prediction result.

[0169] It should be noted that the example of the second prediction result can be referred to the first prediction result mentioned above, which will not be repeated here.

[0170] Step 8: If the second prediction result indicates that the face corresponding to the first fusion ID and the third user ID is the same person's face, the mobile phone assigns a second fusion ID and merges the user information corresponding to the first fusion ID and the third user ID to obtain the user information corresponding to the second fusion ID.

[0171] For example, suppose the first fusion ID is represented as people_id1 and the third user ID is represented as tag_id3; then the second fusion ID can be represented as people_id2 or person_id2.

[0172] For example, the user information corresponding to the second fusion ID includes at least: the number of photos containing the second fusion ID, the number of image sets containing the first fusion ID, and the image set location type corresponding to the photos containing the second fusion ID. It is understood that the image set location type may include, for example, home, hometown, indoor, or outdoor, etc., without limitation.

[0173] It should be noted that the explanation of the user information corresponding to the second fusion ID can be found in the above embodiments, and will not be repeated here.

[0174] Step 9: The mobile phone inputs the user information corresponding to the second fusion ID and the user information of the phone owner into the relationship prediction model to obtain the social relationship between the phone owner and the user corresponding to the second fusion ID.

[0175] It should be noted that for an example illustrating the social relationship between the mobile phone owner and the user corresponding to the second merged ID, please refer to the example in step 6 above, which will not be repeated here.

[0176] Optionally, after step 5 above, if the first prediction result indicates that the faces corresponding to the first user ID and the second user ID are not the same person's faces, the mobile phone inputs the user information corresponding to the first user ID and the third user ID from among the multiple user IDs into the same person prediction model to predict whether the faces corresponding to the first user ID and the third user ID are the same person's faces. That is, in this embodiment of the application, for multiple user IDs corresponding to multiple photos, the mobile phone predicts whether the faces corresponding to every two user IDs are the same person's faces. If they are the same person's faces, the mobile phone continues to execute the following steps (as in step 6); if they are not the same person's faces, the mobile phone re-predicts whether the faces corresponding to the other two user IDs are the same person's faces, and so on iteratively.

[0177] For example, if the first prediction result indicates that the faces corresponding to the first user ID and the second user ID are not the same person's faces, the mobile phone inputs the user information corresponding to the first user ID and the fourth user ID from among multiple user IDs into the same person prediction model to obtain a third prediction result. The third prediction result indicates whether the faces corresponding to the first user ID and the fourth user ID are the same person's faces.

[0178] If the third prediction result indicates that the faces corresponding to the first user ID and the fourth user ID are the same person's faces, a third fusion ID is assigned, and the user information corresponding to the first user ID and the fourth user ID is merged to obtain the user information corresponding to the third fusion ID. Then, the mobile phone inputs the user information corresponding to the third fusion ID and the phone owner's user information into the relationship prediction model to obtain the social relationship between the phone owner and the user corresponding to the third fusion ID.

[0179] As can be seen from the above embodiments, in this application embodiment, for the user ID corresponding to the same person's face, the mobile phone can iterate the social relationships of the user ID multiple times. When the social relationships predicted by the mobile phone are stable, the mobile phone ends the prediction of the social relationships of that user ID. That is, the mobile phone iterates through steps 4 to 5 above, and ends the loop iteration when the social relationships predicted by the mobile phone are stable.

[0180] In some embodiments of this application, if the second prediction result indicates that the faces corresponding to the first fusion ID and the third user ID are not the same person's faces, the electronic device saves the social relationship between the device owner and the user corresponding to the first fusion ID, and no longer updates the social relationship between the device owner and the user corresponding to the first fusion ID. That is, during the iterative process, if the user ID included in the currently allocated fusion ID is consistent with the user ID included in the previously allocated fusion ID, the phone stops the iterative process and saves the predicted social relationship between the device owner and the user corresponding to the fusion ID in the phone.

[0181] Alternatively, in some other embodiments of this application, for the same face corresponding to a user ID, the mobile phone updates the social relationship of the user ID once, incrementing the social relationship iteration count of the user ID by 1; when the social relationship iteration count of the user ID reaches a preset number, the mobile phone stops updating the social relationship of the user ID, that is, the mobile phone stops looping and iterating.

[0182] The preset number of times is not specifically limited and is subject to actual settings. For example, the preset number of times could be 5 times, or any other suitable number, without restriction.

[0183] In summary, in this embodiment of the application, the mobile phone can predict the social relationship between the phone owner and the user ID corresponding to the same face through the above-described method. It is understood that the phone's photo library application includes multiple photos, and these photos contain multiple faces. Therefore, the phone can predict the social relationship between each face and the corresponding user ID, thereby predicting multiple social relationships and storing the social relationship between the phone owner and the user ID corresponding to the same face.

[0184] Based on this, in response to the user's first voice input, the phone determines the existence of a first social relationship with the owner. Then, responding to the first voice, the phone identifies the first social relationship from multiple social relationships stored on the phone and obtains the target user ID corresponding to the first social relationship. Next, the phone retrieves a first photo containing the user corresponding to the target user ID from the gallery application and generates a target video based on this first photo. The interface interaction process can be referred to in the above embodiment and will not be repeated here.

[0185] In the embodiments of this application, such as Figure 13 As shown, the phone first predicts whether a person is the same based on multiple photos included in the gallery app, merging the user IDs of faces belonging to the same person. Then, based on the merged ID, it predicts the social relationship between the phone owner and the user corresponding to the merged ID.

[0186] For example, such as Figure 14 As shown, suppose the phone's photo library application contains multiple photos, each corresponding to x user IDs, where x is an integer greater than or equal to 2. The phone performs a same-person prediction by fusing user IDs belonging to the same person's face. For example, fusing user ID1 and user ID2 results in a fused ID, represented as fused ID1. Similarly, fusing user ID3 and user ID4 results in a fused ID, represented as fused ID2.

[0187] Furthermore, such as Figure 14 As shown, the phone predicts the social relationship between the phone owner and the user corresponding to Fusion ID1 (e.g., "parents"), and predicts the social relationship between the phone owner and the user corresponding to Fusion ID2 (e.g., "child").

[0188] In summary, by adopting the solution of this application embodiment, the mobile phone can predict in advance the social relationship between the phone owner and the user ID corresponding to the same face. Then, after receiving the voice input by the user, the mobile phone determines the instruction used to indicate the existence of a social relationship with the phone owner. In response to the instruction used to indicate the existence of a social relationship with the phone owner, the mobile phone generates a video based on the photos of the user ID corresponding to the user with a social relationship with the phone owner included in the photo library application. This can solve the problems of poor logic, weak relevance, and poor human-computer interaction efficiency of the generated video.

[0189] It should be noted that the contents described in the various embodiments of this application can be used to explain the technical solutions in other embodiments of this application. The technical features described in each embodiment can also be applied in other embodiments, and new solutions can be formed by combining the technical features in other embodiments. This application only provides an exemplary list of several embodiments for illustration and does not mean that this application is limited thereto.

[0190] This application provides an electronic device that may include a memory, one or more processors, and a display screen. The memory stores computer program code, which includes computer instructions. When the computer instructions are executed by the processor, the electronic device performs the various functions or steps performed by the mobile phone in the above embodiment. The structure of the electronic device can be referred to the above. Figure 1 The structure of the electronic device 100 shown.

[0191] This application also provides a chip system for use in electronic devices. For example... Figure 15 As shown, the chip system 1100 includes at least one processor 1101 and at least one interface circuit 1102. The processor 1101 can be one of the embodiments described above. Figure 1 The processor 110 is shown. Based on this, the interface circuit 1102 can be, for example, an interface circuit between the processor 110 and external memory; or an interface circuit between the processor 110 and internal memory.

[0192] The processor 1101 and interface circuit 1102 described above can be interconnected via lines. For example, interface circuit 1102 can be used to receive signals from other devices (e.g., the memory of electronic device 100). As another example, interface circuit 1102 can be used to send signals to other devices (e.g., processor 1101). Exemplarily, interface circuit 1102 can read instructions stored in memory and send those instructions to processor 1101. When the instructions are executed by processor 1101, the electronic device can perform the various functions or steps performed by the mobile phone in the above embodiments. Of course, the chip system may also include other discrete components, and this application embodiment does not specifically limit this.

[0193] This application also provides a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform various functions or steps performed by the mobile phone in the above method embodiments.

[0194] This application also provides a computer program product that, when run on a computer, causes the computer to perform the various functions or steps performed by the mobile phone in the above method embodiments.

[0195] It should be noted that the terms "first" and "second," etc., in the embodiments and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0196] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0197] It should be understood that in this application, "at least one (item)" means one or more. "More than one" means two or more. "At least two (items)" means two or three or more. "And / or" is used to describe the relationship between related objects, indicating that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple. "...when" and "if" both mean that a corresponding action will be taken under certain objective circumstances, not a time limit, nor do they require a judgment action at the time of implementation, nor do they imply any other limitations.

[0198] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner to facilitate understanding.

[0199] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules according to the system, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0200] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0201] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0202] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware or as a software functional unit.

[0203] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0204] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A photo processing method, characterized in that, Applied to electronic devices, including: The electronic device receives a first voice input from a user and displays a first text corresponding to the first voice on a first interactive interface; the first text includes information indicating the device owner and information indicating a first social relationship with the device owner. In response to the first voice, a first photo and a first control are displayed on the first interactive interface; wherein, the first photo contains a user corresponding to a target user ID, and the user corresponding to the target user ID has a first social relationship with the device owner; the target user ID is determined from multiple social relationships stored in the electronic device based on the first social relationship indicated by the first text, and the multiple social relationships are obtained by the electronic device iteratively analyzing the social relationships between the device owner and others; In response to a first operation input by the user to the first control, the electronic device displays a thumbnail of the target video; the target video is generated by the electronic device based on a first photo of the user corresponding to the target user ID in a gallery application.

2. The method according to claim 1, characterized in that, The electronic device includes multiple photos, each photo corresponding to a multiple user ID, and each user ID corresponding to a face; the electronic device stores multiple social relationships, specifically: the electronic device stores the social relationships between the device owner and the user IDs corresponding to the same face; the method further includes: The electronic device inputs the user information corresponding to the first user ID and the second user ID from the plurality of user IDs into the same person prediction model to obtain a first prediction result. The first prediction result indicates whether the face corresponding to the first user ID and the second user ID is the face of the same person. If the first prediction result indicates that the faces corresponding to the first user ID and the second user ID are the same person's face, the electronic device assigns a first fusion ID and fuses the user information corresponding to the first user ID and the second user ID to obtain the user information corresponding to the first fusion ID; The electronic device inputs the user information corresponding to the first fusion ID and the user information of the device owner into a relationship prediction model to obtain the social relationship between the device owner and the user corresponding to the first fusion ID.

3. The method according to claim 2, characterized in that, The method further includes: The electronic device inputs the user information corresponding to the first fusion ID and the third user ID among the plurality of user IDs into the same person prediction model to obtain a second prediction result. The second prediction result indicates whether the face corresponding to the first fusion ID and the third user ID is the face of the same person. If the second prediction result indicates that the face corresponding to the first fusion ID and the third user ID is the same person's face, the electronic device assigns a second fusion ID and merges the user information corresponding to the first fusion ID and the third user ID to obtain the user information corresponding to the second fusion ID; The electronic device inputs the user information corresponding to the second fusion ID and the user information of the device owner into a relationship prediction model to obtain the social relationship between the device owner and the user corresponding to the second fusion ID.

4. The method according to claim 3, characterized in that, The method further includes: If the second prediction result indicates that the faces corresponding to the first fusion ID and the third user ID are not the same person's faces, the electronic device saves the social relationship between the device owner and the user corresponding to the first fusion ID, and no longer updates the social relationship between the device owner and the user corresponding to the first fusion ID.

5. The method according to any one of claims 2-3, characterized in that, The method further includes: For the same face corresponding to a user ID, the electronic device updates the social relationship of the user ID once by incrementing the social relationship iteration number of the user ID by 1; When the number of iterations of the social relationships of the user ID reaches a preset number, the electronic device stops updating the social relationships of the user ID.

6. The method according to claim 2, characterized in that, The user information corresponding to the first user ID and the second user ID includes at least: facial features and co-occurrence frequency; wherein, the facial features include at least the average age, gender difference, and facial similarity of the users corresponding to the first user ID and the second user ID; the co-occurrence frequency includes at least the frequency of the users corresponding to the first user ID and the second user ID appearing together in the same photo.

7. The method according to claim 6, characterized in that, The co-occurrence frequency also includes the frequency of the first user ID and the second user ID appearing together in the same image set; wherein, the same image set includes M photos, the shooting time interval between any two photos in the M photos does not exceed a preset interval, and the distance difference between the shooting positions of any two photos does not exceed a preset distance; M is a positive integer.

8. The method according to claim 3, characterized in that, The user information corresponding to the first fusion ID and the third user ID includes at least: facial features, co-occurrence frequency, and relationship features; wherein, the facial features include at least the average age, gender difference, and facial similarity of the users corresponding to the first fusion ID and the third user ID; the co-occurrence frequency includes at least the frequency of the users corresponding to the first fusion ID and the third user ID appearing together in the same photo; and the relationship features include the social relationship between the device owner and the user corresponding to the first fusion ID.

9. The method according to any one of claims 2-4 and 6-8, characterized in that, The electronic device inputs the user information corresponding to the first user ID and the second user ID from the plurality of user IDs into the same person prediction model to obtain a first prediction result, including: The electronic device inputs the user information corresponding to the first user ID and the second user ID from the plurality of user IDs into the same person prediction model to obtain the probability of the same person. The electronic device obtains the first prediction result based on the probability of the same person; If the probability of being the same person is greater than or equal to the probability threshold, the first prediction result indicates that the faces corresponding to the first user ID and the second user ID are the faces of the same person; if the probability of being the same person is less than the probability threshold, the first prediction result indicates that the faces corresponding to the first user ID and the second user ID are not the faces of the same person.

10. The method according to any one of claims 2-4 and 6-8, characterized in that, The user information corresponding to the first fusion ID includes at least: the number of photos containing the first fusion ID, the number of image sets containing the first fusion ID, and the image set location type corresponding to the photos containing the first fusion ID; The user information of the device owner includes: the number of photos containing the device owner ID, the number of image sets containing the device owner ID, and the image set location type corresponding to the photos containing the device owner ID; The location types of the atlas include: home, hometown, indoor or outdoor.

11. The method according to claim 10, characterized in that, The user information corresponding to the first fusion ID and the user information corresponding to the device owner also include: the visual relationship between the user corresponding to the first fusion ID and the device owner; the visual relationship includes: close, back to back, near, or hugging.

12. An electronic device, characterized in that, include: Memory and one or more processors, display screen; The memory stores computer program code, which includes computer instructions; when the computer instructions are executed by the processor, the electronic device performs the method as described in any one of claims 1-11.

13. A chip system, characterized in that, include: At least one processor and an interface; The interface is used to receive instructions and transmit them to the at least one processor; The at least one processor executes the instructions to cause the electronic device to perform the method as described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-11.

15. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1-11.

Images