Feature association method and electronic device

Abstract

The application provides a feature association method and an electronic device. The method comprises feature extraction of a first sub-file for a first speaking object in a first file by the electronic device to obtain a first modal feature and a second modal feature of the first speaking object, and determining a first feature connector according to the first modal feature and the second modal feature, wherein the first modal feature is different from the second modal feature, the first feature connector is used to describe an association relationship between the first modal feature and the second modal feature, the first feature connector is also used to connect information corresponding to the first modal feature and information corresponding to the second modal feature, and the information corresponding to the first modal feature and the information corresponding to the second modal feature are located in different applications of the electronic device. The association of different modal data can be realized through the feature connector, so that the information corresponding to multiple modal features located in different applications can be accurately connected, and effective information interworking between different applications is facilitated.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a feature association method and an electronic device. Background Technology

[0002] With the rapid development of electronic devices (such as smartphones), the applications installed on these devices are becoming increasingly diverse. Take photo gallery applications as an example: these applications can store user media data collected by cameras or webcams. Among this user media data, portraits account for nearly half or even more. In this context, facial recognition technology has become a standard feature of electronic devices. Facial recognition technology uses computer vision techniques to process user media data (such as face detection, feature extraction, and clustering) to create a portrait album.

[0003] However, it is unknown who the people in the portrait album are, and users need to actively annotate the portraits. However, few users actively annotate portraits at present, which makes it impossible to link the portrait data in the photo library application with external data. As a result, different modal data (such as visual modal portrait data and audio modal sound data) cannot be effectively linked.

[0004] Further research is needed to determine how to correlate data from different modalities. Summary of the Invention

[0005] This application provides a feature association method and an electronic device for realizing the association of different modal data, so that the information corresponding to multiple modal features in different applications can be accurately connected, facilitating effective information sharing between different applications.

[0006] Firstly, this application provides a feature association method, which can be executed by an electronic device or a component within the electronic device (such as a processor, processing unit, or chip). For example, the following describes the execution of the feature association method by an electronic device. The method may include the following steps: The electronic device extracts features from a first sub-file of a first file targeting a first speaking object, obtaining a first modal feature and a second modal feature of the first speaking object. Then, the electronic device determines a first feature connector based on the first and second modal features. The first feature connector describes the association relationship between the first and second modal features. The first feature connector also connects information corresponding to the first and second modal features. The information corresponding to the first and second modal features resides in different applications of the electronic device. The first and second modal features are different and can be any two of facial features, voice features, semantic features, and behavioral features.

[0007] In this method, multimodal features of the first sub-file in the first file (the multimodal features of the first sub-file are multimodal features existing in the same time and space) are extracted, and then the correlation between multiple modal features is established to form a feature connector. It can be understood that multiple modal features existing in the same time and space are highly correlated. Thus, the feature connector enables the association of different modal data, allowing the information corresponding to multiple modal features in different applications to be accurately connected. This facilitates effective information sharing (or information linkage) between different applications, that is, it establishes information (or data) connections between different applications, which helps to lay a foundation for better data understanding and data mining in the future, and can bring users a more intelligent experience.

[0008] In one possible implementation, the first file includes an audio file and an image file; the method further includes: the electronic device acquiring M sub-audio files containing sound information from the audio file, where M is an integer greater than or equal to 1, and the M sub-audio files correspond to a first speaking object; then, the electronic device can acquire M sub-image files corresponding to the M sub-audio files from the image file; then, the electronic device can acquire N sub-image files from the M sub-image files, wherein the N sub-image files contain human image information and the lips of the human image show movement changes; the first sub-file may include the N sub-image files and the N sub-audio files corresponding to the N sub-image files from the M sub-audio files.

[0009] In the above implementation, based on the high correlation between modal features in the same time and space, the first file rich in multimodal information is decomposed (or divided) into at least one sub-file. Each sub-file includes modal information (such as image information, audio information, etc.) in the same time and space. This makes it easier for electronic devices to extract modal features in the same time and space from the sub-files in a timely and effective manner. In turn, it makes it easier for electronic devices to establish associations between modal features in the same time and space, associate modal features in the same time and space, and form data linkage.

[0010] In one possible implementation, the first modal feature is a human face feature, and the second modal feature is a voice feature; the electronic device performs feature extraction on a first sub-file of the first file targeting the first speaking object to obtain the first modal feature and the second modal feature of the first speaking object, including: the electronic device can use a human face recognition model to extract human face features from the first sub-file to obtain the human face features of the first speaking object, and the electronic device can use a voice recognition model to extract voice features from the first sub-file to obtain the voice features of the first speaking object.

[0011] In the above implementation, by using different models to extract different modal features, the modal feature extraction can be made more accurate and better meet the actual feature extraction needs. This makes the electronic device's extraction of different modal features relatively independent, with each modal feature corresponding to an independent feature extraction capability, that is, each modal feature corresponding to a matching feature extraction model. For example, a facial recognition model can be used for facial features, and a voice recognition model can be used for voice features, and so on.

[0012] In one possible implementation, the first modal feature is a human face feature, and the second modal feature is a behavioral feature; the electronic device performs feature extraction on a first sub-file of the first file targeting the first speaking object to obtain the first modal feature and the second modal feature of the first speaking object, including: the electronic device can use a human face recognition model to extract human face features from the first sub-file to obtain the human face features of the first speaking object, and the electronic device can use a behavioral recognition model to extract behavioral features from the first sub-file to obtain the behavioral features of the first speaking object.

[0013] In the above implementation, by using different models to extract different modal features, the modal feature extraction can be made more accurate and better meet the actual feature extraction needs. This allows electronic devices to extract different modal features relatively independently, with each modal feature corresponding to an independent feature extraction capability, that is, each modal feature corresponds to a matching feature extraction model. For example, a facial recognition model can be used for facial features, and a behavior recognition model can be used for behavioral features, and so on.

[0014] In one possible implementation, the first modal feature is a human image feature, and the second modal feature is a semantic feature; the electronic device performs feature extraction on a first sub-file of the first file targeting the first speaking object to obtain the first modal feature and the second modal feature of the first speaking object, including: the electronic device can use a human image recognition model to extract human image features from the first sub-file to obtain the human image features of the first speaking object, and the electronic device can use a semantic recognition model to extract semantic features from the first sub-file to obtain the semantic features of the first speaking object.

[0015] In the above implementation, by using different models to extract different modal features, the modal feature extraction can be made more accurate and better meet the actual feature extraction needs. This makes the electronic device's extraction of different modal features relatively independent, with each modal feature corresponding to an independent feature extraction capability, that is, each modal feature corresponding to a matching feature extraction model. For example, a facial recognition model can be used for facial features, and a semantic recognition model can be used for semantic features, and so on.

[0016] In one possible implementation, the first modal feature is a sound feature, and the second modal feature is a semantic feature; the electronic device performs feature extraction on a first sub-file of the first file targeting the first speaking object to obtain the first modal feature and the second modal feature of the first speaking object, including: the electronic device can use a sound recognition model to extract sound features from the first sub-file to obtain the sound features of the first speaking object, and the electronic device can use a semantic recognition model to extract semantic features from the first sub-file to obtain the semantic features of the first speaking object.

[0017] In the above implementation, by using different models to extract different modal features, the modal feature extraction can be made more accurate and better meet the actual feature extraction needs. This makes the electronic device's extraction of different modal features relatively independent, with each modal feature corresponding to an independent feature extraction capability, that is, each modal feature corresponding to a matching feature extraction model. For example, a sound recognition model can be used for sound features, and a semantic recognition model can be used for semantic features, and so on.

[0018] In one possible implementation, the first modal feature is a voice feature, and the second modal feature is a behavior feature; the electronic device performs feature extraction on a first sub-file of the first file targeting the first speaking object to obtain the first modal feature and the second modal feature of the first speaking object, including: the electronic device can use a voice recognition model to extract voice features from the first sub-file to obtain the voice features of the first speaking object, and the electronic device can use a behavior recognition model to extract behavior features from the first sub-file to obtain the behavior features of the first speaking object.

[0019] In the above implementation, by using different models to extract different modal features, the modal feature extraction can be made more accurate and better meet the actual feature extraction needs. This makes the electronic device's extraction of different modal features relatively independent, with each modal feature corresponding to an independent feature extraction capability, that is, each modal feature corresponding to a matching feature extraction model. For example, a sound recognition model can be used for sound features, and a behavior recognition model can be used for behavior features, and so on.

[0020] In one possible implementation, the first modal feature is a semantic feature, and the second modal feature is a behavioral feature; the electronic device extracts features from a first sub-file of the first file targeting the first speaking object to obtain the first modal feature and the second modal feature of the first speaking object, including: the electronic device can use a semantic recognition model to extract semantic features from the first sub-file to obtain the semantic features of the first speaking object, and the electronic device can use a behavioral recognition model to extract behavioral features from the first sub-file to obtain the behavioral features of the first speaking object.

[0021] In the above implementation, by using different models to extract different modal features, the modal feature extraction can be made more accurate and better meet the actual feature extraction needs. This makes the electronic device's extraction of different modal features relatively independent, with each modal feature corresponding to an independent feature extraction capability, that is, each modal feature corresponding to a matching feature extraction model. For example, a semantic recognition model can be used for semantic features, and a behavior recognition model can be used for behavioral features, and so on.

[0022] In one possible implementation, the method further includes: the electronic device obtaining first information corresponding to a first speaking object from a first application of the electronic device; then, the electronic device extracting features from the first information to obtain third modal features; if the third modal features match the first modal features indicated by the first feature connector, the electronic device can obtain second information corresponding to the second modal features from a second application of the electronic device; then, the electronic device can display prompt information in the first application, wherein the prompt information is used to prompt the user to set the second information in the first application.

[0023] In the above implementation, after obtaining one or more feature connectors, the electronic device can perform cross-modal feature matching, thereby enabling data communication between different applications. For example, the electronic device can obtain first information from a first application, match the third modal feature extracted from the first information with one or more feature connectors, and determine which feature connector indicates which modal feature the third modal feature matches. If the third modal feature matches the first modal feature indicated by the first feature connector, the electronic device can obtain second information corresponding to the second modal feature from a second application and display a prompt message in the first application to prompt the user to set the corresponding second information for the first information in the first application. For example, taking a gallery application as the first application and a calling application as the second application, the prompt message can be used to prompt the user to set contact information for the portrait image corresponding to the first person being spoken to in the gallery application. In this way, this implementation can establish a relationship between the first application and the second application, that is, establish a relationship in different structured data, thereby enabling data communication between these two different applications, allowing them to link information and share each other's information. Furthermore, feature connectors can link two originally unrelated modal features or modal information together, which helps to facilitate information linkage between different applications.

[0024] In one possible implementation, the first application is an application with audio acquisition capabilities, and the first information is audio data. The electronic device performs feature extraction on the first information to obtain third modal features, including: the electronic device segments and clusters the audio data to obtain K cluster information, wherein each of the K cluster information can be used to indicate data corresponding to a speaking object in the audio data, or each of the K cluster information can be used to describe the time information of a speaking object in the audio data, where K is an integer greater than or equal to 1. Then, the electronic device can segment and cut the audio data according to the K cluster information to obtain K audio segments. Then, the electronic device can perform feature extraction on the K audio segments to obtain K modal features, wherein the K modal features include the third modal features.

[0025] In the above implementation, feature extraction is performed on audio data from applications with audio capture capabilities (such as call applications). The extracted features (such as voice features or semantic features) are then matched with one or more feature connectors to determine which feature connector indicates which feature (such as voice or semantic features) matches the feature. If the feature matches a feature indicated by a particular connector, the electronic device can obtain second information (such as a portrait image) corresponding to the second modal feature (such as a portrait feature) from a second application (such as a gallery application). A prompt message is then displayed in the application with audio capture capabilities, prompting the user to set a profile picture for the contact corresponding to that voice feature within the application. This establishes a connection between the application with audio capture capabilities and the gallery application, facilitating data exchange between these two different applications and enabling information sharing between them.

[0026] In one possible implementation, the electronic device extracts features from K audio segments to obtain K modal features, including: the electronic device extracts sound features from the K audio segments to obtain K sound features, wherein the K modal features are K sound features; or, the electronic device extracts semantic features from the K audio segments to obtain K semantic features, wherein the K modal features are K semantic features.

[0027] In the above implementation, sound feature extraction or semantic feature extraction can be performed on K audio segments according to actual needs. This makes the feature extraction of K audio segments more flexible and can meet different user needs.

[0028] In one possible implementation, the first application is a gallery application, and the first information is image data; the electronic device performs feature extraction on the first information to obtain third modality features, including: the electronic device uses a human face recognition model to extract human face features from the image data to obtain human face features of the first speaking object, wherein the third modality features are human face features; or, the electronic device uses a behavior recognition model to extract behavior features from the image data to obtain behavior features of the first speaking object, wherein the third modality features are behavior features.

[0029] In the above implementation, facial features are extracted from image data in the gallery application. These extracted features are then matched with one or more feature connectors to determine which feature connector indicates which facial feature matches. If the facial feature matches the feature indicated by the first feature connector, the electronic device can obtain second information (such as contact information) corresponding to the second modal feature (such as voice feature) from a second application (such as a calling application). A prompt message is then displayed in the gallery application, prompting the user to set contact information for the facial image corresponding to that feature in the gallery application. This establishes a connection between the gallery application and the second application, facilitating data exchange between these two different applications and enabling information sharing between them.

[0030] Secondly, this application provides an electronic device that has the functions involved in the first aspect above. For example, the electronic device includes modules, units, or means for performing the operations involved in the first aspect above. The functions, units, or means can be implemented by software, or by hardware, or by hardware executing corresponding software.

[0031] Thirdly, embodiments of this application provide an electronic device including at least one processor, which can be coupled to a memory.

[0032] The memory may store the necessary computer programs or instructions for implementing the functions involved in the first aspect described above. The at least one processor may execute the computer programs or instructions stored in the memory, and when the computer programs or instructions are executed, cause the electronic device to implement the methods in the first aspect described above and any possible implementation of the first aspect.

[0033] In one possible implementation, the electronic device may also include a display screen for displaying a user interface.

[0034] In one possible implementation, the electronic device also includes the aforementioned memory.

[0035] In one possible implementation, the aforementioned electronic device may also include an interface circuit (or communication interface or transceiver). The interface circuit enables the electronic device to communicate with other devices, for example, to receive signals from other devices and transmit them to the at least one processor, or to send signals from the at least one processor of the electronic device to other devices, such as the transmission or reception of data and / or signals. Exemplarily, the communication interface may be a transceiver, circuit, bus, module, or other type of communication interface.

[0036] Fourthly, this application provides a computer program product comprising a computer program or instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0037] Fifthly, this application provides a computer-readable storage medium storing a computer program or instructions that, when executed by a computer, cause the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0038] Sixthly, this application provides a chip that may include a processor, the chip being coupled to memory, and the chip executing program instructions in the memory to perform the methods described in the first aspect or any possible implementation thereof. Here, "coupling" means that two components are directly or indirectly connected to each other, such as coupling can refer to an electrical connection between two components.

[0039] In one possible implementation, the memory is included in the chip.

[0040] Seventhly, this application also provides a chip system including a processor for supporting a computer device in implementing the methods described in the first aspect or any possible implementation thereof. In one possible implementation, the chip system further includes a memory for storing programs and data necessary for the computer device. The chip system may be composed of chips or may include chips and other discrete devices.

[0041] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0042] Figure 1a An exemplary schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application is shown;

[0043] Figure 1b An exemplary schematic diagram of the software structure of an electronic device provided in an embodiment of this application is shown;

[0044] Figure 2a An exemplary diagram illustrates a feature association process provided in an embodiment of this application;

[0045] Figure 2b An exemplary schematic diagram of a feature matching process provided in an embodiment of this application is shown;

[0046] Figure 3 This illustration shows a schematic diagram of associating facial features with voice features according to an embodiment of this application;

[0047] Figure 4 An exemplary flowchart of a feature association method provided in an embodiment of this application is shown.

[0048] Figure 5 An exemplary illustration shows a video frame and audio segment provided in an embodiment of this application;

[0049] Figure 6 This illustration provides a schematic diagram illustrating the relationship between different features in the same time and space, as provided in an embodiment of this application.

[0050] Figure 7a An exemplary diagram of a vector database provided in an embodiment of this application is shown;

[0051] Figure 7b This illustration shows another vector database provided in an embodiment of this application;

[0052] Figure 8 An exemplary schematic diagram illustrates a form of audio data provided in an embodiment of this application;

[0053] Figure 9a An exemplary diagram illustrating the presentation of a prompt message 1 provided in an embodiment of this application is shown.

[0054] Figure 9b An exemplary diagram illustrating another prompt message 1 provided in an embodiment of this application is shown.

[0055] Figure 9c This illustration shows a schematic diagram of setting an avatar for a contact P1 according to an embodiment of this application;

[0056] Figure 10a An exemplary diagram illustrating the presentation of a prompt message 2 provided in an embodiment of this application is shown.

[0057] Figure 10b An exemplary diagram illustrating another type of prompt information 2 provided in an embodiment of this application is shown.

[0058] Figure 10cThis illustration shows a schematic diagram of setting contact information for a portrait image according to an embodiment of this application;

[0059] Figure 11 An exemplary schematic diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0060] The feature association method provided in this application can be applied to electronic devices. Electronic devices, also known as user equipment (UE), mobile station (MS), mobile terminal (MT), etc., are devices that provide voice or data connectivity to users, and can also be Internet of Things (IoT) devices. For example, electronic devices include handheld devices with wireless connectivity, vehicle-mounted devices, etc. Currently, electronic devices can include: mobile phones, tablets, laptops, PDAs, mobile internet devices (MIDs), wearable devices (such as smartwatches, smart bracelets, pedometers, etc.), in-vehicle equipment (such as cars, bicycles, electric vehicles, airplanes, ships, trains, high-speed trains, etc.), virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, smart home devices (such as refrigerators, televisions, air conditioners, electricity meters, etc.), intelligent robots, workshop equipment, wireless terminals in self-driving vehicles, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, or wireless terminals in smart homes, and flying equipment (such as intelligent robots, hot air balloons, drones, airplanes), etc. Electronic devices can also be devices that act as terminals in device-to-device (D2D) communication. The electronic devices involved in the embodiments of this application may be foldable electronic devices, such as foldable mobile phones, foldable tablets, etc., and this application is not limited thereto. Furthermore, exemplary embodiments of the electronic devices include, but are not limited to, those equipped with… HarmonyOS Or electronic devices with other operating systems.

[0061] The following explanations cover the relevant terms used in the embodiments of this application. Unless otherwise specified, these explanations are provided to support the meanings of the relevant terms and to make the embodiments of this application easier to understand, and should not be construed as strict limitations on the relevant terms within the scope of protection claimed in this application.

[0062] (1) Application (APP): A software program that can perform one or more specific functions. Typically, multiple applications can be installed on an electronic device. Examples include browser applications, camera applications, SMS applications, email applications, video applications, and music applications. The applications mentioned below can be those pre-installed at the factory or those downloaded by the user from the internet or obtained from other electronic devices during use.

[0063] The application can be an instant app (IAPP), which does not require explicit installation or uninstallation. Instant apps are installed automatically by the system's application framework and can be used immediately. Uninstallation of instant apps can be managed automatically by the system's application framework. In some examples, instant apps include quick apps, mini-programs, and service cards.

[0064] (2) A process can refer to a single running activity of a program with a certain independent function. It is the carrier for the running of an application. It can also be understood as a running instance of an application, a dynamic execution of the application. For example, when a user runs the Notepad program, the user creates a process to contain the code that makes up Notepad.exe and the dynamic link libraries it needs to call. A process is an independent unit for system resource allocation and scheduling. Each process has its own memory space, and different processes are independent of each other.

[0065] (3) System services refer to one of the functions provided by the operating system, which are programs, routines or processes that perform specified system functions in order to support other programs, especially low-level (close to hardware) programs.

[0066] It should be noted that in the description of this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, "at least one of A, B, and C" includes A, B, C, AB, AC, BC, or ABC. Furthermore, unless otherwise specified, the ordinal numbers "first," "second," "third," etc., mentioned in the embodiments of this application are used to distinguish multiple objects and are not used to limit the order, sequence, priority, or importance of multiple objects. In addition, the terms "including," "comprising," "having," and their variations appearing in this application all mean "including but not limited to," unless otherwise specifically emphasized.

[0067] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0068] Figure 1a An exemplary illustration shows a hardware structure diagram of an electronic device provided in an embodiment of this application. Figure 1a Based on the example shown, other variations in the structure are also possible. For example... Figure 1a As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0069] The following is about Figure 1aThe components of the electronic device 100 shown are described in detail.

[0070] Processor 110 may include one or more processing units, such as a central processing unit (CPU), application processor (AP), microcontroller unit (MCU), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. The controller may serve as the central nervous system and command center of the electronic device 100. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution. Processor 110 may also include memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has recently used or is reusing. If the processor 110 needs to reuse the instruction or data, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves system efficiency.

[0071] The feature association method provided in this application embodiment can be implemented by the processor 110 controlling or calling other components, such as calling the software program of this application embodiment stored in the internal memory 121, or calling the software program of this application embodiment stored in a third-party device through the external memory interface 120, so that the electronic device 100 can automatically perform feature association. The processor 110 may include different devices. For example, when integrating a CPU and a GPU, the CPU and GPU can cooperate to execute the feature association method provided in this application embodiment. For example, part of the software operation in this method is executed by the CPU, and another part is executed by the GPU. For example, the GPU extracts human portrait features from video images and extracts sound features (or voice features) from audio data to obtain faster processing efficiency.

[0072] USB interface 130 is an interface compliant with the USB standard specification, specifically a mini USB interface, a micro USB interface, a USB Type-C interface, etc. USB interface 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. Charging management module 140 receives charging input from the charger. Power management module 141 connects battery 142, charging management module 140, and processor 110. Power management module 141 receives input from battery 142 and / or charging management module 140 to power processor 110, internal memory 121, external memory, display 194, camera 193, and wireless communication module 160, etc.

[0073] The wireless communication function of electronic device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0074] The mobile communication module 150 can provide a wireless communication solution applied to the electronic device 100, including second-generation (2G), third-generation (3G), fourth-generation (4G), fifth-generation (5G), or future communication. The mobile communication module 150 may include filters, switches, power amplifiers, low-noise amplifiers (LNAs), etc., and each of these components may be present in one or more units. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0075] The wireless communication module 160 can provide a wireless communication solution for use on the electronic device 100. This wireless communication includes, for example, wireless local area networks (WLANs) (such as Wi-Fi networks), Bluetooth (BT), global navigation satellite systems (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0076] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include one or more of the following: Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-CDMA), Long Term Evolution (LTE), 5G mobile communication system, BT, GNSS, WLAN, NFC, FM, IR, or future communication systems. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0077] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1. In this embodiment, the display screen 194 can be used to display a main interface, an application interface, granularity adjustment controls, etc.

[0078] Camera 193 is used to capture still images or videos. Camera 193 may include a front-facing camera and a rear-facing camera.

[0079] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system and software code for at least one application (e.g., Huawei Video, Changlian, etc.). The data storage area may store data generated during the use of electronic device 100 (e.g., images, videos, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0080] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, images, videos, and other files can be saved on the external memory card.

[0081] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0082] Understandable. Figure 1a The components shown do not constitute a specific limitation on the electronic device. The electronic device may also include more or fewer components than shown, or combine some components, or split some components, or have different component arrangements. In the following embodiments, [the following text is incomplete and likely refers to a different embodiment]. Figure 1a The electronic device 100 shown is used as an example for introduction.

[0083] Figure 1a The hardware structure of the electronic device has been introduced; the software structure will now be described. The software system of the electronic device provided in this application embodiment can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses a layered architecture as an example, referring to... Figure 1b The software structure of an electronic device is illustrated by example.

[0084] A layered architecture divides the software within an electronic device into several layers, each with a clear role and function. Layers can communicate with each other through software interfaces. For example... Figure 1b As shown, the software architecture of an electronic device, from top to bottom, includes at least the following: application layer, system service layer, and kernel layer. Additionally, an electronic device may include a hardware abstraction layer (HAL), which provides hardware resource support for the electronic device.

[0085] The application layer is the top layer of the operating system, including native operating system applications and installed third-party applications, such as camera, settings, skin modules, user interface (UI), and other third-party applications. Third-party applications can include gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, SMS, and a multitasking center. The multitasking center can be a system application for the electronic device, used to display information about the applications currently running on the device.

[0086] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer can include some predefined functions. For example... Figure 1bAs shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, and capability manager service (AMS).

[0087] The window manager is used to manage windowed applications. It can obtain the screen size, determine if a status bar is present, lock the screen, and capture screenshots. The content provider stores and retrieves data, making this data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0088] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0089] A phone manager is used to provide communication functions for electronic devices. For example, it manages call status (including connection and disconnection).

[0090] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0091] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0092] In this embodiment of the application, AMS can be used to manage tasks running in the application, such as obtaining application information from the task stack currently running in the application and sending the obtained application information to the multi-task center so that the multi-task center can display the page corresponding to the application based on the application information.

[0093] The runtime includes the core libraries and the virtual machine. The runtime is responsible for the scheduling and management of the operating system.

[0094] The core library consists of two parts: one part contains the functionalities that the Java language needs to call, and the other part contains the core libraries of the operating system. The application layer and application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0095] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), image processing libraries, etc.

[0096] The Surface Manager is used to manage the display subsystem and provides fusion of two-dimensional (2D) and three-dimensional (3D) layers for multiple applications.

[0097] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0098] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0099] A 2D graphics engine is a graphics engine for 2D drawing.

[0100] The kernel layer provides core system services for the operating system, such as security, memory management, process management, network protocol stacks, and driver models. It also serves as an abstraction layer between the hardware and software stacks. This layer contains many drivers related to electronic devices, including at least display drivers, camera drivers, audio drivers, and sensor drivers.

[0101] The hardware layer is the interface layer between the operating system kernel and the hardware circuitry of an electronic device, and its purpose is to abstract the hardware. For example, the hardware layer can include various sensors, such as accelerometers, gyroscopes, and touch sensors.

[0102] It is important to understand that the functional services described above are just an example. In practical applications, electronic devices may be divided into more or fewer functional services based on other factors, or the functions of each service may be divided in other ways, or they may not be divided into functional services but work as a whole.

[0103] With the rapid development of electronic devices (such as smartphones), the applications installed on these devices are becoming increasingly diverse. Take photo gallery apps as an example: these apps can store user media data collected by cameras or webcams. Among this user media data, portraits account for nearly half or even more. In this context, users' demands for facial recognition functionality are growing. Facial recognition technology can bring users a more intelligent and convenient photo album creation experience and has gradually become a basic feature of electronic devices.

[0104] However, current on-device facial recognition solutions are purely visual, primarily performing facial detection, image feature extraction, and clustering on user media data to create a photo album. However, this approach simply groups people with similar facial features together, and the extracted features are only single-dimensional visual features, thus failing to identify who is in the photo album. Furthermore, even if the resulting photo album contains rich data, it can only be used within the photo library application and cannot be correlated with external information outside of the application.

[0105] In view of this, this application provides a feature association method, which extracts multiple modal features in the same time and space from a first file, and then establishes the association relationship between the multiple modal features to form a feature connector. It is understood that multiple modal features in the same time and space are highly correlated. Through the feature connector, the information corresponding to multiple modal features in different applications can be accurately connected, facilitating effective information sharing (or information linkage) between different applications. That is, it establishes information (or data) connections between different applications, which helps to lay a foundation for better data understanding and data mining in the future, and can bring users a more intelligent experience.

[0106] It is understandable that multimodal data, such as video data or spatial files, is rich in multimodal information (e.g., visual modality image information, audio modality sound information, etc.). By performing different modal recognition (or different feature extraction) on data within the same time and space (or the same time and spatial dimension) of multimodal data, different modal features can be obtained. It should be understood that because different modal features within the same time and space are highly correlated, the electronic device 100 can associate different modal features within the same time and space to form a feature connector, that is, to complete the association of cross-modal features. For specific implementation details, please refer to [reference needed]. Figure 2a .in, Figure 2a The feature association process shown is illustrated using electronic device 100 as the execution subject as an example. Figure 2aThe illustrated feature association process includes the following steps: S1, the electronic device 100 acquires multimodal data. S2, the electronic device 100 performs cross-modal recognition on the multimodal data to obtain modal data of at least one speaking object in the same time and space. S3, for each speaking object, the electronic device 100 extracts different features from the modal data corresponding to that speaking object to obtain different modal features of that speaking object. S4, the electronic device 100 associates the different modal features of the speaking object to obtain feature connectors. S5, the electronic device 100 stores the feature connectors corresponding to the speaking object in a vector database.

[0107] For example, taking the installation of a gallery application, a call application, and a recorder application (or other applications with audio acquisition functions, such as applications with recording functions) on the aforementioned electronic device 100 as an example, this paper introduces the relevant implementation of associating data from different modalities.

[0108] Please refer to Figure 3 , Figure 3 This is a schematic diagram illustrating the association between facial features and voice features, provided as an embodiment of this application. Figure 3As shown, the electronic device 100 obtains a first video from a gallery application. The first video includes image information and audio information. It can be understood that the image and audio information of the same speaking object (e.g., the same speaker) are recorded in the same time and space dimensions. The image and audio information of the same speaking object can be feature extracted separately to obtain visual modal portrait features and audio modal sound features. Then, the electronic device 100 can perform cross-modal recognition on the first video to obtain video segments of at least one speaking object. The specific implementation can be found in the determination process of the first sub-file below, and will not be elaborated here. Each video segment of at least one speaking object includes the image and audio information of that speaking object. After obtaining the video segments of at least one speaking object, the electronic device 100 can extract portrait features and sound features from each video segment of at least one speaking object, obtaining portrait features (or visual modal portrait features) and sound features (or audio modal sound features) of each speaking object. In one possible implementation, for each video segment of a speaking subject, the electronic device 100 can obtain the image information and audio information of the speaking subject from the video segment. Then, the electronic device 100 performs facial recognition (or facial feature extraction) on the image information of the speaking subject to obtain the facial features of the speaking subject. The electronic device 100 also performs sound feature recognition (or sound feature extraction) on the audio information of the speaking subject to obtain the sound features of the speaking subject. Alternatively, the electronic device 100 can also perform facial feature extraction on the image information included in the video segment of the speaking subject to obtain the facial features of the speaking subject, and can also perform sound feature extraction on the audio information included in the video segment of the speaking subject to obtain the sound features of the speaking subject.

[0109] Since the facial and vocal features of the same speaker are extracted simultaneously in the same time and space dimensions, and these features are highly correlated (or strongly correlated), the electronic device 100 can associate them to form a feature connector. This feature connector describes (or records) the relationship between facial and vocal features in the same time and space dimensions. Alternatively, it can be understood as connecting (associating) two previously unrelated modal features (here referring to two previously unrelated facial and vocal features) or modal information (here referring to two previously unrelated image and audio information) together. The electronic device 100 can then store one or more of the obtained feature connectors (which can be understood as feature connectors corresponding to one or more speaking subjects) in a vector database. Optionally, in this application, the feature connector can be described as a feature association relationship, feature mapping relationship, feature correspondence relationship, feature correlator, or cross-modal feature connector.

[0110] For example, when a feature connector is used to describe the relationship between two features, the feature connector can be represented as: Feature Connector Name, Feature 1, Feature 2. When a feature connector is used to describe the relationship between three features, the feature connector can be represented as: Feature Connector Name, Feature 1, Feature 2, Feature 3. When a feature connector is used to describe the relationship between four features, the feature connector can be represented as: Feature Connector Name, Feature 1, Feature 2, Feature 3, Feature 4. It can be understood that feature connectors can also be used to describe the relationship between more than four features; the representations of feature connectors will not be listed here.

[0111] For example, a feature connector is used to describe the relationship between facial features and voice features. This feature connector can be represented as: a person feature connector, facial features, and voice features.

[0112] Optionally, after storing one or more feature connectors in a vector database, the electronic device 100 can perform cross-modal feature matching based on the one or more feature connectors stored in the vector database, thereby enabling data communication between different applications. For specific implementation details, please refer to [reference needed]. Figure 2b .in, Figure 2b The feature matching process shown is illustrated using electronic device 100 as the execution subject as an example. Figure 2b The intended feature matching process includes the following steps:

[0113] S1', Electronic device 100 acquires modal data.

[0114] For example, modal data can be unimodal or multimodal. Unimodal data can refer to data within a single modality or dimension, such as visual data (data within the visual modality or visual dimension, like image data), audio data (audio data within the audio modality or audio dimension), or text data (data within the text modality or text dimension). Multimodal data can refer to data across multiple modalities or dimensions, such as video data (video data includes multi-dimensional data, such as image and audio data), data containing both speech and text, or data containing both images and text.

[0115] S2', Electronic device 100 extracts features from modal data to obtain fourth modal features.

[0116] S3', the electronic device 100 matches the fourth modal feature with at least one feature connector in the vector database to determine a modal feature associated with the fourth modal feature.

[0117] It should be understood that the modal feature associated with the fourth modal feature is another modal feature indicated in the feature connector that matches the fourth modal feature. Specifically, the modal feature indicated by this feature connector matches the fourth modal feature.

[0118] S4', the electronic device 100 obtains the information corresponding to a modal feature associated with the fourth modal feature.

[0119] S5', the electronic device 100 associates the information corresponding to the fourth modality feature with the information corresponding to the current modality feature. In this way, the electronic device 100 can establish an association between the application containing the information corresponding to the fourth modality feature and the application containing the information corresponding to the current modality feature. That is, it establishes an association in different structured data, thereby enabling data communication between these two different applications, allowing them to interact and share each other's information.

[0120] For example, consider a new image in a photo gallery application. Electronic device 100 can extract facial features from this new image to obtain a facial feature, such as facial feature 1. Then, electronic device 100 can query a vector database for voice features that match facial feature 1. For instance, electronic device 100 can match facial feature 1 with one or more feature connectors included in the vector database to determine the feature connector corresponding to facial feature 1. Furthermore, it can obtain the voice feature corresponding to facial feature 1 from the feature connector, and can also obtain the speaker information (or contact information, such as name, identification identifier (e.g., grandfather, grandmother, father, mother, son, daughter, aunt, etc., or other forms of identification information) corresponding to that voice feature from a call application. Alternatively, electronic device 100 can also query the vector database to find which feature connector describes facial feature 1. When a feature connector in the vector database describes a portrait feature as portrait feature 1, the electronic device 100 can obtain the corresponding voice feature based on the feature connector, and can also obtain the speaker information corresponding to the voice feature from the call application based on the voice feature. Then, the electronic device 100 can display a notification message (or prompt message, or alert message) u1 in the gallery application. The notification message u1 prompts the user to set contact information for the new picture or the portrait album containing the new picture in the gallery application, that is, to identify who the new picture or the portrait album containing the new picture belongs to or which contact. This can be understood, or the example can be taken with other pictures in the gallery application (such as existing pictures). The specific implementation can refer to the feature matching process of the new picture described above, which will not be repeated here.

[0121] Optionally, the electronic device 100 can also display notification message u2 in the calling application. Notification message u2 can prompt the user to set a profile picture or an image containing a profile picture for that contact in the calling application (e.g., selecting a new image from the gallery application or choosing an image from the portrait album containing that new image in the gallery application). Alternatively, it can prompt the user to set a profile picture or an image containing a profile picture for that contact as the call cover for that contact (e.g., selecting a new image from the gallery application or choosing an image from the portrait album containing that new image in the gallery application). This establishes data communication between the gallery application and the calling application, providing a better user experience. For example, notification message u1 or notification message u2 can be presented as a pop-up window, an interface, or other forms; this application does not limit the presentation of these forms. It is understandable that when displaying notification message u2, electronic device 100 can first retrieve a headshot (or an image containing a headshot) corresponding to portrait feature 1 from the gallery application. Then, electronic device 100 can describe in notification message u2, "A portrait image of contact XXX has been found in the gallery application. Please confirm whether to set a headshot for contact XXX." Optionally, in response to the user's first operation, electronic device 100 can set a headshot for contact XXX in the call application. Here, the first operation is used to instruct the user to click "Yes." Optionally, electronic device 100 can also describe in notification message u2, "Multiple portrait images of contact XXX have been found in the gallery application. Please select one portrait image from these multiple portrait images to set a headshot for contact XXX." In response to the user's second operation, electronic device 100 can set a first portrait image as a headshot for contact XXX in the call application. Here, the second operation is used to instruct the user to select the first portrait image.

[0122] For example, consider new audio data from a call application. Electronic device 100 can segment and cluster this new audio data to obtain at least one clustering information. Each clustering information can describe the temporal information of a speaker in the new audio data, or it can indicate data (e.g., sub-audio data) corresponding to a speaker in the new audio data. Then, electronic device 100 can segment and cut the new audio data according to the at least one clustering information to obtain at least one audio segment. Next, electronic device 100 can perform feature extraction (e.g., voice feature extraction) on the at least one audio segment to obtain at least one feature (e.g., voice feature). Taking the electronic device 100 obtaining at least one voice feature as an example, for each of the at least one voice features, electronic device 100 can match the voice feature with one or more feature connectors included in a vector database to determine the feature connector corresponding to the voice feature. Furthermore, it can obtain the corresponding portrait feature from the feature connector corresponding to the voice feature, and can also obtain the corresponding portrait information (e.g., a headshot or an image containing a headshot) from a gallery application based on the portrait feature. Alternatively, electronic device 100 can query the vector database to find which feature connector describes the aforementioned voice feature. When a feature connector in the vector database describes the aforementioned voice feature, electronic device 100 can obtain the corresponding facial features based on that feature connector, and can also obtain the corresponding facial information from the gallery application based on that facial feature. Furthermore, electronic device 100 can display notification message u3 in the call application. Notification message u3 can prompt the user to set a profile picture for that contact in the call application, or it can prompt the user to set a profile picture for that contact as a call cover image for that contact. This enables data communication between the gallery application and the call application, and provides a better user experience. For examples of notification message u3 not described in detail, please refer to the relevant introduction of notification message u2 above; it will not be repeated here.

[0123] The specific implementation of the feature association method in the embodiments of this application will be described in detail below with reference to the accompanying drawings. It is understood that this application uses an electronic device as the executor of the feature association method as an example, but this application does not limit the executor shown. For example, the feature association method executed by the electronic device in this application can also be executed by a module applied to the electronic device (e.g., a chip, chip system, processor, application program, system service, or service process), or by a logical node, logical module, or software capable of implementing all or part of the functions of the electronic device.

[0124] Figure 4 This illustration shows a flowchart of a feature association method provided in an embodiment of this application. The method is applicable to electronic devices, such as... Figure 1a or Figure 1b The illustrated electronic device 100. (e.g.) Figure 4 As shown, the process of this method includes:

[0125] S401: The electronic device extracts features from the first sub-file of the first file targeting the first speaking object, and obtains the first modal features and the second modal features of the first speaking object.

[0126] Optionally, the first speaking object can be replaced with the description "first speaker" or "first speaker".

[0127] In this design, the first modality feature (or first feature) and the second modality feature (or second feature) are not the same. For example, the first modality feature and the second modality feature can be any two of the following features: face feature, voice feature, semantic features, and behavioral features. For example, the first modality feature is a face feature, and the second modality feature is a voice feature. Optionally, the first modality feature and the second modality feature can also be any two features other than those mentioned above. Alternatively, the first modality feature can be one of the features mentioned above, and the second modality feature can be one of other features (such as clothing features or behavioral habits). Or, the second modality feature can be one of the features mentioned above, and the first modality feature can be one of other features.

[0128] In this embodiment, the first file is rich in multimodal information (such as image information of the visual modality, sound information of the audio modality, semantic information of the audio modality, etc.). For example, the first file may refer to a video file, a spatial file, or other files rich in multimodal features.

[0129] For example, when the first file is a video file, the first sub-file can be the first sub-video file. As another example, when the first file is a spatial file, the first sub-file can be the first sub-spatial file.

[0130] In one possible example, the first file may include both an image file and an audio file. This way, the electronic device can extract the audio file from the first file when it needs to use the audio file, or extract the image file from the first file when it needs to use the image file.

[0131] The following example, which includes both image and audio files, illustrates the process of determining the first sub-file.

[0132] In one possible implementation, the electronic device acquires M sub-audio files containing sound information from an audio file. Here, M is an integer greater than or equal to 1, and the M sub-audio files correspond to a first speaking object. Next, the electronic device can acquire M sub-image files from an image file that correspond to the M sub-audio files. Then, the electronic device can acquire N sub-image files from the M sub-image files. These N sub-image files contain facial image information (which can be understood as the facial image information of the first speaking object), and the facial image's lips show movement. At this point, the electronic device can use the N sub-image files and the N sub-audio files corresponding to the N sub-image files from the M sub-audio files as a first sub-file. Here, N is an integer greater than or equal to 1 and less than or equal to M.

[0133] It is understood that the first file may also include one or more sub-files similar to the first sub-file. These one or more sub-files similar to the first sub-file include other speaking objects. For example, the first file may also include a second sub-file for a second speaking object, or the first file may also include a third sub-file for a third speaking object, and so on. When the first file includes one or more sub-files similar to the first sub-file, the process by which the electronic device determines the one or more sub-files similar to the first sub-file can refer to the process for determining the first sub-file, and will not be elaborated here.

[0134] The following examples illustrate the specific implementation of an electronic device acquiring M sub-audio files containing sound information from an audio file.

[0135] Example a1: An electronic device performs voice recognition on an audio file to determine whether a human voice exists in the audio file, or to determine which segment of the audio file contains a human voice (or voice information). When the electronic device determines that a human voice exists in the audio file, it can extract (extract) the audio segment containing the human voice. Then, the electronic device can segment and cluster the audio segment to obtain at least one clustering information. Each clustering information can describe the time information of a speaker in the audio segment, or each clustering information can indicate a sub-audio file (or sub-audio segment) corresponding to a speaker in the audio segment. Then, the electronic device can obtain M sub-audio files corresponding to the first speaker from the audio segment based on the clustering information corresponding to the first speaker in the at least one clustering information. For example, the electronic device can obtain M sub-audio files corresponding to the first speaker from the audio segment based on the time information (such as speaking duration or speaking time range) of the first speaker in the audio segment. Optionally, the electronic device can also obtain one or more sub-audio files corresponding to each speaker in at least one segment from the audio segment based on at least one clustering information. In this process, at least one of the speaking objects includes the first speaking object. It should be understood that if there is no voice in the audio file, the electronic device may halt subsequent processing of the first file.

[0136] Optionally, the electronic device may use an audio voice recognition model, such as an audio voice recognition artificial intelligence (AI) model, or other models capable of recognizing human voices, to perform voice recognition on audio files. Similarly, the electronic device may use a speaker diarization model, or other models capable of voice segmentation, to segment and cluster audio segments.

[0137] The example a1 above enables electronic devices to more promptly and accurately identify one or more sub-audio files corresponding to each speaking object, which helps to speed up the establishment of feature connectors.

[0138] Example a2: An electronic device segments and clusters an audio file to obtain at least one clustering information. Each clustering information can describe the time information of a speaker in the audio file, or it can indicate a sub-audio file corresponding to a speaker in the audio file. Then, the electronic device can obtain M sub-audio files corresponding to a first speaker from the audio file based on the clustering information corresponding to the first speaker in the at least one clustering information. For example, the electronic device can obtain the M sub-audio files corresponding to the first speaker from the audio file based on the time information (such as speaking duration or speaking time range) of the first speaker in the audio file. Optionally, the electronic device can also obtain one or more sub-audio files corresponding to each speaker in at least one speaker from the audio file based on at least one clustering information. The at least one speaker includes the first speaker.

[0139] In one possible implementation, when the electronic device acquires M sub-image files corresponding to M sub-audio files in an image file, it may include the following steps: The electronic device selects an image portion (or image segment) within the time range corresponding to each of the M sub-audio files from the image file as the sub-image file corresponding to that sub-audio file, based on the time range corresponding to each of the M sub-audio files. For example, taking a video file as the first file, the video file includes audio files and image files. The electronic device can select one or more video frames (or video image segments) within the time range corresponding to each of the M sub-audio files from the image file as the sub-image files corresponding to that sub-audio file, thereby obtaining the M sub-image files corresponding to the M sub-audio files.

[0140] In one possible implementation, when an electronic device acquires N sub-image files from M sub-image files, it may include the following steps: For each sub-image file among the M sub-image files, the electronic device performs face detection (or person detection) on that sub-image file to determine whether it contains human image information. For example, the electronic device can use a person detection model (or a human image recognition model or human image detection model, such as a face detection model or a face detection AI model) to perform face detection on the sub-image file to determine whether it contains human image information. When the sub-image file contains human image information, the electronic device can perform lip movement detection on the sub-image file to determine whether the lips of the person in the sub-image file have changed movement. If the lips of the person in the sub-image file have changed movement, the electronic device can treat that sub-image file as one of the N sub-image files.

[0141] For example, taking the first file as a video file and the sub-image files as video frames, the electronic device can use a face detection model to perform face detection on each of the M video frames, determining whether each sub-video frame contains human image information. Then, the electronic device can use a person tracking model to track and lock onto consecutive video frames where human image information is detected. Next, the electronic device can use a lip movement detection model to detect the lip movements of the locked person, determining whether there are any changes in lip movement, thus locking onto the speaker. In this way, the electronic device can obtain at least one video frame from the M video frames that contains the speaker and where the speaker's lip movements have changed, and it can also obtain the corresponding audio segment (or audio data) for that at least one video frame, for example... Figure 5 The diagram illustrates a pairing of a video frame (which can be understood as a lip-movement video frame) and an audio clip. It can be understood that in... Figure 5 In this process, each lip movement video frame corresponds to an audio segment. For example, the audio segment can be in WAV format or other formats. Thus, the electronic device can decompose (or divide) the video file into multiple sub-video files. Each sub-video file can include at least one sub-video frame corresponding to a speaking subject and an audio segment corresponding to that at least one video frame. This can be understood as each sub-video file including the speaker's facial and audio information. Since the speaker's facial and audio information reside in the same time-space, they can be considered strongly correlated. Therefore, the electronic device can extract the speaker's facial and vocal features from the sub-video file and correlate them to form a feature connector. This feature connector can then be used to establish information connections between different applications, thus providing a foundation for better data understanding and data mining in the future.

[0142] It is understandable that existing feature extraction schemes are all aimed at extracting unimodal features, such as facial feature extraction in face recognition schemes and voice feature extraction in voice recognition schemes. They lack correlation between features of different modalities, such as the lack of correlation between voice features and facial features in the original data. However, this application targets cross-modal feature extraction, that is, extracting features of different modalities that exist in the same time and space, such as… Figure 6 The relationship between different features that are in the same time and space is shown. Figure 6This explanation uses features 1 (e.g., facial features), 2 (e.g., voice features), and 3 (e.g., semantic features) as examples. Features 1 and 2 exist in the same time space, features 1 and 3 exist in the same time space, and features 2 and 3 exist in the same time space. Feature connectors are formed by associating different modal features within the same time space and storing these connectors in a vector database. This allows for the association of previously isolated (or unrelated) pieces of information. For example, a facial and voice feature connector can directly link facial information in a photo gallery application with contact information in a calling application, thus establishing a data connection between the two applications.

[0143] Optionally, when modal feature similarity in one dimension (such as the portrait dimension) can be matched, the data corresponding to the modal features in that dimension can be extended to other dimensions (such as the voice dimension or audio dimension). For example, taking a portrait-voice feature connector as an example, when the portrait features of a certain portrait image in a gallery application match the portrait features in the portrait-voice feature connector, the electronic device can obtain the contact information corresponding to the voice features in the calling application based on the voice features corresponding to the portrait features. Then, the electronic device can associate the portrait image in the gallery application with the contact information in the calling application. For example, the electronic device can recommend setting contact information for the portrait image to the user in the gallery application, or it can recommend setting a portrait image for the contact to the user in the calling application.

[0144] The following examples illustrate the specific implementation of feature extraction from the first sub-file by an electronic device to obtain the first modal features and the second modal features of the first speaking object.

[0145] Example b1: The first modality feature is the human face feature, and the second modality feature is the voice feature.

[0146] In one implementation, the electronic device can use a facial recognition model to extract facial features from the first sub-file to obtain the facial features of the first speaker. For example, the electronic device can use a facial recognition model to extract facial features from the sub-image files, image data, or image information included in the first sub-file to obtain the facial features of the first speaker. Similarly, the electronic device can use a sound recognition model to extract sound features from the first sub-file to obtain the sound features of the first speaker. For example, the electronic device can use a sound recognition model to extract sound features from the sub-audio files, audio data, or audio information included in the first sub-file to obtain the sound features of the first speaker.

[0147] It is understood that the facial recognition model can be an existing pre-trained model (such as a facial recognition model already deployed on the device), or it can be obtained by training (or iteratively training) an initial neural network on an electronic device based on a first training sample set until the initial neural network converges or meets the training requirement. The first training sample set can include multiple facial image samples with pre-labeled facial features.

[0148] The voice recognition model can be an existing, pre-trained model, or it can be obtained by an electronic device training (or iteratively training) an initial neural network based on a second training sample set until the initial neural network converges or meets the training requirement. The second training sample set can include multiple audio samples with pre-labeled voice features.

[0149] Optionally, in example b1, the electronic device can also use a behavior recognition model to extract behavior features from the first sub-file to obtain the behavior features of the first speaking object. Alternatively, the electronic device can also use a semantic recognition model to extract semantic features from the first sub-file to obtain the semantic features of the first speaking object. In this way, the electronic device can also form a feature connector based on facial features, voice features, and behavior features, which describes the relationship between facial features, voice features, and behavior features. Alternatively, the electronic device can also form a feature connector based on facial features, voice features, and semantic features, which describes the relationship between facial features, voice features, behavior features, and semantic features. Alternatively, the electronic device can also form a feature connector based on facial features, voice features, behavior features, and semantic features, which describes the relationship between facial features, voice features, behavior features, and semantic features.

[0150] Example b2: The first modality is the facial features, and the second modality is the behavioral features.

[0151] In one implementation, the electronic device can use a facial recognition model to extract facial features from the first sub-file to obtain the facial features of the first speaking object. Additionally, the electronic device can use a behavior recognition model to extract behavior features from the first sub-file to obtain the behavior features of the first speaking object. For example, the electronic device can use a behavior recognition model to extract behavior features from the sub-image files, image data, or image information included in the first sub-file to obtain the behavior features of the first speaking object.

[0152] It is understood that the behavior recognition model can be an existing pre-trained model, or it can be obtained by training an initial neural network on an electronic device based on a third training sample set until the initial neural network converges or meets the training requirements. The third training sample set can include multiple behavior image samples pre-labeled with behavioral features.

[0153] Optionally, in example b2, the electronic device can also use a voice recognition model to extract voice features from the first sub-file to obtain the voice features of the first speaking object. Alternatively, the electronic device can also use a semantic recognition model to extract semantic features from the first sub-file to obtain the semantic features of the first speaking object. In this way, the electronic device can also form a feature connector based on facial features, behavioral features, and voice features, which describes the relationship between facial features, behavioral features, and voice features. Alternatively, the electronic device can also form a feature connector based on facial features, behavioral features, and semantic features, which describes the relationship between facial features, behavioral features, voice features, and semantic features. Alternatively, the electronic device can also form a feature connector based on facial features, behavioral features, voice features, and semantic features, which describes the relationship between facial features, behavioral features, voice features, and semantic features.

[0154] Example b3: The first modality feature is the human image feature, and the second modality feature is the semantic feature.

[0155] In one implementation, the electronic device can use a facial recognition model to extract facial features from the first sub-file to obtain the facial features of the first speaking object. Additionally, the electronic device can use a semantic recognition model to extract semantic features from the first sub-file to obtain the semantic features of the first speaking object. For example, the electronic device can use a semantic recognition model to extract semantic features from the sub-audio files, audio data, or audio information included in the first sub-file to obtain the semantic features of the first speaking object.

[0156] It is understood that the semantic recognition model can be an existing pre-trained model, or it can be obtained by training an initial neural network on an electronic device based on a fourth training sample set until the initial neural network converges or meets the training requirement. The fourth training sample set can include multiple audio samples with pre-annotated semantic features.

[0157] Optionally, in example b3, the electronic device can also use a voice recognition model to extract voice features from the first sub-file to obtain the voice features of the first speaking object. Alternatively, the electronic device can also use a behavior recognition model to extract behavior features from the first sub-file to obtain the behavior features of the first speaking object. In this way, the electronic device can also form a feature connector based on facial features, semantic features, and voice features, which is used to describe the relationship between facial features, semantic features, and voice features. Alternatively, the electronic device can also form a feature connector based on facial features, semantic features, and behavior features, which is used to describe the relationship between facial features, semantic features, voice features, and behavior features. Alternatively, the electronic device can also form a feature connector based on facial features, semantic features, voice features, and behavior features, which is used to describe the relationship between facial features, semantic features, voice features, and behavior features.

[0158] Example b4: The first modality feature is the sound feature, and the second modality feature is the semantic feature.

[0159] In one implementation, the electronic device can use a voice recognition model to extract voice features from the first sub-file to obtain the voice features of the first speaking object. Additionally, the electronic device can use a semantic recognition model to extract semantic features from the first sub-file to obtain the semantic features of the first speaking object.

[0160] Optionally, in example b4, the electronic device can also use a facial recognition model to extract facial features from the first sub-file to obtain the facial features of the first speaking object. Alternatively, the electronic device can also use a behavior recognition model to extract behavior features from the first sub-file to obtain the behavior features of the first speaking object. In this way, the electronic device can also form a feature connector based on voice features, semantic features, and facial features, which describes the relationship between voice features, semantic features, and facial features. Alternatively, the electronic device can also form a feature connector based on voice features, semantic features, and behavior features, which describes the relationship between voice features, semantic features, and behavior features. Alternatively, the electronic device can also form a feature connector based on voice features, semantic features, facial features, and behavior features, which describes the relationship between voice features, semantic features, facial features, and behavior features.

[0161] Example b5: The first modality feature is the voice feature, and the second modality feature is the behavior feature.

[0162] In one implementation, the electronic device can use a voice recognition model to extract voice features from the first sub-file to obtain the voice features of the first speaker. Additionally, the electronic device can use a behavior recognition model to extract behavior features from the first sub-file to obtain the behavior features of the first speaker.

[0163] Optionally, in example b5, the electronic device can also use a facial recognition model to extract facial features from the first sub-file to obtain the facial features of the first speaking object. Alternatively, the electronic device can also use a semantic recognition model to extract semantic features from the first sub-file to obtain the semantic features of the first speaking object. In this way, the electronic device can also form a feature connector based on voice features, behavioral features, and facial features, which describes the relationship between voice features, behavioral features, and facial features. Alternatively, the electronic device can also form a feature connector based on voice features, behavioral features, and semantic features, which describes the relationship between voice features, behavioral features, and semantic features. Alternatively, the electronic device can also form a feature connector based on voice features, behavioral features, semantic features, and facial features, which describes the relationship between voice features, behavioral features, semantic features, and facial features.

[0164] Example b6: The first modality feature is a semantic feature, and the second modality feature is a behavioral feature.

[0165] In one implementation, the electronic device can use a semantic recognition model to extract semantic features from the first sub-file to obtain the semantic features of the first speaking object. Additionally, the electronic device can use a behavior recognition model to extract behavior features from the first sub-file to obtain the behavior features of the first speaking object.

[0166] Optionally, in example b6, the electronic device can also use a facial recognition model to extract facial features from the first sub-file to obtain the facial features of the first speaking object. Alternatively, the electronic device can also use a voice recognition model to extract voice features from the first sub-file to obtain the voice features of the first speaking object. In this way, the electronic device can also form a feature connector based on semantic features, behavioral features, and facial features, which describes the relationship between semantic features, behavioral features, and facial features. Alternatively, the electronic device can also form a feature connector based on semantic features, behavioral features, and voice features, which describes the relationship between semantic features, behavioral features, and voice features. Alternatively, the electronic device can also form a feature connector based on semantic features, behavioral features, voice features, and facial features, which describes the relationship between semantic features, behavioral features, voice features, and facial features.

[0167] S402: The electronic device determines the first feature connector based on the first modal feature and the second modal feature.

[0168] The first feature connector can be used to describe (or represent) the association between the first modal feature and the second modal feature.

[0169] It is understood that the first feature connector can also be used to connect (or link) information corresponding to the first modal feature and information corresponding to the second modal feature. Here, the information corresponding to the first modal feature and the information corresponding to the second modal feature reside in different applications of the electronic device. That is to say, the first feature connector can be used by the electronic device to achieve linkage between the information corresponding to the first modal feature and the information corresponding to the second modal feature, enabling different applications (such as the application containing the information corresponding to the first modal feature and the application containing the information corresponding to the second modal feature) to effectively share each other's data, greatly enriching the connectivity between data from different applications.

[0170] In this embodiment of the application, after obtaining the first feature connector, the electronic device can save (or record) the first feature connector. For example, the electronic device can store the first feature connector in... Figure 7a The vector database shown is used to store one or more feature connectors.

[0171] Optionally, if the electronic device also performs other feature extraction on the first sub-file to obtain other modal features, the electronic device can also determine the second feature connector based on the first modal feature, the second modal feature, and other modal features. The second feature connector can be used to describe the association between the first modal feature, the second modal feature, and other modal features. It can be understood that the second feature connector can also be used to connect information corresponding to the first modal feature, information corresponding to the second modal feature, and information corresponding to other modal features. After obtaining the second feature connector, the electronic device can also save it. For example, the electronic device can store the second feature connector in... Figure 7a The vector database shown.

[0172] Optionally, after storing one or more feature connectors in a vector database, the electronic device can perform cross-modal feature matching based on one or more feature connectors stored in the vector database, thereby realizing the association between information corresponding to different modal features, that is, establishing the association between different structured data.

[0173] For example, let's take the first speaking object as an example to illustrate the specific implementation of cross-modal feature matching. In one possible implementation, the electronic device obtains first information corresponding to the first speaking object from a first application. After obtaining the first information, the electronic device can perform feature extraction on the first information to obtain third modal features. Then, the electronic device can match the third modal features with at least one feature connector in a vector database to determine which feature connector indicates which modal feature the third modal feature matches. If the third modal feature matches the first modal feature indicated by the first feature connector, the electronic device can obtain second information corresponding to the second modal feature from a second application. Then, the electronic device can display a prompt message in the first application. This prompt message prompts the user to set the second information in the first application. In other words, the prompt message can prompt the user to set the second information for the first information corresponding to the first speaking object in the first application. For example, if the first application is a gallery application and the second application is a calling application, the prompt message can prompt the user to set contact information for the portrait image corresponding to the first speaking object in the gallery application.

[0174] The following examples illustrate the specific implementation of electronic devices extracting features from first information to obtain third modality features.

[0175] Example c1: The first application is an application with audio acquisition function, and the first information is audio data.

[0176] For example, an application with audio capture capabilities could be a call application, a voice recorder application, or another application with audio capture capabilities.

[0177] In one implementation, the electronic device segments and clusters the audio data to obtain K cluster information. Each of the K cluster information indicates data in the audio data corresponding to a speaker, or each of the K cluster information can describe the temporal information of a speaker in the audio data. Here, K is an integer greater than or equal to 1. Then, the electronic device can segment (or cut into segments) the audio data according to the K cluster information to obtain K audio segments. Next, the electronic device can extract features from the K audio segments to obtain K modal features. The K modal features include a third modal feature.

[0178] For example, an electronic device can extract sound features from K audio segments, resulting in K sound features. Among these, the K modal features are the K sound features, and the third modal feature is one of the K sound features. Alternatively, the electronic device can also extract semantic features from K audio segments, resulting in K semantic features. Again, the K modal features are the K semantic features, and the third modal feature is one of the K semantic features.

[0179] For example, taking a first application as a calling application, a second application as a gallery application, and a third modal feature as a voice feature as an example, the electronic device obtains audio data 1 from the calling application. For example, the format of audio data 1 can be seen in [reference needed]. Figure 8 .exist Figure 8 In the audio data, there are audio segments from multiple speakers, such as audio segments from speaker 1, speaker 2, and speaker 3. After acquiring audio data 1, the electronic device can use a speaker segmentation clustering model to segment and cluster the audio data 1, obtaining at least one clustering information. Each clustering information can describe the temporal information of a speaker in the audio data 1, or it can indicate the voice segment (or audio segment) corresponding to a speaker in the audio data 1. Then, the electronic device can segment and extract audio segments from the audio data 1 based on the at least one clustering information, obtaining at least one audio segment corresponding to a speaker. This facilitates feature extraction based on the characteristics of different speakers' voices. Then, the electronic device can use a voice feature extraction model to extract voice features from each audio segment corresponding to at least one speaker, obtaining the voice features of at least one speaker. This allows for clearer differentiation between the caller's and callee's audio data, as the caller and callee are distributed across different audio channels. Electronic devices extract voice features from the called party's audio data and associate the extracted voice features with the called party's contact information in the call application.

[0180] It's understandable that in a phone call scenario, considering the inherent error range of the voice feature extraction model, after extracting the corresponding voice features, repeated verification through multiple samplings (which can be understood as multi-sample cross-validation) can more accurately identify whether the extracted voice features are accurate. For example, the voice features of the caller P1 (which can be understood as the speaker P1) in a phone call may exhibit unstable feature extraction. Multiple calls may result in variations in environment and timbre. To accommodate these variations, more stable features are selected after multiple samplings, or clustering is used as the voice features. After confirmation through multi-sample engineering, all features are stored in a vector database as multi-dimensional vectors. For example, voice features are stored in the vector database as multi-dimensional vectors, such as... Figure 7b The image shown represents the sound features stored in the vector database. Similarly, the facial features extracted above can be stored in the vector database as 256-dimensional vectors.

[0181] After obtaining the voice features of at least one speaking object, the electronic device performs multi-sample cross-validation on the voice features of each speaking object, and then uses the more stable voice features or clusters them to obtain a voice feature as the final voice feature of that speaking object. Optionally, the electronic device can store the relationship between the final voice feature of each speaking object and the speaking object in a vector database.

[0182] In one possible implementation, for each speaking subject, the electronic device can match the final voice feature of that speaking subject with one or more feature connectors included in the vector database to determine which feature connector indicates which voice feature the final voice feature of the speaking subject matches. When the final voice feature of the speaking subject matches the voice feature indicated by a certain feature connector (e.g., person feature connector 1), the electronic device obtains the portrait feature corresponding to the final voice feature of the speaking subject from person feature connector 1, i.e., the portrait feature indicated by person feature connector 1. Then, the electronic device can obtain the portrait image corresponding to the portrait feature indicated by person feature connector 1 from the image library application.

[0183] In one example, the vector database stores portrait features and image IDs (e.g., image IDs) corresponding to multiple portrait images in a gallery application. The electronic device can query the vector database for the image ID corresponding to the portrait feature indicated by the portrait feature connector 1. Then, the electronic device can retrieve the portrait image corresponding to that image ID from the gallery application based on the image ID corresponding to the portrait feature indicated by the portrait feature connector 1.

[0184] In another example, a gallery application stores multiple portrait images and corresponding portrait features. An electronic device can retrieve the portrait image corresponding to the portrait features indicated by Person Feature Connector 1 from the gallery application.

[0185] It is understandable that electronic devices can extract features from one or more portrait images in a photo gallery application, and can store the features and identifiers of one or more portrait images in a vector database. Alternatively, electronic devices can also store the features and identifiers of one or more portrait images in a photo gallery application.

[0186] In another example, the electronic device can also extract portrait features from multiple portrait images in a gallery application, obtaining the portrait features corresponding to each of the multiple portrait images. Then, the electronic device can match the portrait features indicated by portrait feature connector 1 with the portrait features corresponding to the multiple portrait images, determining which portrait image's portrait features match the features indicated by portrait feature connector 1. When the portrait features of a particular portrait image match the features indicated by portrait feature connector 1, the electronic device can retrieve that portrait image from the gallery application.

[0187] Then, the electronic device can display prompt message 1 in the calling application. Prompt message 1 is used to prompt the user to set a portrait picture for the person being spoken to (e.g., person P2) in the calling application.

[0188] For example, see Figure 9a This is one way to present the prompt message 1. Figure 9a In the first example, message 1 states, "Find the portrait image of the person you are speaking to, P1 (which can be understood as contact P1), in the Gallery app. Please confirm whether you want to set a profile picture for the person you are speaking to, P1." For another example, see... Figure 9b This is another way of presenting prompt message 1. In Figure 9b In the message 1, it says, "Multiple portrait images of the person speaking to P1 have been found in the gallery application. Please select one of these images to set as the avatar for the person speaking to P1." In response to the user's first or second action, the electronic device sets the avatar for the person speaking to P1 in the call application, for example... Figure 9c The image shown is the avatar of the intended speaker, P1. The first operation instructs the user to click "Yes," and the second operation instructs the user to select the first avatar image.

[0189] Example c2: The first application is a gallery application, and the first information is image data.

[0190] In one implementation, the electronic device can use a facial recognition model to extract facial features from image data to obtain the facial features of the first speaker. The third modality feature is the facial feature itself.

[0191] In another implementation, the electronic device can use a behavior recognition model to extract behavioral features from image data to obtain the behavioral features of the first speaking object. The third modality feature is the behavioral feature.

[0192] For example, taking a gallery application as the first application, a calling application as the second application, and a portrait feature as the third modality, the electronic device obtains a portrait image X (e.g., portrait image 1) from the gallery application. After obtaining portrait image 1, the electronic device can use a portrait recognition model to extract portrait features from portrait image 1, obtaining portrait feature 1. Then, the electronic device can match portrait feature 1 with one or more feature connectors included in the vector database to determine which feature connector indicates which portrait feature 1 matches. When portrait feature 1 matches the portrait feature indicated by a certain feature connector (e.g., person feature connector 1), the electronic device obtains the corresponding voice feature from person feature connector 1, i.e., the voice feature indicated by person feature connector 1. Then, the electronic device can obtain the contact information corresponding to this voice feature in the calling application based on this voice feature.

[0193] In one example, the vector database stores the relationships between the voice features of multiple speaking entities in a call application and information about those entities (such as speaking entity identifiers). The electronic device can query the vector database for the speaking entity information corresponding to the voice features indicated by person feature connector 1. Then, based on the speaking entity information corresponding to the facial features indicated by person feature connector 1, the electronic device can retrieve the contact information corresponding to that speaking entity from the call application. It can be understood that the electronic device can extract the voice features corresponding to one or more speaking entities from the audio data in the call application. Subsequently, the electronic device can store the voice features corresponding to one or more speaking entities and one or more speaking entity information in the vector database.

[0194] In another example, the calling application stores voice features and information for multiple contacts. The electronic device can retrieve the contact information corresponding to the voice features indicated by Person Feature Connector 1 from the calling application. It is understood that the electronic device can extract the voice features corresponding to one or more contacts from the audio data in the calling application. Subsequently, the electronic device can also store the voice features corresponding to one or more contacts and the information for one or more contacts in the calling application.

[0195] In another example, the electronic device can also extract voice features corresponding to one or more contacts from audio data in a calling application. The electronic device can then match the voice features indicated by Person Feature Connector 1 with the voice features corresponding to one or more contacts to determine which of the contacts' voice features matches the voice features indicated by Person Feature Connector 1. When the voice feature of a contact matches the voice feature indicated by Person Feature Connector 1, the electronic device can obtain information about that contact from the calling application.

[0196] The electronic device can then display prompt message 2 in the Gallery app. Prompt message 2 prompts the user to set contact information for portrait photo X in the Gallery app.

[0197] For example, see Figure 10a This is one way to present the prompt message 2. Figure 10a In the message, message 2 describes, "Find the contact information corresponding to portrait image X (or the album containing portrait image X) in the call application. Please confirm whether you want to set contact information for portrait image X." For another example, see... Figure 10b This is another way of presenting prompt message 2. Figure 10b The message 2 describes, "In the calling app, we found the contact information for person image X corresponding to XXX. Is this person XXX?" In response to the user's first action, the electronic device sets contact information (such as name, full name, or nickname) for person image X in the gallery app. Figure 10c The image shown is a portrait image 1 with corresponding contact information (e.g., Shuai Shuai). Optionally, the electronic device can also set corresponding contact information for other portrait images, such as... Figure 10c The image shown in picture 2 contains contact information (e.g., the mother). Figure 10c The contact information for the person shown in image 3 (e.g., the father). Figure 10c The contact information for the person in the illustrated image 4 (e.g., the older brother). Figure 10c The image shown in picture 5 contains contact information (e.g., an uncle), etc. This is understandable. Figure 10c Other portrait images shown can be set up in the same way as described above, and will not be elaborated here. The first action is used to instruct the user to click "Yes".

[0198] As can be seen from S401 to S402 above, by extracting the multimodal features of the first sub-file in the first file (the multimodal features of the first sub-file are multimodal features existing in the same time and space), the correlation between multiple modal features is established, forming a feature connector. It can be understood that multiple modal features existing in the same time and space are highly correlated. Thus, the feature connector enables the association of different modal data, allowing the information corresponding to multiple modal features in different applications to be accurately connected. This facilitates effective information sharing (or information linkage) between different applications, that is, it establishes information (or data) connections between different applications, helping to lay a foundation for better data understanding and data mining in the future, and providing users with a more intelligent experience.

[0199] It should be noted that the specific implementation processes provided in the above embodiments are only illustrative examples of the applicable method processes in the embodiments of this application. The execution order of each step can be adjusted according to actual needs, and other steps can be added or some steps can be reduced.

[0200] The methods provided in the embodiments of this application above are described from the perspective of an electronic device as the executing entity. To implement the functions of the methods provided in the embodiments of this application above, the electronic device may include hardware structures and / or software modules, implementing the above functions in the form of hardware structures, software modules, or a combination of hardware structures and software modules. Whether a particular function is executed in the form of hardware structures, software modules, or a combination of hardware structures and software modules depends on the specific application and design constraints of the technical solution.

[0201] Based on the same concept, embodiments of this application also provide an electronic device for implementing the feature association method provided in embodiments of this application. For example... Figure 11 As shown, electronic device 1100 may include one or more processors 1101. Optionally, it may also include memory 1102. Optionally, electronic device 1100 may also include display screen 1103. For example, the above-described devices may be connected (or coupled) via one or more communication buses (such as communication bus 1104).

[0202] The memory 1102 stores one or more computer programs (code), which include computer instructions. One or more processors 1101 call the computer instructions stored in the memory 1102, causing the electronic device 1100 to execute the feature association method provided in the embodiments of this application. For example, the electronic device 1100 can perform the following steps: The electronic device 1100 extracts features from a first sub-file of a first file targeting a first speaking object, obtaining a first modal feature and a second modal feature of the first speaking object. Then, the electronic device 1100 determines a first feature connector based on the first modal feature and the second modal feature. The first feature connector is used to describe the association relationship between the first modal feature and the second modal feature. The first feature connector is also used to connect the information corresponding to the first modal feature and the information corresponding to the second modal feature. The information corresponding to the first modal feature and the information corresponding to the second modal feature are located in different applications of the electronic device 1100. The first modal feature is different from the second modal feature and is any two of facial features, voice features, semantic features, and behavioral features.

[0203] In a specific implementation, memory 1102 may include high-speed random access memory, and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 1102 may store an operating system (hereinafter referred to as the system), such as embedded operating systems like Android, iOS, Windows, or Linux. Memory 1102 can be used to store implementation programs of the embodiments of this application. Memory 1102 may also store network communication programs, which can be used to communicate with one or more additional devices, one or more user devices, or one or more network devices.

[0204] One or more processors 1101 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of programs according to the present application.

[0205] The display screen 1103 is used to display application interfaces (such as the interface of the first application or the interface of the second application) and other related user interfaces.

[0206] It should be noted that, Figure 11 This is merely one implementation of the electronic device 1100 provided in this application embodiment. In practical applications, the electronic device 1100 may include more or fewer components, as detailed in the following references. Figure 1aor Figure 1b The specific structure and description shown are not limited here.

[0207] Based on the same concept, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when run on an electronic device, causes the electronic device to perform the methods provided in the above embodiments.

[0208] Based on the same concept, this application also provides a computer program product, which includes a computer program or instructions that, when run on an electronic device, cause the electronic device to perform the methods provided in the above embodiments.

[0209] Based on the same concept, embodiments of this application also provide a chip located on an electronic device. This chip may include a processor and is coupled to memory. The chip executes program instructions in the memory to cause the electronic device to perform the methods provided in the above embodiments. Here, "coupling" refers to two components being directly or indirectly connected to each other; for example, coupling may refer to an electrical connection between two components. Optionally, the chip may also include the aforementioned memory.

[0210] The methods provided in this application can be implemented, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. A computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs (DVDs), or semiconductor media (e.g., SSDs), etc.

[0211] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A feature association method, characterized in that, Applied to electronic devices, the method includes: Feature extraction is performed on the first sub-file of the first file targeting the first speaking object to obtain the first modal features and the second modal features of the first speaking object; Based on the first modal feature and the second modal feature, a first feature connector is determined. The first feature connector is used to describe the association between the first modal feature and the second modal feature. The first feature connector is also used to connect the information corresponding to the first modal feature and the information corresponding to the second modal feature. The information corresponding to the first modal feature and the information corresponding to the second modal feature are located in different applications of the electronic device. The first modal feature is different from the second modal feature, and is any two of the following: facial features, voice features, semantic features, and behavioral features.

2. The method as described in claim 1, characterized in that, The first file includes audio files and image files; The method further includes: Obtain M sub-audio files containing sound information from the audio file, where M is an integer greater than or equal to 1, and the M sub-audio files correspond to the first speaking object; Obtain the M sub-image files corresponding to the M sub-audio files in the image file; Obtain N sub-image files from the M sub-image files, wherein the N sub-image files contain portrait information and the lips of the portraits show movement changes; The first sub-file includes the N sub-image files and the N sub-audio files in the M sub-audio files that correspond to the N sub-image files.

3. The method as described in claim 1 or 2, characterized in that, The first modal feature is a human facial feature, and the second modal feature is a voice feature; The step of extracting features from a first sub-file of the first file targeting the first speaking object to obtain the first modal features and the second modal features of the first speaking object includes: Using a human face recognition model, human face features are extracted from the first sub-file to obtain the human face features of the first speaking object; Using a voice recognition model, voice features are extracted from the first sub-file to obtain the voice features of the first speaker.

4. The method as described in claim 1 or 2, characterized in that, The first modal feature is a human portrait feature, and the second modal feature is a behavioral feature; The step of extracting features from a first sub-file of the first file targeting the first speaking object to obtain the first modal features and the second modal features of the first speaking object includes: Using a human face recognition model, human face features are extracted from the first sub-file to obtain the human face features of the first speaking object; Using a behavior recognition model, behavior features are extracted from the first sub-file to obtain the behavior features of the first speaking object.

5. The method as described in claim 1 or 2, characterized in that, The first modal feature is a human image feature, and the second modal feature is a semantic feature; The step of extracting features from a first sub-file of the first file targeting the first speaking object to obtain the first modal features and the second modal features of the first speaking object includes: Using a human face recognition model, human face features are extracted from the first sub-file to obtain the human face features of the first speaking object; Using a semantic recognition model, semantic features are extracted from the first sub-file to obtain the semantic features of the first speaking object.

6. The method as described in claim 1 or 2, characterized in that, The first modal feature is an audio feature, and the second modal feature is a semantic feature; The step of extracting features from a first sub-file of the first file targeting the first speaking object to obtain the first modal features and the second modal features of the first speaking object includes: Using a voice recognition model, voice features are extracted from the first sub-file to obtain the voice features of the first speaker. Using a semantic recognition model, semantic features are extracted from the first sub-file to obtain the semantic features of the first speaking object.

7. The method as described in claim 1 or 2, characterized in that, The first modal feature is a vocal feature, and the second modal feature is a behavioral feature; The step of extracting features from a first sub-file of the first file targeting the first speaking object to obtain the first modal features and the second modal features of the first speaking object includes: Using a voice recognition model, voice features are extracted from the first sub-file to obtain the voice features of the first speaker. Using a behavior recognition model, behavior features are extracted from the first sub-file to obtain the behavior features of the first speaking object.

8. The method as described in claim 1 or 2, characterized in that, The first modality feature is a semantic feature, and the second modality feature is a behavioral feature; The step of extracting features from a first sub-file of the first file targeting the first speaking object to obtain the first modal features and the second modal features of the first speaking object includes: Using a semantic recognition model, semantic features are extracted from the first sub-file to obtain the semantic features of the first speaking object; Using a behavior recognition model, behavior features are extracted from the first sub-file to obtain the behavior features of the first speaking object.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: First information corresponding to the first speaking object is obtained from the first application of the electronic device; Feature extraction is performed on the first information to obtain the third modality features; If the third modal feature matches the first modal feature indicated by the first feature connector, second information corresponding to the second modal feature is obtained from the second application of the electronic device; A prompt message is displayed in the first application, which prompts the user to set the second information in the first application.

10. The method as described in claim 9, characterized in that, The first application is an application with audio acquisition function, and the first information is audio data; The step of extracting features from the first information to obtain third modality features includes: The audio data is segmented and clustered to obtain K cluster information. Each of the K cluster information is used to indicate the data in the audio data corresponding to a speaking object, where K is an integer greater than or equal to 1. Based on the K clustering information, the audio data is segmented and cut into K audio segments; Feature extraction is performed on the K audio segments to obtain the K modal features, including the third modal feature.

11. The method as described in claim 10, characterized in that, The step of extracting features from the K audio segments to obtain the K modal features includes: Sound features are extracted from the K audio segments to obtain K sound features, and the K modal features are the K sound features; or, Semantic features are extracted from the K audio segments to obtain K semantic features, and the K modal features are the K semantic features.

12. The method as described in claim 9, characterized in that, The first application is a gallery application, and the first information is image data; The step of extracting features from the first information to obtain third modality features includes: Using a facial recognition model, facial features are extracted from the image data to obtain the facial features of the first speaking object, and the third modality feature is the facial feature; or... Using a behavior recognition model, behavioral features are extracted from the image data to obtain the behavioral features of the first speaking object, and the third modality feature is the behavioral feature.

13. An electronic device, characterized in that, Includes at least one processor and at least one memory; The at least one memory is used to store computer program instructions and data; The at least one processor is configured to execute computer program instructions and data that call the at least one memory to cause the electronic device to perform the method as described in any one of claims 1-12.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program or instructions that, when executed by an electronic device, cause the method as described in any one of claims 1-12 to be implemented.

15. A computer program product, characterized in that, The computer program product includes a computer program or instructions that, when executed on an electronic device, cause the method as described in any one of claims 1-12 to be implemented.

Images