Method for transmitting multi-modal encoded / decoded information and electronic device

Abstract

The present application provides a method for transmitting multi-modal encoded / decoded information and an electronic device. A first electronic device can encode first information to be sent into an optical signal and an acoustic wave signal, the optical signal comprising any one of an image and light, and can display the optical signal and play the acoustic wave signal. After acquiring the optical signal and the acoustic wave signal, a second electronic device can determine the first information on the basis of the optical signal and the acoustic wave signal. Since the first information is carried in a plurality of channels, blockage and interference of a single channel can be avoided, thereby improving the transmission efficiency and success rate of the information.

Description

A method for multimodal encoding and decoding information transmission and an electronic device

[0001] This application claims priority to Chinese Patent Application No. 202410680401.X, filed on May 29, 2024, entitled "A Method for Multimodal Encoding and Decoding Information Transmission and an Electronic Device", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of electronic devices, and more specifically, to a method for multimodal encoding and decoding information transmission and an electronic device. Background Technology

[0003] With users owning an increasing number of electronic devices, scenarios requiring connections and data transfer between these devices are frequent. For example, after purchasing a new phone, the old phone can transfer data to the new phone after a connection is established. Currently, this connection is primarily established by scanning a QR code to obtain a personal identification number (PIN). However, this method is susceptible to factors such as screen moiré patterns and reflections, reducing the connection success rate and negatively impacting user experience. Therefore, how to quickly establish connections and transfer data between devices has become a pressing technical challenge. Summary of the Invention

[0004] This application provides a method and electronic device for multimodal encoding and decoding information transmission. The first electronic device can encode the first information to be transmitted into one or more channels, so that the second electronic device can obtain the first information by decoding the one or more channels, thereby avoiding the blocking and interference of a single channel and improving the efficiency and success rate of information transmission.

[0005] In a first aspect, a multimodal encoding method is provided, which is applied to a first electronic device. The method includes: encoding first information into an optical signal and an acoustic signal, wherein the optical signal is either an image or light; displaying the optical signal and playing the acoustic signal.

[0006] In this embodiment, the first electronic device can encode the first information to be sent into optical and acoustic signals, so that the second electronic device can obtain the first information by decoding the optical and acoustic signals, thereby avoiding the blocking and interference of a single channel and improving the transmission efficiency and success rate of information.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, encoding the first information into an optical signal and an acoustic signal includes: dividing the first information into a first part and a second part, wherein the data amount of the first part is greater than the data amount of the second part; encoding the first part into the optical signal and encoding the second part into the acoustic signal.

[0008] In this embodiment, since the first electronic device can encode the first information onto multiple channels, and these different channels can carry different parts of the first information, the transmission rate of the first information can be increased. Furthermore, compared to a single channel, the amount of encoding can be increased, allowing the first electronic device to transmit larger amounts of information.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the first part and the second part contain at least partially identical information.

[0010] In this embodiment, the first part and the second part contain at least partially identical information, which can improve the success rate of decoding.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the size of the first information is greater than or equal to a first threshold, which is the maximum information carrying capacity of the optical signal.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, when the duration of displaying the optical signal and playing the acoustic signal is greater than or equal to a first duration and no response to the first information is received within the first duration, the first part is encoded into the acoustic signal and the second part is encoded into the optical signal, wherein the first duration is the longest duration for the first electronic device to wait to receive the response to the first information.

[0013] In this embodiment, when the first electronic device displays light signals and plays sound signals for a long time and it is determined that the second electronic device has not obtained the first information, the encoding strategy can be flexibly adjusted, which can improve the success rate of the second electronic device finally obtaining the first information and help improve the user experience.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, the second part only includes the verification information in the first information.

[0015] In this embodiment, the first information includes verification information, and the second electronic device can verify the first part of the information through the verification information, thereby improving the success rate of decoding.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, when the optical signal is the image, the image comprises multiple frames that encode the first portion.

[0017] In conjunction with the first aspect, in some implementations of the first aspect, the first information includes file identification information of the first file, and the method further includes: receiving a first request, wherein the first request is a request generated by the second electronic device based on the first information, requesting the first electronic device to send the first file, and the first electronic device and the second electronic device have established a communication connection; and in response to the first request, sending the first file to the second electronic device.

[0018] In this embodiment, the first information may be the file identification information of a first file. The first electronic device can encode the file identification information into optical and acoustic signals. After obtaining the file identification information, the second electronic device can request the first file from the first electronic device. In response to the request, the first electronic device can send the first file to the second electronic device. Compared with traditional file sending methods, the file sending method in this embodiment is simple, requires no multiple steps from the user, and improves the user experience.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, the response to the first request to send the first file to the second electronic device includes: determining first identification information according to the first request; determining the first file according to the first identification information, wherein the first identification information is at least partially identical to the file identification information of the first file; and sending the first file to the second electronic device.

[0020] In conjunction with the first aspect, in some implementations of the first aspect, the first information includes file identification information of the first file, and the method further includes: receiving a second request, wherein the second request is a request generated by the second electronic device based on the first information to establish a communication connection with the first electronic device; and in response to the second request information, establishing the communication connection with the second electronic device and sending the first file to the second electronic device.

[0021] In conjunction with the first aspect, in some implementations of the first aspect, the first information also includes the progress information of the first file.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, the first information includes device connection information.

[0023] In conjunction with the first aspect, in some implementations of the first aspect, the first information includes the URL of the first webpage.

[0024] In conjunction with the first aspect, in some implementations of the first aspect, the first information includes authentication information of the first application, and the method further includes: receiving a first token sent by the server of the first application, wherein the first token is generated by the server of the first application based on the authorization of the second electronic device; and logging into the first application based on the first token.

[0025] Secondly, a multimodal decoding method is provided, which is applied to a second electronic device. The method includes: acquiring an optical signal and an acoustic signal, wherein the optical signal is either an image or a light source; and determining first information based on the optical signal and the acoustic signal.

[0026] In conjunction with the second aspect, in some implementations of the second aspect, the first information includes a first part and a second part, the first part being carried by the optical signal and the second part being carried by the acoustic signal, the data amount of the first part being greater than or equal to the data amount of the second part, and determining the first information based on the optical signal and the acoustic signal includes: determining the first part and the second part respectively based on the optical signal and the acoustic signal; performing frame alignment on the first part and the second part based on the frame header of the first part and the frame header of the second part; and splicing the frame-aligned first part and the second part to determine the first information.

[0027] In conjunction with the second aspect, in some implementations of the second aspect, the first part and the second part have at least partially identical information, and the method further includes: eliminating redundant information in the at least partially identical information.

[0028] In conjunction with the second aspect, in some implementations of the second aspect, the second part only includes the verification information in the first information, and the method further includes: verifying the information in the first part according to the verification information.

[0029] In conjunction with the second aspect, in some implementations of the second aspect, the size of the first information is greater than or equal to a first threshold, which is the maximum information carrying capacity of the optical signal.

[0030] In conjunction with the second aspect, in some implementations of the second aspect, when the optical signal is the image, the image comprises multiple frames that encode the first part.

[0031] In conjunction with the second aspect, in some implementations of the second aspect, the first information includes device connection information. When the second electronic device fails to establish a connection with the first electronic device based on the first information, or determines that the first information is incomplete, the method further includes: sending a first indication message to the first electronic device, the first indication message being used to indicate that the second electronic device has not obtained complete information, or that the second electronic device has decoded incorrectly.

[0032] In conjunction with the second aspect, in some implementations of the second aspect, the first information includes file identification information of the first file, and the method further includes: generating and sending a first request based on the first information, the first request being used to request a first electronic device to send the first file, wherein the first electronic device and the second electronic device have established a communication connection; and receiving the first file sent by the first electronic device.

[0033] In conjunction with the second aspect, in some implementations of the second aspect, the first information includes the URL of the first webpage, and the method further includes: displaying the first webpage according to the first information.

[0034] In conjunction with the second aspect, in some implementations of the second aspect, the first information includes authentication information of the first application, and the method further includes: displaying a first interface based on the first information, the first interface being an authorization login interface; and in response to the user's confirmation of authorization login, sending instruction information to the server of the first application, the instruction information being used to instruct the authorized first electronic device to log in to the first application using a first account.

[0035] Thirdly, a method for multimodal information transmission is provided, which is applied to a system including a first electronic device and a second electronic device, wherein the first electronic device is used to execute the first aspect or any possible implementation thereof, and the second electronic device is used to execute the second aspect or any possible implementation thereof.

[0036] Fourthly, an electronic device is provided, comprising one or more processors; one or more memories; the one or more memories storing one or more computer programs, the one or more computer programs including instructions that, when executed by the one or more processors, cause the foregoing aspects or any possible implementation thereof to be performed.

[0037] Fifthly, a computer-readable storage medium is provided, comprising a computer program or instructions that, when executed on a computer, cause the first aspect and any possible implementation of the method of the first aspect to be performed.

[0038] In a sixth aspect, a computer program product is provided, comprising a computer program or instructions that, when executed on a computer, cause the first aspect and any possible implementation of the method of the first aspect to be performed.

[0039] In a seventh aspect, a computer program is provided that, when run on a computer, causes the methods described in the first aspect and any possible implementation thereof to be executed.

[0040] Eighthly, an electronic device according to an embodiment of this application includes modules / units for performing the above aspects or any possible design of the above aspects; these modules / units can be implemented in hardware or by hardware executing corresponding software.

[0041] Ninth aspect, a smart driving device is provided, which includes electronic devices as described in the fourth aspect.

[0042] In a tenth aspect, an information transmission system is provided, including a first electronic device and a second electronic device, the first electronic device being used to execute the first aspect or any possible implementation thereof, and the second electronic device being used to execute the second aspect or any possible implementation thereof.

[0043] For the beneficial effects of aspects four through ten, please refer to the beneficial effects of aspects one and two, and they will not be repeated here. Attached Figure Description

[0044] Figure 1 is a schematic diagram of the structure of the electronic device provided in an embodiment of this application.

[0045] Figure 2 is a software structure block diagram of the electronic device provided in an embodiment of this application.

[0046] Figure 3 is a schematic diagram of the system architecture provided in an embodiment of this application.

[0047] Figure 4 is a schematic diagram of a ring-shaped region provided in an embodiment of this application.

[0048] Figure 5 is a schematic diagram of a scenario provided by an embodiment of this application.

[0049] Figure 6 is a schematic diagram of a scenario provided by an embodiment of this application.

[0050] Figure 7 is a schematic diagram of a scenario provided by an embodiment of this application.

[0051] Figure 8 is a schematic diagram of a scenario provided by an embodiment of this application.

[0052] Figure 9 is a schematic diagram of a scenario provided by an embodiment of this application.

[0053] Figure 10 is a schematic diagram of a scenario provided by an embodiment of this application.

[0054] Figure 11 is a schematic flowchart of the multimodal encoding and decoding method provided in the embodiments of this application.

[0055] Figure 12 is a schematic flowchart of the multimodal encoding and decoding method provided in the embodiments of this application. Detailed Implementation

[0056] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0057] The terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, “at least one” and “one or more” refer to one, two, or more than two. The term “and / or” is used to describe the relationship between related objects, indicating that three relationships may exist; for example, A and / or B can indicate: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship.

[0058] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0059] The following describes an electronic device and embodiments for using such an electronic device. In some embodiments, the electronic device may be a portable electronic device that also includes other functions such as a personal digital assistant and / or music player, such as a mobile phone, tablet computer, wearable electronic device with wireless communication capabilities (such as a smartwatch), etc. Exemplary embodiments of the portable electronic device include, but are not limited to, carrying... Alternatively, it can be a portable electronic device with another operating system. The aforementioned portable electronic device can also be other portable electronic devices, such as laptops. It should also be understood that in some other embodiments, the aforementioned electronic device may not be a portable electronic device, but rather a desktop computer. In some embodiments, the electronic device can be a component of a smart driving device.

[0060] For example, Figure 1 shows a schematic diagram of the structure of an electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0061] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0062] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0063] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0064] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0065] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0066] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0067] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0068] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0069] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0070] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0071] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-CDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0072] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0073] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0074] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0075] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0076] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0077] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0078] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0079] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0080] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0081] Figure 2 is a software structure block diagram of an electronic device 100 according to an embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer. The application layer may include a series of application packages.

[0082] As shown in Figure 2, the application layer can include camera, settings, third-party applications, etc. Third-party applications can include gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, SMS, etc.

[0083] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer may include some predefined functions.

[0084] As shown in Figure 2, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0085] The window manager is used to manage windowed applications. It can obtain the screen size, determine if a status bar is present, lock the screen, and capture screenshots. The content provider stores and retrieves data, making this data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0086] The view system includes visual controls, such as controls for displaying text, controls for displaying images, and such as the indicator information for displaying the virtual shutter button in the embodiments of this application. The view system can be used to build applications. The display interface can consist of one or more views. For example, a display interface including a text message notification icon can include a view for displaying text and a view for displaying images.

[0087] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection and disconnection).

[0088] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0089] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0090] The Android runtime consists of core libraries and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system.

[0091] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0092] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0093] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0094] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0095] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0096] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0097] A 2D graphics engine is a graphics engine for 2D drawing.

[0098] In addition, the system library may also include status monitoring service modules, such as a physical status recognition module for analyzing and recognizing user gestures; and a sensor service module for monitoring sensor data uploaded by various sensors at the hardware layer to determine the physical status of the electronic device 100.

[0099] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0100] The hardware layer may include various types of sensors, such as the various sensors shown in Figure 1, including accelerometers, gyroscopes, and touch sensors involved in the embodiments of this application.

[0101] It should be noted that Figure 2 only illustrates one way of dividing the system framework and should not be construed as a specific limitation on the embodiments of this application. In the embodiments of this application, when the electronic device is equipped with different operating systems, different frameworks can be used for different operating systems. It is understood that when different frameworks are adopted, the way the framework layers are divided, the specific naming, and the specific layer in which each of the above modules is located can be different.

[0102] With users owning an increasing number of electronic devices, scenarios requiring connections and data transfer between these devices are frequent. For example, after purchasing a new phone, the old phone can transfer data to the new phone after a connection is established. Currently, this connection is primarily established by scanning a QR code to obtain a personal identification number (PIN). However, this method is susceptible to factors such as screen moiré patterns and reflections, reducing the connection success rate and negatively impacting user experience. Therefore, how to quickly establish connections and transfer data between devices has become a pressing technical challenge.

[0103] Based on this, embodiments of this application provide a multimodal encoding and decoding method, allowing an electronic device to transmit data to other electronic devices in parallel through various methods, without limitation on the specific data content. For example, the data may be device connection information used to establish a device connection, a file (e.g., an image, a phone number, a piece of text, etc.), file identification information (e.g., a file identifier corresponding to a video, a storage path corresponding to an image, etc.), or a URL link, etc.

[0104] The following is a schematic diagram of the system architecture provided in the embodiments of this application.

[0105] Figure 3 shows a schematic diagram of the system architecture provided in an embodiment of this application.

[0106] As shown in Figure 3, the first electronic device and the second electronic device may include a distributed service module, an authentication module, and a fusion sensing module.

[0107] The fusion sensing module may include one or more of the following: sound wave encoding / decoding module, image encoding / decoding module, vibration waveform encoding / decoding module, etc.

[0108] The acoustic wave encoding / decoding module is used to encode information into acoustic waves and decode the information contained in acoustic waves.

[0109] The image encoding / decoding module is used to encode information into an image and to decode the information contained in the image.

[0110] The vibration waveform encoding and decoding module is used to encode information into the vibration waveform and decode the information contained in the vibration waveform.

[0111] It should be noted that the above description of the fusion perception module is only an example, and the fusion perception module may include more or fewer modules.

[0112] The communication module may include a communication transmission module and a proximity detection module.

[0113] The communication transmission module is used to transmit data to other electronic devices.

[0114] The proximity detection module is used to detect nearby electronic devices.

[0115] The authentication module is used for authentication between electronic devices. After successful authentication, electronic devices can transmit data to complete specific distributed services such as device switching, screen mirroring, and data transfer.

[0116] Distributed services refer to various services that require collaboration between a first electronic device and a second electronic device. A distributed service module manages these specific services. For example, a distributed service module might be responsible for one or more specific services such as device switching, screen mirroring, or data transfer.

[0117] This application provides a multimodal encoding and decoding technical solution, in which a first electronic device can encode the first information to be sent into one or more different channels, and a second electronic device can obtain the first information through the one or more different channels.

[0118] When encoding using N channels, the first information is first sliced ​​into N data segments, where N is an integer greater than or equal to 2. The first information can be divided into N equal data segments based on data volume; in this case, any one of the data segments can be the main information for multimodal coding. Alternatively, the first information can be divided unequally, in which case the data segment with the largest data volume among the N segments is the main information for multimodal coding.

[0119] In some embodiments, the first information includes data information, which can be understood as the information that the first electronic device actually needs to send.

[0120] The first electronic device can encode the transmitted data information into one or more channels, so that the second electronic device can obtain the data information through the one or more channels.

[0121] For example, data information can include device connection information, file identification information, URLs, phone numbers, or a piece of text. File identification information is used to indicate the file to be sent.

[0122] For example, file identification information can be a file identifier, file handle, file storage path, etc. Device connectivity information is used for authentication and identification.

[0123] For example, device connection information may be a PIN code, International Mobile Equipment Identity (IMEI), Unique Device Identifier (UDID), Universally Unique Identifier (UUID), Hardware Identifier (a unique identifier of a device in a wireless LAN service, including Wi-Fi MAC address and Bluetooth MAC address), or other identifiers that can uniquely identify the first electronic device.

[0124] In some embodiments, the first information includes data information and verification information, the verification information being used to verify the data information.

[0125] In this embodiment of the application, the type of verification information is not specifically limited. The verification information may be parity check bit, Hamming code, cyclic redundancy check code, etc.

[0126] The above describes how the first electronic device can encode first information into one or more channels and the type of first information. These one or more channels can be visible light channels, sound wave channels, vibration sensing channels, touch sensing channels, etc. After receiving data from multiple channels, the second electronic device decodes the corresponding first information.

[0127] Taking the fusion of visible light and audio channels as an example, the first electronic device can encode the first information into both visible light and sound waves. The method by which the first electronic device encodes the first information into visible light will be described firstly below.

[0128] Visible light encoding:

[0129] Visible light coding includes various coding methods. Several possible implementations are illustrated below:

[0130] Encode the first information into the image:

[0131] This application does not specifically limit the method of encoding the first information into the image. Several possible implementation methods are described below by example.

[0132] One possible implementation is that the first electronic device can encode the first information into an encoded pattern such as a barcode or a two-dimensional barcode, and then display the barcode or two-dimensional barcode. This encoding method can be referred to as explicitly encoding the first information into an image.

[0133] One possible implementation involves the first electronic device encoding the first information into feature points and then adding these feature points to the image. This encoding method can be called implicitly encoding the first information into the image. In this implementation, the encoded pattern is not a traditional barcode or two-dimensional barcode, but rather the first electronic device converts the information into feature points and then combines these feature points with the image content. For the user, this results in a normal image, where the overlaid information is imperceptible.

[0134] The first electronic device can select a ring-shaped image composed of particle dots. The first electronic device can superimpose feature points onto this image. It is understood that the size and brightness of the superimposed feature points can be similar to or comparable to the particle dots; for a user, when directly viewing the image with the naked eye, the feature points hidden in the image are invisible or unrecognizable. When the second electronic device scans the image, it can identify the feature points, thereby determining the corresponding first information based on the feature points.

[0135] Example 1: The first electronic device can divide the annular region composed of particle points in the image into multiple regions on an average basis, determine the number of feature points according to the first information, and then add the corresponding number of feature points to each of the multiple regions.

[0136] For example, Figure 4 is a schematic diagram of a ring-shaped region provided in an embodiment of this application. Referring to Figure 4, the first electronic device can divide the ring-shaped region into six regions in sequence, namely the first region, the second region, the third region, the fourth region, the fifth region, and the sixth region. Assuming that the first information includes a PIN code, which may include a 6-digit number such as 656579, the first electronic device can add 6 feature points in the first region, 5 feature points in the second region, 6 feature points in the third region, 5 feature points in the fourth region, 7 feature points in the fifth region, and 9 feature points in the sixth region. When the first electronic device displays the image shown in Figure 4, in the T-frame image, the brightness value of the feature point pixel is increased compared to the brightness of the particle point in the image, and the increased brightness value is less than a certain threshold. In this case, the human eye cannot distinguish between the particle point and the feature point in the image. In the T+1-frame image, the brightness value of the feature point pixel is decreased compared to the brightness of the particle point. The second electronic device scan can identify the ring-shaped region composed of particle points. Since the brightness of the feature points in the T-th frame image and the T+1-th frame image is different, the number of feature points in the above 6 regions can be obtained by performing differential processing on the T-th frame image and the T+1-th frame image, and then the PIN code can be determined.

[0137] Example 2: The first electronic device can divide the ring-shaped region composed of particle points in the image into multiple regions on an average basis, then divide each of the multiple regions into 10 sub-regions, and then add feature points to a certain sub-region of each region according to the received information.

[0138] For example, referring to Figure 4 again, and still assuming the first information includes a PIN code, and the PIN code is 6 digits long, each region in the figure corresponds to one digit of the PIN code. For instance, the first region corresponds to the first digit of the PIN code, the second region to the second digit, the third region to the third digit, the fourth region to the fourth digit, the fifth region to the fifth digit, and the sixth region to the sixth digit. Each of these six regions can be further divided into 10 sub-regions (not shown in the figure) to correspond to specific numbers. The PIN code can include a 6-digit number, such as 656579. The first electronic device can add feature points to sub-region 6 of the first region, sub-region 5 of the second region, sub-region 6 of the third region, sub-region 5 of the fourth region, sub-region 7 of the fifth region, and sub-region 9 of the sixth region. When the first electronic device displays the image shown in Figure 4, in the T-frame image, the brightness value of the feature point pixels is increased slightly compared to the brightness of the particle points, and the increase is less than a certain threshold. This makes it impossible for the human eye to distinguish between the particle points and the feature points in the image. In the T+1-frame image, the brightness value of the feature point pixels is decreased slightly compared to the brightness of the particle points. After the second electronic device scans these two frames, since the brightness of the feature points in the T-frame and T+1-frame images is opposite, by performing differential processing on the T-frame and T+1-frame images, the specific sub-region location of the feature point can be obtained, and thus the PIN code can be determined. It is understood that in this example, the number of feature points added by the first electronic device to the sub-region is not limited.

[0139] One possible implementation involves a first electronic device converting the first information into digital watermark information, which is then hidden and overlaid into an image, making it invisible to the user. This digital watermark information can be a barcode, a two-dimensional barcode, or similar format. This encoding method can also be referred to as implicitly encoding information into an image.

[0140] It should be understood that the first electronic device superimposed on the image means that the digital watermark information is invisible or substantially invisible to the user. Alternatively, it can be understood that when the user views the image directly with the naked eye without the aid of other devices, they cannot see or recognize the digital watermark information.

[0141] For example, a first electronic device can overlay digital watermark information onto a certain area of ​​a displayed image. This area has a high texture complexity, allowing the digital watermark information to be well hidden within the image, making it invisible to the user. A second electronic device, after scanning the image displayed by the first device with a camera, can perform differential processing on this image and the original image without the overlaid digital watermark information to obtain the hidden digital watermark information within the image.

[0142] For example, when the first electronic device overlays digital watermark information onto a displayed image, it can overlay the watermark information onto two adjacent frames of the image. For instance, in frame T, the brightness value of the digital watermark information pixels is increased slightly, but the increase is less than a certain threshold. In this case, the image and the digital watermark information are indistinguishable to the human eye. In frame T+1, the brightness value of the digital watermark information pixels is decreased slightly. This makes the digital watermark information well-hidden, invisible or virtually invisible to the user. After the second electronic device scans these two frames, since the brightness of the digital watermark information in frame T and frame T+1 is opposite, by performing differential processing on the two frames, the digital watermark information excluding the original image can be obtained.

[0143] One possible implementation is that the first electronic device can convert the first information into binary encoded information, and then represent the binary encoded information through the brightness changes of multiple frames of images. The image content of these multiple frames may be the same or different. This encoding method can be called implicitly encoding information into the image.

[0144] For example, taking a PIN code as the first piece of information, this PIN code can include a 6-digit number, such as 656579. The corresponding binary encoding information for this PIN code is "0110 0101 0110 0101 0111 1001". A binary code "0" indicates a dark display, and a binary code "1" indicates a bright display. The first electronic device can control the display screen to adjust the brightness of the image based on this binary encoding information. The image content displayed on the screen can be arbitrary. When the second electronic device scans the display screen with a camera, it can capture changes in brightness and thus determine the binary encoding information.

[0145] Understandably, because the frequency of changes on a display screen is too high, exceeding the range of human visual perception, users cannot perceive the changes in brightness. Therefore, from the user's perspective, the display screen appears to be showing images normally.

[0146] In this embodiment of the application, when the first electronic device encodes the first information into an image, it can encode the first information into one frame of the image, or it can encode the information into multiple frames of the image.

[0147] In some embodiments, when the first electronic device encodes the first information into multiple frames of images, some of the frames may carry the same information.

[0148] The first electronic device can divide the first information into multiple data slices. When encoding the multiple data slices into the multiple frame images, each frame image can carry a portion of the data slices of the first information, wherein the data slices carried by the partial frame images are the same.

[0149] For example, the first electronic device can divide the first information into six data slices, namely the first data slice, the second data slice, the third data slice, the fourth data slice, the fifth data slice, and the sixth data slice. The first electronic device can encode the first data slice, the second data slice, and the third data slice into the first frame image and the second frame image, and encode the fourth data slice, the fifth data slice, and the sixth data slice into the third frame image and the fourth frame image. That is, the first frame image and the second frame image carry the same data slices, and the third frame image and the fourth frame image carry the same data slices.

[0150] In some embodiments, when the first electronic device encodes first information into multiple frames of images, some of the frames may carry the same information.

[0151] The first electronic device can divide the first information into multiple data slices. When encoding the multiple data slices into the multiple frame images, each frame image can carry a portion of the data slices of the first information, wherein the data slices carried by the partial frame images are partially the same.

[0152] For example, the first electronic device can divide the first information into six data slices, namely, the first data slice, the second data slice, the third data slice, the fourth data slice, the fifth data slice, and the sixth data slice. The first electronic device can encode the first data slice, the second data slice, and the third data slice into the first frame image and the second frame image; encode the third data slice, the fourth data slice, and the fifth data slice into the third frame image and the fourth frame image; and encode the fourth data slice, the fifth data slice, and the sixth data slice into the fifth frame image and the sixth frame image. That is, the data slices carried by the first frame image and the second frame image are partially the same as the data slices carried by the third frame image and the fourth frame image; and the data slices carried by the third frame image and the fourth frame image are partially the same as the data slices carried by the fifth frame image and the sixth frame image.

[0153] In some embodiments, when the first electronic device encodes first information into a multi-frame image, each frame of the multi-frame image carries different information.

[0154] The first electronic device can divide the first information into multiple data slices, and then encode each data slice into a frame of image. It is understood that, in this embodiment, the number of data slices is the same as the number of frames in the image.

[0155] For example, the first electronic device can divide the first information into six data slices, namely the first data slice, the second data slice, the third data slice, the fourth data slice, the fifth data slice, and the sixth data slice. The first electronic device can encode the first data slice into the first frame image, the second data slice into the second frame image, the third data slice into the third frame image, the fourth data slice into the fourth frame image, the fifth data slice into the fifth frame image, and the sixth data slice into the sixth frame image.

[0156] The above describes how to encode the first information into an image; the following describes how to encode the first information into light.

[0157] Encode the first piece of information into the light:

[0158] In this embodiment of the application, encoding the first information into the light can include the following possible implementation methods.

[0159] One possible implementation is that the first electronic device can encode the first information into different colors and then control the light to display those different colors.

[0160] For example, taking a PIN code (656579) as the first piece of information, the first electronic device can determine that the digit 0 corresponds to the first color, the digit 1 to the second color, the digit 2 to the third color, the digit 3 to the fourth color, the digit 4 to the fifth color, the digit 5 ​​to the sixth color, the digit 6 to the seventh color, the digit 7 to the eighth color, the digit 8 to the ninth color, and the digit 9 to the tenth color. Therefore, the colors corresponding to this PIN code are the seventh, sixth, seventh, sixth, eighth, and ninth colors. The first electronic device can control the lights to display these colors. The second electronic device can identify these colors through a camera, thereby determining the PIN code.

[0161] In some embodiments, the RGB value difference between the first color, second color, third color, fourth color, fifth color, sixth color, seventh color, eighth color, ninth color and tenth color can be set to be less than a certain threshold, so that the user's eyes cannot perceive or can hardly perceive the color change.

[0162] One possible implementation is that the first electronic device can encode the first information into different brightness levels, and then control the light to present those different brightness levels.

[0163] For example, assuming the first piece of information includes a PIN code, and the PIN is 656579, the first electronic device determines that the digit 0 corresponds to the first brightness level, the digit 1 to the second brightness level, the digit 2 to the third brightness level, the digit 3 to the fourth brightness level, the digit 4 to the fifth brightness level, the digit 5 ​​to the sixth brightness level, the digit 6 to the seventh brightness level, the digit 7 to the eighth brightness level, the digit 8 to the ninth brightness level, and the digit 9 to the tenth brightness level. The second electronic device can identify these brightness levels through a camera, and thus determine the PIN code.

[0164] In some embodiments, the difference between the brightness values ​​of the first brightness, second brightness, third brightness, fourth brightness, fifth brightness, sixth brightness, seventh brightness, eighth brightness, ninth brightness, and tenth brightness can be set to be less than a certain threshold, so that the user's eyes cannot perceive or can barely perceive the change in brightness.

[0165] For example, taking a PIN code as the first piece of information, this PIN code can include a 6-digit number, such as 656579. The corresponding binary encoding information for this PIN code is "0110 0101 0110 0101 0111 1001". A binary code "0" indicates a dark display, and a binary code "1" indicates a bright display. The first electronic device can then control the light to turn on or off based on this binary encoding information. When the second electronic device scans with a camera, it can capture changes in brightness and thus determine the binary encoding information.

[0166] Acoustic coding:

[0167] Acoustic encoding refers to a first electronic device encoding first information into a sound wave, and then playing the sound wave through a speaker to transmit the first information carried in the sound wave. The sound wave may correspond to the ultrasonic frequency band or the audible sound frequency band. This application does not specifically limit the method of encoding the first information into the sound wave; several possible implementation methods are exemplarily described below.

[0168] One possible implementation is that the first electronic device can transmit the first information through changes in the frequency of sound waves.

[0169] For example, taking a PIN code as the first piece of information, this PIN code can include a 6-digit number, such as 656579. The corresponding binary encoding information for this PIN code is "0110 0101 0110 0101 0111 1001". In this binary encoding, "0" represents the first frequency, and "1" represents the second frequency. The first electronic device can then control the speaker to change its frequency according to this binary encoding information. The second electronic device, after receiving the sound wave, can capture the frequency changes and thus determine the binary encoding information.

[0170] One possible implementation is that the first electronic device can transmit the first information through changes in the loudness of sound waves.

[0171] For example, taking a PIN code as the first piece of information, this PIN code can include a 6-digit number, such as 656579. The corresponding binary encoding information for this PIN code is "0110 0101 0110 0101 0111 1001". A binary code "0" represents the first loudness, and a binary code "1" represents the second loudness. The first electronic device can then control the speaker to change the loudness according to this binary encoding information. The second electronic device, after receiving the sound wave, can capture the change in loudness and thus determine the binary encoding information.

[0172] The foregoing has exemplarily described several methods for a first electronic device to encode information into visible light and sound waves, but the embodiments of this application do not specifically limit these methods. In other embodiments of this application, information can also be encoded into visible light and sound waves by other methods.

[0173] In some embodiments of this application, the first electronic device may encode first information into images and sound waves.

[0174] In other embodiments of this application, the first electronic device may encode the first information into light and sound waves.

[0175] The two encoding methods described above can also be called fusion encoding. The method of fusion encoding will be introduced in detail below.

[0176] In some embodiments, the size of the first information is less than a first threshold. The first electronic device can encode all of the first information into an image, and encode all or part of the first information into sound waves. The first threshold is the maximum information carrying capacity of the image; that is, when the first information is small and the image can fully carry the first information, then all of the first information can be encoded into the image.

[0177] When encoding, the first electronic device can encode all the information of the first information into an image, and encode part or all of the information of the first information into a sound wave.

[0178] It is understandable that, since the first electronic device encodes all the information of the first information into the image, the second electronic device can directly determine the first information by decoding the image.

[0179] In some embodiments, the first electronic device may encode part or all of the first information into an image, and encode all of the first information into a sound wave.

[0180] It should be understood that descriptions of encoding part or all of the first information into an image and encoding all of the first information into a sound wave by the first electronic device can be found in the descriptions of encoding all of the first information into an image and encoding all or part of the first information into a sound wave by the first electronic device, which will not be repeated here for the sake of brevity.

[0181] In some embodiments, the size of the first information is greater than or equal to a first threshold, and the first electronic device can encode a portion of the first information into an image and a portion of the first information into a sound wave. The information carried in the image and the information carried in the sound wave may be different or partially the same.

[0182] The first electronic device may include the following possible implementations when encoding a portion of the first information into an image and when encoding a portion of the first information into a sound wave.

[0183] One possible implementation is that the first electronic device can divide the first information into a first part and a second part, encode the first part into an image, and encode the second part into a sound wave, wherein the first part and the second part are different.

[0184] When encoding, the first electronic device can divide the first information into a first part and a second part. The first part consists of one or more data slices, and the second part consists of one or more data slices. The data slices of the first part and the second part are different. The first electronic device can encode the first part into an image and the second part into a sound wave.

[0185] For example, the first information includes data information, the first part is a portion of the data information, and the second part is another portion of the data information.

[0186] For example, the first information includes data information and verification information. The first part is data information, and the second part is verification information.

[0187] It should be noted that the above examples only take the first part as a part of the data information and the second part as another part of the data information, or the first part as the data information and the second part as the verification information, and should not be construed as a specific limitation on the embodiments of this application. In the embodiments of this application, there may be other ways to divide the first information.

[0188] In one possible implementation, the first electronic device can divide the first information into a first part and a second part, encode the first part into an image, and encode the second part into a sound wave, wherein the first part and the second part contain at least partially identical information.

[0189] When encoding, the first electronic device can divide the first information into a first part and a second part. The first part consists of one or more data slices, and the second part consists of one or more data slices. The data slices of the first part and the second part are the same. The first electronic device can encode the first part into an image and the second part into a sound wave.

[0190] For example, if the first part includes the first to the fifth data slices and the second part includes the third to the sixth data slices, then the first part and the second part contain overlapping third, fourth and fifth data slices.

[0191] In some embodiments, the size of the first portion carried in the image is equal to the size of the second portion carried in the sound wave.

[0192] The first electronic device can divide the first information into a first part and a second part, wherein the first part is encoded into an image and the second part is encoded into a sound wave, and the size of the first part is equal to the size of the second part.

[0193] For example, the first electronic device divides the first information into a first part and a second part. The first part includes a first data slice to a third data slice, and the second part includes a fourth data slice to a sixth data slice. The sum of the first data slice to the third data slice is equal to the sum of the fourth data slice to the sixth data slice.

[0194] In some embodiments, the size of the first portion of the image carried is larger than the size of the second portion of information carried in the sound waves.

[0195] The first electronic device can divide the first information into a first part and a second part, wherein the first part is encoded into an image and the second part is encoded into a sound wave, and the size of the first part is larger than the size of the second part.

[0196] For example, the first electronic device divides the first information into a first part and a second part. The first part includes a first data slice to a fifth data slice, and the second part includes a sixth data slice. The sum of the first data slice to the fifth data slice is greater than the size of the sixth data slice.

[0197] In some embodiments, the size of the first portion carried on the image is smaller than the size of the second portion carried on the sound wave.

[0198] The first electronic device can divide the first information into a first part and a second part, wherein the first part is encoded into an image and the second part is encoded into a sound wave, and the size of the first part is smaller than the size of the second part.

[0199] For example, the first electronic device divides the first information into a first part and a second part. The first part includes a first data slice and a second data slice, and the second part includes a third data slice to a sixth data slice. The sum of the first data slice and the second data slice is less than the sum of the third data slice to the sixth data slice.

[0200] In some embodiments, when the first electronic device encodes the first information into an image and a sound wave, the image may be a multi-frame image over a period of time, and the sound wave may be a sound wave over a period of time.

[0201] It should be understood that the description of the first electronic device encoding the first information into multiple frames of images can be found above, and will not be repeated here for the sake of brevity.

[0202] In some embodiments, the first electronic device may encode all the information of the first information into sound waves, and encode all or part of the information of the first information into light.

[0203] In some embodiments, the first electronic device may encode part or all of the first information into sound waves, and encode all of the first information into light.

[0204] In some embodiments, the first electronic device may encode a portion of the first information into sound waves and a portion of the first information into light.

[0205] It should be understood that the description of encoding all or part of the first information into the light and encoding all or part of the first information into the sound wave by the first electronic device can be found above, and will not be repeated here for the sake of brevity.

[0206] As described above, the first electronic device can divide the first information into multiple parts and encode these parts onto different channels. Each of these parts can consist of one or more data slices. Each data slice of the first information corresponds to a frame number, and the frame numbers corresponding to each data slice are different. When encoding the data slices, the first electronic device also encodes the frame numbers of the data slices, that is, it can encode the frame number and the data slice corresponding to that frame number as a complete data frame. The data slices obtained by the second electronic device during decoding may be out of order. The second electronic device can sort them according to the frame numbers, that is, perform data frame alignment, and then organize the data slices according to the frame numbers to determine the first information.

[0207] In some embodiments, when the first electronic device encodes the first information into an image and a sound wave, the image may be a multi-frame image. Since it is a multi-frame image, the first electronic device needs to display the multi-frame image and play the sound wave for a period of time. Correspondingly, the second electronic device can acquire the multi-frame image and the sound wave for that period of time and decode the first part of the first information carried by the multi-frame image and the second part of the first information carried by the sound wave, respectively.

[0208] In some embodiments, when encoding, the first electronic device may encode all the information of the first information into one channel, and encode part or all of the information of the first information into another channel. It is understood that since the first electronic device encodes all the information of the first information into one channel, the second electronic device can directly determine the first information by decoding that channel.

[0209] In some embodiments, the first electronic device divides the first information into a first part and a second part, encodes the first part into one channel, and encodes the second part into another channel. The second electronic device decodes the first channel to determine the first part and decodes the second channel to determine the second part. The second electronic device performs frame alignment on the first part and the second part according to their frame headers, and then the second electronic device can splice the frame-aligned first part and the second part to determine the first information.

[0210] It is understandable that the second electronic device can organize the data slices of the first part and the second part differently depending on the encoding method of the first electronic device.

[0211] In some embodiments, the data slices carried by visible light are different from the data slices carried by sound waves, and the second electronic device can splice the data slices in the order of frame numbers.

[0212] After the second electronic device decodes visible light (images or lights) and decodes sound waves, it can determine the data slices carried in visible light and the data slices carried in sound waves respectively. The second electronic device determines that there are no duplicate data slices based on the frame number, and then splices the data slices according to the frame number.

[0213] For example, a first electronic device divides the first information into six data slices: a first data slice, a second data slice, a third data slice, a fourth data slice, a fifth data slice, and a sixth data slice. The frame numbers of these data slices are the first frame number, the second frame number, the third frame number, the fourth frame number, the fifth frame number, and the sixth frame number, respectively. The first electronic device encodes the first to fifth data slices into an image and encodes the sixth data slice into a sound wave. A second electronic device can determine the first to fifth data slices by decoding the image and can determine the sixth data slice by decoding the sound wave. Since there are no duplicate frame numbers among these data slices, the second electronic device can sequentially concatenate the first, second, third, fourth, fifth, and sixth data slices according to their frame numbers to determine the first information.

[0214] In some embodiments, the data slices carried by visible light are partially the same as the data slices carried by sound waves, and the second electronic device can perform redundancy processing and splicing processing on the data slices according to the frame number.

[0215] After the second electronic device decodes visible light (images or lights) and decodes sound waves, it can determine the data slices carried in visible light and the data slices carried in sound waves respectively. If the second electronic device determines that there are duplicate data slices based on the frame number, it can first eliminate the duplicate data slices and then splice the data slices according to the order of the frame numbers.

[0216] For example, a first electronic device divides the first information into six data slices: a first data slice, a second data slice, a third data slice, a fourth data slice, a fifth data slice, and a sixth data slice. The frame numbers of these data slices are the first frame number, the second frame number, the third frame number, the fourth frame number, the fifth frame number, and the sixth frame number, respectively. The first electronic device encodes the first to fifth data slices into an image and encodes the fifth and sixth data slices into a sound wave. A second electronic device decodes the image to determine the first to fifth data slices and decodes the sound wave to determine the fifth and sixth data slices. Since the data slices contain a duplicate fifth frame number (i.e., the fifth data slice is a duplicate), the second electronic device first eliminates one fifth data slice and then, according to the frame number order, sequentially concatenates the first, second, third, fourth, fifth, and sixth data slices together to determine the first information.

[0217] In some embodiments, the data slices carried by visible light are the same as the data slices carried by sound waves, and the second electronic device can perform redundancy processing and splicing processing on the data slices according to the frame number.

[0218] After the second electronic device decodes visible light (images or lights) and decodes sound waves, it can determine the data slices carried in visible light and the data slices carried in sound waves respectively. If the second electronic device determines that there are duplicate data slices based on the frame number, it can first eliminate the duplicate data slices and then splice the data slices according to the order of the frame numbers.

[0219] For example, a first electronic device divides the first information into six data slices: a first data slice, a second data slice, a third data slice, a fourth data slice, a fifth data slice, and a sixth data slice. The frame numbers of these data slices are the first frame number, the second frame number, the third frame number, the fourth frame number, the fifth frame number, and the sixth frame number, respectively. The first electronic device encodes the first to sixth data slices into an image and encodes them into a sound wave. A second electronic device can determine the first to sixth data slices by decoding the image and by decoding the sound wave. Since there are duplicate first to sixth frame numbers among the data slices (i.e., the first to sixth data slices are duplicated), the second electronic device first eliminates one of the first to sixth data slices, and then sequentially concatenates the first, second, third, fourth, fifth, and sixth data slices according to their frame numbers to determine the first information.

[0220] As mentioned above, the first electronic device can divide the first information into multiple data slices. When the first electronic device encodes multiple data slices into the visible light (light or image) channel and the sound wave channel, the more slices there are, the greater the amount of data carried. It can also adjust the number of data slices carried in the visible light and sound waves, that is, dynamically adjust the amount of data encoded and transmitted in each channel.

[0221] In some embodiments, when the time for the first electronic device to display visible light and play sound waves is greater than or equal to a first duration, and it is determined that the second electronic device has not responded to the first information, the first electronic device may increase the number of data slices carried in the sound waves. The first duration can be understood as the longest time for the first electronic device to wait for the second electronic device to respond to the first information.

[0222] Since the first electronic device has displayed visible light and played sound waves for a considerable period, but determined that the second electronic device has not responded to the first information, it indicates that the second electronic device may not have yet acquired the first information. The reasons why the second electronic device may not have received the first information could include excessive ambient light intensity, moiré patterns generated when the second electronic device scans visible light, etc. Therefore, the first electronic device could increase the number of data slices carried in the sound waves.

[0223] Furthermore, the first electronic device can also reduce the number of data slices carried in visible light.

[0224] In some embodiments, when the first electronic device displays visible light and plays sound waves for a duration greater than or equal to a first duration, and determines that the intensity of ambient light is greater than or equal to a third threshold, and determines that the second electronic device does not respond to the first information, the first electronic device may increase the number of data slices carried in the sound waves.

[0225] Since the first electronic device has been displaying visible light and playing sound waves for a relatively long time, but the first electronic device has determined that the second electronic device has not responded to the first information, it indicates that the second electronic device may not have obtained the first information yet. Furthermore, the first electronic device has detected that the intensity of ambient light is greater than or equal to the third threshold. Therefore, the reason why the second electronic device has not received the first information may be that the intensity of ambient light is too strong. Thus, the first electronic device can increase the number of data slices carried in the sound waves.

[0226] Furthermore, the first electronic device can also reduce the number of data slices carried in visible light.

[0227] In some embodiments, when the time for which the first electronic device displays visible light and plays sound waves is greater than or equal to a first duration, the first electronic device may increase the number of data slices carried in the visible light.

[0228] Since the first electronic device has displayed visible light and played sound waves for a considerable period, but determined that the second electronic device has not responded to the first information, it indicates that the second electronic device may not have yet acquired the first information. The reason the second electronic device did not receive the first information may be due to excessive noise in the environment; therefore, the first electronic device could increase the number of data slices carried in the visible light.

[0229] Furthermore, the first electronic device can also reduce the number of data slices carried in the sound waves.

[0230] In some embodiments, when the first electronic device displays visible light and plays sound waves for a duration greater than or equal to a first duration, and determines that the intensity of ambient noise is greater than or equal to a fourth threshold, the first electronic device may increase the number of data slices carried in visible light.

[0231] Since the first electronic device has been displaying visible light and playing sound waves for a relatively long time, but the first electronic device determines that the second electronic device has not responded to the first information, it indicates that the second electronic device may not have obtained the first information yet. Furthermore, the first electronic device detects that the intensity of the ambient noise is greater than or equal to the fourth threshold. Therefore, the reason why the second electronic device has not received the first information may be that the intensity of the ambient noise is too strong. Thus, the first electronic device can increase the number of data slices carried in the visible light.

[0232] Furthermore, the first electronic device can also reduce the number of data slices carried in the sound waves.

[0233] In some embodiments, when the first electronic device determines that the intensity of ambient light is greater than or equal to a third threshold, the first electronic device may increase the number of data slices carried in the sound wave.

[0234] Furthermore, the first electronic device can also reduce the number of data slices carried in visible light.

[0235] In some embodiments, when the first electronic device determines that the intensity of ambient noise is greater than or equal to a fourth threshold, the first electronic device may increase the number of data slices carried in visible light.

[0236] Furthermore, the first electronic device can also reduce the number of data slices carried in the sound waves.

[0237] In some embodiments, when it is determined that the triggering condition is met, the first electronic device may be adjusted to encode all the information of the first information into a channel.

[0238] The first electronic device can first encode the first information into multiple channels, such as visible light and sound waves. When the first electronic device determines that the triggering condition is met, it can adjust the encoding strategy and no longer encode the first information into multiple channels, but instead encode all the information of the first information into one channel.

[0239] For example, the trigger condition is detecting that the display is off.

[0240] The first electronic device can first encode the first information into visible light and sound waves. When it detects that the display screen is off, since visible light can no longer be displayed on the display screen, the first electronic device can adjust to encode all the information of the first information into sound waves.

[0241] It should be noted that the above triggering conditions are merely examples and should not be construed as specific limitations on the embodiments of this application.

[0242] In this embodiment of the application, due to the influence of environmental parameters, the data information included in the first information obtained by the second electronic device may be incomplete and / or there may be decoding errors during decoding. The second electronic device may perform operations according to the different types of data information.

[0243] In some embodiments, when the data information included in the first information is data information of a first type, the second electronic device may send an indication message to the second electronic device to indicate that complete or correct data information has not been obtained, or it may indicate missing data slices.

[0244] The first type of data information can be understood as data information that the second electronic device needs to respond based on complete data information. Therefore, when the second electronic device determines that the data information included in the first information is incomplete or has a decoding error, it can send an indication message to the second electronic device. This indication message can be used to indicate that complete data information has not been obtained, or that a decoding error has occurred, or to indicate which specific data slice has been lost.

[0245] For example, the first information includes device connection information. The second electronic device needs to determine the complete device connection information to establish a connection with the first electronic device. This first information is divided into six data slices, designated as the first, second, third, fourth, fifth, and sixth data slices. The first electronic device encodes the first to fifth data slices into an image and the sixth data slice into an acoustic wave. The second electronic device decodes the image to obtain the first to fourth data slices and decodes the acoustic wave to obtain the sixth data slice. The second electronic device can determine that the fifth data slice is missing based on the frame number. If it does, it can send an indication message to the first electronic device to indicate that incomplete data information has not been obtained or that the fifth data slice is missing. Since the first and second electronic devices have not yet established a connection, this indication message can be a broadcast message. After receiving this indication message, the first electronic device can increase the number of data slices in the acoustic wave, such as by encoding the first to fifth data slices into the acoustic wave, or by separately encoding the fifth data slice into the acoustic wave and / or the image.

[0246] For example, the first information includes device connection information, which is a PIN code, specifically 656597. The second electronic device decodes the PIN code to 656595. Since the second electronic device cannot establish a connection with the first electronic device based on the decoded PIN code, indicating an error in the decoded PIN code, it can send an indication message to the first electronic device to indicate the decoding error. After receiving this indication message, the first electronic device can adjust the number of data slices on different channels.

[0247] In some embodiments, when the data information included in the first information is data information of the second type, the second electronic device may attempt to respond to the data information.

[0248] The second type of data information is the opposite of the first type of data information. That is, even if the data information is incomplete or contains decoding errors, the second electronic device can still respond to the incomplete or erroneous data information. Therefore, when the second electronic device determines that the data information included in the first information is of the second type, it can first attempt to respond to the data information. If the response fails, it can then send an indication message to the first electronic device to indicate that incomplete data information was not obtained or that a decoding error occurred, or to indicate which specific data slice was lost.

[0249] For example, the first information includes file identification information, which is divided into six data slices: first data slice, second data slice, third data slice, fourth data slice, fifth data slice, and sixth data slice. The first electronic device encodes the first to fifth data slices into an image and the sixth data slice into a sound wave. This file identification information is the storage path of the first file, whose filename is ABC.PDF and storage path is / storage / emulated / 0 / ABC.PDF. The second electronic device decodes the image to obtain the first to fourth data slices and decodes the sound wave to obtain the sixth data slice. The storage path determined by the second electronic device by splicing the first to fourth data slices is / storage / emulated / 0. The second electronic device can first request to send a file to the first electronic device based on / storage / emulated / 0. After receiving the request, the first electronic device determines that there is only one file, ABC.PDF, under the path / storage / emulated / 0, and can then send that file to the second electronic device.

[0250] For example, the first information includes file identification information, which is divided into six data slices: the first data slice, the second data slice, the third data slice, the fourth data slice, the fifth data slice, and the sixth data slice. The first electronic device encodes the first to fifth data slices into an image and the sixth data slice into a sound wave. This file identification information is the file identifier of the first file, which is AABBCCDDEEFF. The second electronic device decodes the image to obtain the first to fourth data slices and decodes the sound wave to obtain the sixth data slice. The file identifier determined by the second electronic device by splicing the first to fourth data slices is AABBCCDDEE. The second electronic device can first request to send a file to the first electronic device based on the file identifier AABBCCDDEE. After receiving the request from the second electronic device, the first electronic device determines that there is only one file matching the file identifier AABBCCDDEE, and then sends that file to the second electronic device.

[0251] For example, the first information includes file identification information, which is the file identifier of the first file, AABBCCDDEEFF. The file identifier decoded by the second electronic device is AABBCCDDEEFG. The second electronic device can first request to send a file to the first electronic device based on the file identifier AABBCCDDEEFG. After receiving the request from the second electronic device, the first electronic device can compare the similarity between the file identifiers of various files and the file identifier decoded by the second electronic device. When it is determined that the similarity between the file identifier of a certain file and the file identifier decoded by the second electronic device is greater than or equal to a fifth threshold, the first electronic device can send the file to the second electronic device. The fifth threshold can be understood as a threshold value for the first electronic device to send files. When the similarity between the file identifier of a certain file and the file identifier decoded by the second electronic device is greater than or equal to the fifth threshold, it indicates that the file identifier decoded by the second electronic device is likely the file identifier of that file.

[0252] The above section introduced the system architecture diagram and the method of multimodal encoding and decoding. The following section will introduce several scenarios applicable to the embodiments of this application, in conjunction with Figures 5-10.

[0253] Figure 5 shows a schematic diagram of a scenario provided by an embodiment of this application.

[0254] As shown in Figure 5, this scenario includes a vehicle 501 and a mobile phone 502. The vehicle 501 includes a display screen 501-1 (or a central control screen). When a user wants to project the interface content of the mobile phone 502 onto the display screen 501-1, the mobile phone 502 needs to establish a short-range wireless communication connection with the vehicle 501.

[0255] Vehicle 501 can encode part or all of the first information into one or more frames of images, and then display the one or more frames of images through display screen 501-1. In addition, vehicle 501 can also encode part or all of the first information into sound waves, and then play the sound waves through a speaker.

[0256] In some embodiments, the first information includes device connectivity information.

[0257] In some embodiments, the first information includes device connection information and verification information.

[0258] Mobile phone 502 can scan one or more frames of images displayed on display screen 501-1 through its camera and receive the sound waves through its microphone. It can determine the device connection information based on the scanned images and the sound waves received by the microphone, and then establish a connection with vehicle 501 based on the device connection information. In this way, the content of the interface can be projected onto display screen 501.

[0259] In some embodiments, the mobile phone 501 may encode device connection information into one or more frames of image and encode verification information into sound waves.

[0260] Understandably, when mobile phone 501 can encode device connection information into a frame of image, mobile phone 502 can decode that frame of image to determine the device connection information and decode the sound wave to determine the verification information. Mobile phone 502 can then verify the device connection information using the verification information.

[0261] It is also understandable that when mobile phone 501 can encode device connection information into multiple frames of images, mobile phone 502 can decode these multiple frames of images to determine multiple data slices, perform data frame alignment and splicing operations on these multiple data slices to determine the device connection information, and decode the sound waves to determine the verification information. Mobile phone 502 can verify the device connection information through the verification information.

[0262] It should be noted that Figure 5 only illustrates the example of vehicle 501 including a display screen, but this application embodiment does not specifically limit this. Vehicle 501 may also include more display screens. For example, vehicle 501 may also be equipped with a second-row left-side entertainment display screen, and vehicle 501 may also display one or more frames of images through the second-row left-side entertainment display screen.

[0263] Figure 6 shows another scenario diagram provided by an embodiment of this application.

[0264] As shown in Figures 6(a) and (b), the scenario includes tablet computer 601 and tablet computer 602. Tablet computer 601 can display presentation 603, specifically page 4 of presentation 603, which can also be referred to as PPT document 603. When displaying presentation 603, tablet computer 601 can encode part or all of the first information into the display interface corresponding to presentation 603, and also encode part or all of the first information into sound waves, which are then played through a speaker. The first information includes information about presentation 603, such as its file identifier, storage path, or file handle.

[0265] In some embodiments, the first information may further include verification information.

[0266] It is understandable that, in the example shown in Figure 6, in order to avoid affecting the user, the tablet computer 601 may implicitly encode part or all of the first information into the display interface corresponding to the presentation 603, as described above. This can be achieved by using methods such as brightness variation or digital watermark overlay to implicitly encode part or all of the first information into the display interface corresponding to each page of the presentation 603.

[0267] The tablet computer 602 can scan the display interface corresponding to the presentation 603 displayed on the tablet computer 601 using its camera, and receive sound waves played by the tablet computer 601 using its microphone to identify information about the presentation 603. Based on this information, the tablet computer 602 can then request the tablet computer 601 to send the presentation 603. After receiving the presentation 603 sent by the tablet computer 601, the tablet computer 602 can display the presentation 603.

[0268] In some embodiments, the tablet computer 601 may encode part or all of the first information into the presentation 603 and encode part or all of the first information into sound waves upon detecting that a trigger condition is met.

[0269] For example, when displaying presentation 603, tablet computer 601 can also simultaneously display a control for sharing presentation 603. The user's action could be clicking the control. In response to the user's click, tablet computer 601 encodes some or all of the first information into presentation 603, and also encodes some or all of the first information into sound waves.

[0270] For example, when the tablet computer 601 is displaying the presentation 603, it detects the user's voice command: "Hey Celia, share the content being displayed." In response to the user's voice command, it encodes part or all of the first information into the presentation 603 and encodes part or all of the first information into sound waves.

[0271] For example, tablet computer 601 and tablet computer 602 are connected. Tablet computer 601 and tablet computer 602 are placed on a table. The user taps the table. After tablet computer 601 detects the vibration, it can communicate with tablet computer 602 to determine whether the same waveform vibration is detected. When it is determined that the same waveform vibration is detected, part or all of the first information can be encoded into the display interface corresponding to the presentation 603, and part or all of the first information can be encoded into the audio.

[0272] It is understandable that in the example shown in Figure 6, tablet computer 601 and tablet computer 602 can be tablet computers that are already connected to a network. For example, tablet computer 601 and tablet computer 602 are connected via Bluetooth, or tablet computer 601 and tablet computer 602 are on the same Wi-Fi network, or tablet computer 601 and tablet computer 602 are logged into the same account, so that tablet computer 601 can send presentation 603 via Bluetooth or Wi-Fi.

[0273] It is also understandable that when tablet 601 is not networked with tablet 602, tablet 601 can encode the device connection information into the display interface and sound waves corresponding to presentation 603. This allows tablet 602 to network with tablet 601 before requesting to send presentation 603, which tablet 601 can then send via Bluetooth or Wi-Fi. Alternatively, after determining the information for presentation 603, tablet 602 can send a request to tablet 601 to establish a short-range wireless communication connection. Upon receiving this request, tablet 601 can display a control on its interface. When tablet 601 detects a user clicking the control, it responds to the request by establishing a short-range wireless communication connection with tablet 602 and then sending presentation 603 to tablet 602.

[0274] In some embodiments, the first information also includes progress information of the presentation 603. When the tablet computer 602 displays the presentation 603, it can directly jump to page 4 of the presentation 603 based on the progress information.

[0275] It should be noted that Figure 6 only uses a presentation as an example. Tablet 601 can also send file identifiers and file handles of data such as videos and images to tablet 602. Similarly, when tablet 601 sends a video file identifier, it can also send the video's playback progress information. Then, tablet 602 can obtain the video based on the video file identifier and play the video according to the playback progress of tablet 601.

[0276] Figure 7 shows another scenario diagram provided by an embodiment of this application.

[0277] As shown in Figures 7(a) and (b), the scenario includes tablet computer 701 and tablet computer 702. Tablet computer 701 can display webpage 703. When displaying webpage 703, tablet computer 701 can transmit part or all of the first information to the display interface corresponding to webpage 703, and encode part or all of the first information into a sound wave, and then play the sound wave through a speaker, wherein the first information includes the URL of webpage 703. Tablet computer 702 can scan the webpage 703 displayed by tablet computer 701 through a camera, and receive the sound wave played by tablet computer 701 through a microphone to determine the URL of webpage 703, and then tablet computer 702 can display webpage 703 according to the URL of webpage 703.

[0278] In some embodiments, the first information may further include verification information.

[0279] Similarly, the tablet computer 701 can encode the information of webpage 703 into webpage 703 and the information of webpage 703 into sound waves when the triggering conditions are met.

[0280] It is understandable that, in the example shown in Figure 7, since the amount of data in the URL of webpage 703 is small, tablet computer 701 can send the URL of webpage 703 to tablet computer 702 by encoding the URL of webpage 703 into webpage 703 and encoding the URL of webpage 703 into sound waves. Therefore, tablet computer 701 and tablet computer 702 can be tablet computers that have been networked or tablet computers that have not been networked.

[0281] Figure 8 shows another scenario diagram provided by an embodiment of this application.

[0282] As shown in Figure 8, the scenario includes mobile phone 801 and mobile phone 802, where mobile phone 801 is a new phone and mobile phone 802 is an old phone. When mobile phone 801 is powered on and the distance between mobile phone 801 and mobile phone 802 is less than a certain threshold, a ring-shaped image composed of particle dots, as shown in Figure 8, can be displayed. When displaying this image, mobile phone 801 has encoded part or all of the first information into the image, and mobile phone 801 can also encode part or all of the first information into a sound wave and play the sound wave through a speaker. The first information includes device connection information. Mobile phone 802 can scan the image and receive the sound wave through its microphone to obtain the device connection information, then establish a connection with mobile phone 801 based on the device connection information, and then transfer data to mobile phone 802 to complete the phone switching task.

[0283] In some embodiments, the first information may further include verification information.

[0284] Figure 9 shows another scenario diagram provided by an embodiment of this application.

[0285] As shown in Figures 9(a) and (b), the scenario includes mobile phones 901 and 902. Mobile phone 901 displays interface 903, which is the QR code login interface for the video application. Mobile phone 902 displays interface 904, which is the QR code scanning interface. Mobile phone 902 has logged into the video application using the first account. The encoded image displayed by mobile phone 901 on interface 903 can be generated by the server of the video application. This encoded image can be a traditional encoded image such as a QR code, or a non-traditional encoded image as shown in Figure 5. The server of the video application can generate authentication information, which can be an identity (ID). This authentication information is associated with the device ID of mobile phone 901. Some or all of the information in this authentication information can be carried in the encoded image of interface 903. Mobile phone 901 can also encode some or all of the information in this authentication information into sound waves. After scanning the encoded image of interface 903 and receiving the sound waves, mobile phone 902 can determine the authentication information and then communicate with the server of the video application to display interface 905. After mobile phone 902 detects that the user clicks on control 906 in interface 905, it can send an instruction to the video application server, instructing mobile phone 901 to log in to the video application using the first account. Upon receiving this instruction, the video application server can send a token to mobile phone 901. This token is associated with the ID of the encoded image, the first account, and the device information of mobile phone 901. After receiving the token, mobile phone 901 can log in to the video application using the token.

[0286] In other embodiments of this application, when the video application supports login using a social application account, mobile phone 902 has already logged into the social application using a second account. Mobile phone 902 can also authorize mobile phone 901 to log into the video application using a second account.

[0287] In the scenarios shown in Figures 5 to 9, the electronic device can encode the data to be transmitted into an image and into a sound wave. In other examples of this application, when the electronic device is an electronic device without a display screen but equipped with ambient lighting, such as a smart speaker, the electronic device can encode the data to be transmitted into different colors and / or into different brightness levels. This will be described below with reference to Figure 10.

[0288] Figure 10 shows another scenario diagram provided by an embodiment of this application.

[0289] As shown in Figure 10, this scenario includes a mobile phone 1001 and a smart speaker 1002. The smart speaker 1002 can encode some or all of the first information into light and into sound waves, then display the light through ambient lighting and play the sound waves through a speaker. The first information includes device connection information. The mobile phone 1001 can scan the smart speaker 1002 with its camera to identify the light and receive the sound waves through its microphone, thereby determining the device connection information based on the light and sound waves.

[0290] In some embodiments, the first information may further include verification information.

[0291] In some embodiments, the amount of information carried in the light is smaller than the amount of information carried in the sound waves.

[0292] In the example shown in Figure 10, the smart speaker 1002 can encode part or all of the first information into the light and part or all of the first information into the sound wave when the trigger condition is detected.

[0293] This application implements two methods, without specifying the triggering conditions corresponding to the smart speaker 1002.

[0294] For example, when the smart speaker 1002 detects that the distance between itself and the mobile phone 1001 is less than a threshold, it can encode part or all of the first information into the light and into the sound wave.

[0295] For example, the smart speaker 1002 can detect user-triggered connection operations (e.g., continuous pressing of the power button), and can encode part or all of the first information into the light, and encode part or all of the first information into the sound waves.

[0296] For example, when the smart speaker 1002 detects that it is being powered on for the first time, it can encode some or all of the first information into the light and into the sound waves.

[0297] The preceding text, with reference to Figures 5 to 10, exemplarily describes several scenarios applicable to the multimodal encoding and decoding methods provided in this application. However, the embodiments of this application are not limited thereto. The multimodal encoding and decoding methods provided in this application can be applied to any scenario requiring information transmission. For example, an applicable scenario could also be adding friends by scanning a QR code. The following will describe in detail the multimodal encoding and decoding methods provided in this application.

[0298] Figure 11 shows a schematic flowchart of a multimodal coding method provided in an embodiment of this application. The method is applied to a first electronic device. As shown in Figure 11, the method 1100 includes:

[0299] S1101, the first information is encoded into an optical signal and an acoustic signal, wherein the optical signal is either an image or a light.

[0300] The first electronic device can encode the first information into two channels: an optical signal and an acoustic signal. The optical signal can be an image or light, and the acoustic signal can be ultrasound or audible sound waves.

[0301] It should be understood that descriptions of encoding the first information into an image or light, and encoding the first information into ultrasonic waves or audible sound waves, for the first electronic device can be found above, and will not be repeated here for the sake of brevity.

[0302] In some embodiments, the first electronic device may divide the first information into a first part and a second part, and encode the first part into an optical signal and the second part into an acoustic signal, wherein the data amount of the first part is greater than or equal to the data amount of the second part.

[0303] In some embodiments, the size of the first information is greater than or equal to a first threshold, which is the maximum information carrying capacity of the optical signal.

[0304] In some embodiments, the first part and the second part contain at least partially identical information.

[0305] In some embodiments, the first information includes device connectivity information.

[0306] In some embodiments, the first information includes file identification information.

[0307] In some embodiments, the first information includes the file to be sent itself.

[0308] In some embodiments, the first information includes the URL of a first webpage.

[0309] It should be understood that a detailed description of the first piece of information can be found in the description of the first piece of information above, and for the sake of brevity, it will not be repeated here.

[0310] In some embodiments, the first information includes data information and verification information, and the second part includes only the verification information in the first information.

[0311] In some embodiments, when the optical signal is an image, the image may include multiple frames that encode the first portion.

[0312] In some embodiments, when the optical signal is an image, the image may be a traditional coded pattern such as a barcode or a two-dimensional barcode.

[0313] In some embodiments, when the light signal is an image, the image may be an image from an animation effect. The first electronic device can hide and encode the first information into the image, making it invisible or substantially invisible to the user. Compared to the first electronic device directly displaying a barcode or two-dimensional barcode, hiding and encoding the first information into the image enhances the user experience by making it imperceptible to the user.

[0314] S1102, display the light signal and play the sound signal.

[0315] After the first electronic device encodes the first information into an optical signal and an acoustic signal, it can display the optical signal through a display device and play the acoustic signal through a playback device.

[0316] For example, the display device of the first electronic device includes, but is not limited to, a display screen and an ambient light, and the playback device includes, but is not limited to, a speaker.

[0317] In this embodiment of the application, the first electronic device can encode the first information to be transmitted into optical signals and acoustic signals, and transmit the first information through multiple signals, which can effectively avoid the blocking and interference of a single channel.

[0318] In some embodiments, the first electronic device divides the first information into a first part and a second part, encodes the first part of the first information into an optical signal, and encodes the second part of the first information into an acoustic signal. The data size of the first part is greater than or equal to the data size of the second part. Simultaneously, the data size of the first part of the first information will not exceed the maximum data capacity of the optical signal, and the data size of the second part of the first information will not exceed the maximum data capacity of the acoustic signal.

[0319] In some embodiments, the method 1100 further includes:

[0320] When the duration of displaying the light signal and playing the sound wave signal is greater than or equal to a first duration and no response to the first information is received within the first duration, the first part is encoded into the sound wave signal and the second part is encoded into the light signal, wherein the first duration is the longest duration for the first electronic device to wait for the response to receive the first information.

[0321] If the duration for which the first electronic device displays the light signal and plays the sound signal is greater than or equal to a first duration, and the first electronic device does not receive a response from the first information within the first duration, it indicates that the second electronic device may not have yet acquired the first information. The reason the second electronic device may not have received the first information could be due to excessively strong ambient light or moiré patterns generated when the second electronic device scans visible light. Therefore, the first electronic device can encode the first part into the sound signal and the second part into the light signal.

[0322] In this embodiment, when the first electronic device displays light signals and plays sound signals for a long time and it is determined that the second electronic device has not obtained the first information, the encoding strategy can be flexibly adjusted, which can improve the success rate of the second electronic device finally obtaining the first information and help improve the user experience.

[0323] In some embodiments, the first information includes file identification information of the first file, and the method 1100 further includes:

[0324] Receive a first request, which is a request generated by the second electronic device based on the first information, and the first electronic device and the second electronic device have established a communication connection.

[0325] In response to the first request, the first file is sent to the second electronic device.

[0326] After the second electronic device decodes the optical signal and the sound wave signal, it can determine that the first information includes file identification information. Therefore, it can request the first electronic device to send the first file corresponding to the first information based on the first information. After receiving the request from the second electronic device, the first electronic device can send the first file to the second electronic device.

[0327] In some embodiments, the first information includes file identification information of the first file, and the first electronic device and the second electronic device have not established a communication connection. The method 1100 further includes:

[0328] Receive a second request, which is a request generated by the second electronic device based on the first information to establish a communication connection with the first electronic device;

[0329] In response to the second request, a communication connection is established with the second electronic device and the first file is sent to the second electronic device.

[0330] After the second electronic device decodes the optical signal and the acoustic signal, it can determine that the first information includes file identification information. Since a communication connection has not yet been established with the first electronic device, the second electronic device can request to establish a communication connection with the first electronic device and send the first file corresponding to the first information based on the first information. After receiving the request from the second electronic device, the first electronic device can establish a communication connection with the second electronic device and send the first file.

[0331] In some embodiments, in response to a first request, sending a first file to a second electronic device includes:

[0332] Determine the first identification information based on the first request information;

[0333] The first file is determined based on the first identification information, wherein the first identification information is at least partially the same as the file identification information of the first file;

[0334] The first file is sent to the second electronic device.

[0335] When the second electronic device decodes optical and acoustic signals, the decoded information may be incomplete or contain partial errors. When the second electronic device sends a first request to the first electronic device, the first request includes first identification information decoded by the second electronic device. This first identification information is at least partially identical to the file identification information of the first file. After determining the first identification information based on this first identification information, the first electronic device can search for the file based on it. When searching for a file, if the first identification information has a high similarity to the file identification information of the first file, the first electronic device can determine that the first file is the file that needs to be sent to the second electronic device. Therefore, the first electronic device can send the first file to the second electronic device.

[0336] In some embodiments, the first information includes authentication information of the first application, and the method 1100 further includes: receiving a first token sent by the server of the first application, wherein the first token is generated by the server of the first application authorized by the second electronic device based on the first information; and logging into the first application based on the first token.

[0337] In some embodiments, the method 1100 further includes:

[0338] When the triggering condition is met, the first electronic device encodes all the information of the first information into a sound wave.

[0339] The first electronic device can first encode the first information into the optical wave signal and the sound wave signal. When the first electronic device determines that the triggering condition is met, it can adjust the encoding strategy and no longer encode the first information into the optical wave signal, but encode all the information of the first information into the sound wave signal.

[0340] For example, the trigger condition is detecting that the display is off.

[0341] The first electronic device can first encode the first information into light wave signals and sound wave signals. When the display screen is detected to be off, since the light wave can no longer be displayed on the display screen, the first electronic device can adjust to encode all the information of the first information into the sound wave.

[0342] Figure 12 shows a schematic flowchart of a multimodal decoding method provided in an embodiment of this application. The method is applied to a second electronic device. As shown in Figure 12, the method 1200 includes:

[0343] S1201, acquire optical signal and sound wave signal, wherein the optical signal is either an image or a light source.

[0344] After the first electronic device displays the light signal and the sound wave signal, the second electronic device can acquire the light signal and the sound wave signal through the receiving device.

[0345] For example, the receiving device of the second electronic device includes, but is not limited to, a camera, a microphone, etc.

[0346] S1202, determine the first information based on the optical signal and the acoustic signal.

[0347] After acquiring the optical signal and the acoustic signal, the second electronic device can decode the information carried by the optical signal and the acoustic signal, and determine the first information based on the decoded information.

[0348] It should be understood that a detailed description of the decoding of optical and acoustic signals by the second electronic device can be found above, and will not be repeated here for the sake of brevity.

[0349] In some embodiments, the first information includes a first part and a second part, the first part being carried by an optical signal and the second part being carried by an acoustic signal, wherein the data amount of the first part is greater than or equal to the data amount of the second part, and determining the first information based on the optical signal and the acoustic signal includes:

[0350] The first part and the second part are determined based on the optical signal and the sound wave signal, respectively;

[0351] The first part and the second part are frame aligned according to the frame header of the first part and the frame header of the second part.

[0352] The first and second parts of the spliced ​​frames are aligned to determine the first information.

[0353] The second electronic device can determine the first part of the first information by decoding the optical signal and the second part of the first information by decoding the acoustic signal. The first part and the second part can be composed of one or more data frames. The second electronic device can perform frame alignment of the data frames of the first part and the second part according to the frame header of the data frames, and then splice the aligned first part and the second part to determine the first information.

[0354] In some embodiments, the first part and the second part contain at least partially identical information, and the method 1200 further includes:

[0355] Eliminate redundant information in information that is at least partially identical.

[0356] In some embodiments, the second part includes only the verification information in the first information, and the method 1200 further includes:

[0357] Verify the information in the first part based on this verification information.

[0358] In some embodiments, the size of the first information is greater than or equal to a first threshold, which is the maximum information carrying capacity of the optical signal.

[0359] In some embodiments, when the optical signal is an image, the image may include multiple frames that encode the first portion.

[0360] In this embodiment of the application, the first electronic device can encode the first information to be sent into optical signals and acoustic signals. After acquiring the optical signals and acoustic signals, the second electronic device can decode the optical signals and acoustic signals to determine the first information, thus avoiding the blocking and interference of a single channel.

[0361] In some embodiments, the first information includes device connection information, and the method 1200 further includes:

[0362] A communication connection is established with the first electronic device based on the first information.

[0363] In some embodiments, the method 1200 further includes:

[0364] After establishing a communication connection with the first electronic device, the device sends first data to the first electronic device. This first data includes, but is not limited to, account information logged in on the second electronic device, wireless network information historically connected to the second electronic device, and application data on the second electronic device.

[0365] In some embodiments, the first information includes device connection information. When the second electronic device fails to establish a connection with the first electronic device based on the first information, or when it is determined that the first information is incomplete, the method 1200 further includes:

[0366] Send a first indication message to the first electronic device. The first indication message is used to indicate that the second electronic device has not obtained complete information or that the second electronic device has decoded incorrectly.

[0367] In some embodiments, the first information includes file identification information of a first file, and a communication connection has been established between the first electronic device and the second electronic device. The method 1200 further includes:

[0368] Generate and send a first request to the first electronic device based on the first information, the first request being used to request the first electronic device to send a first file; and receive the first file sent by the first electronic device.

[0369] In some embodiments, the first information includes file identification information of the first file, and the first electronic device and the second electronic device have not yet established a communication connection. The method 1200 further includes:

[0370] Generate and send a second request to the first electronic device based on the first information, the second request being used to request the establishment of a communication connection with the first electronic device; and receive the first file sent by the first electronic device.

[0371] In some embodiments, the first information includes the URL of a first webpage, and the method 1200 further includes: displaying the first webpage according to the first information.

[0372] In some embodiments, the first information includes authentication information of the first application, and the method 1200 further includes: displaying a first interface based on the first information, the first interface being an authorization login interface; and in response to the user's confirmation of authorization login, sending instruction information to the server of the first application, the instruction information being used to instruct the authorized first electronic device to log in to the first application using a first account.

[0373] The multimodal encoding and decoding method provided in the embodiments of this application has been described in detail above. In the various embodiments of this application, unless otherwise specified or logically conflicting, the terminology and / or descriptions between the various embodiments are consistent and can be referenced by each other. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0374] The foregoing primarily describes a multimodal encoding / decoding method provided in this application from the perspective of an electronic device. It is understood that, in order to achieve the above functions, the electronic device includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0375] This application provides a method for multimodal information transmission. The method is applied to a system including a first electronic device and a second electronic device. The first electronic device is used to execute the multimodal encoding technical solution shown in FIG11, and the second electronic device is used to execute the multimodal decoding technical solution shown in FIG12.

[0376] This application provides an information transmission system, which includes a first electronic device and a second electronic device. The first electronic device is used to execute the multimodal coding technology shown in FIG11, and the second electronic device is used to execute the multimodal decoding technology shown in FIG12.

[0377] This application provides a computer program product that, when run on an electronic device, causes the electronic device to execute the technical solutions described in the above embodiments. Its implementation principle and technical effects are similar to those of the related embodiments described above, and will not be repeated here.

[0378] This application provides a readable storage medium containing instructions that, when executed by an electronic device, cause the electronic device to perform the technical solution described in the above embodiments. The implementation principle and technical effects are similar and will not be repeated here.

[0379] This application provides a chip for executing instructions. When the chip is running, it executes the technical solutions described in the above embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0380] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0381] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0382] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0383] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0384] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0385] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially, or the parts that contribute to the prior art, or parts of the technical solutions, can be embodied in the form of software products. These computer software products are stored in a storage medium and include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0386] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be included within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.

Claims

1. A multimodal coding method, characterized in that, The method is applied to a first electronic device, and the method includes: The first information is encoded into an optical signal and an acoustic signal, wherein the optical signal is either an image or a light source; Display the light signal and play the sound signal.

2. The method according to claim 1, characterized in that, The process of encoding the first information into optical and acoustic signals includes: The first information is divided into a first part and a second part, wherein the amount of data in the first part is greater than or equal to the amount of data in the second part; The first part is encoded into the optical signal, and the second part is encoded into the acoustic signal.

3. The method according to claim 2, characterized in that, The first part and the second part contain at least some of the same information.

4. The method according to claim 2 or 3, characterized in that, The size of the first information is greater than or equal to a first threshold, where the first threshold is the maximum information carrying capacity of the optical signal.

5. The method according to any one of claims 2 to 4, characterized in that, When the duration of displaying the light signal and playing the sound signal is greater than or equal to a first duration and no response to the first information is received within the first duration, the first part is encoded into the sound signal and the second part is encoded into the light signal, wherein the first duration is the longest duration for the first electronic device to wait for a response to receive the first information.

6. The method according to any one of claims 2 to 5, characterized in that, The second part includes only the verification information from the first information.

7. The method according to any one of claims 1 to 6, characterized in that, When the optical signal is the image, the image includes multiple frames, and the multiple frames encode the first part.

8. The method according to any one of claims 1 to 7, characterized in that, The first information includes the file identifier information of the first file, and the method further includes: Receive a first request, wherein the first request is a request generated by the second electronic device based on the first information, and the first electronic device and the second electronic device have established a communication connection; In response to the first request, the first file is sent to the second electronic device.

9. The method according to claim 8, characterized in that, The step of sending the first file to the second electronic device in response to the first request includes: Determine the first identification information according to the first request; The first file is determined based on the first identification information, wherein the first identification information is at least partially the same as the file identification information of the first file. The first file is sent to the second electronic device.

10. The method according to claim 8 or 9, characterized in that, The first information also includes the progress information of the first file.

11. A method for multimodal decoding, characterized in that, The method is applied to a second electronic device, and the method includes: Acquire optical signals and sound wave signals, wherein the optical signal is either an image or a light source; The first information is determined based on the optical signal and the acoustic signal.

12. The method according to claim 11, characterized in that, The first information includes a first part and a second part, wherein the first part is carried by the optical signal and the second part is carried by the acoustic signal, and the data amount of the first part is greater than or equal to the data amount of the second part. Determining the first information based on the optical signal and the acoustic signal includes: The first part and the second part are determined based on the optical signal and the acoustic signal, respectively. The first part and the second part are frame aligned according to the frame header of the first part and the frame header of the second part; The first and second portions of the spliced ​​frames are aligned to determine the first information.

13. The method according to claim 12, characterized in that, The first part and the second part contain at least partially identical information, and the method further includes: Eliminate redundant information in the at least partially identical information.

14. The method according to any one of claims 12 to 13, characterized in that, The second part includes only the verification information in the first information, and the method further includes: The information in the first part is verified based on the verification information.

15. The method according to any one of claims 12 to 14, characterized in that, The size of the first information is greater than or equal to a first threshold, where the first threshold is the maximum information carrying capacity of the optical signal.

16. The method according to any one of claims 11 to 15, characterized in that, When the optical signal is the image, the image includes multiple frames, and the multiple frames encode the first part.

17. The method according to any one of claims 11 to 16, characterized in that, The first information includes device connection information. When the second electronic device fails to establish a connection with the first electronic device based on the first information, or determines that the first information is incomplete, the method further includes: Send a first indication message to the first electronic device, the first indication message being used to indicate that the second electronic device has not obtained complete information, or that the second electronic device has a decoding error.

18. The method according to any one of claims 11 to 16, characterized in that, The first information includes the file identifier information of the first file, and the method further includes: A first request is generated and sent to the first electronic device based on the first information. The first request is used to request the first electronic device to send the first file, wherein the first electronic device and the second electronic device have established a communication connection. Receive the first file sent by the first electronic device.

19. A method for multimodal information transmission, characterized in that, The method is applied to a system including a first electronic device and a second electronic device, and the method includes: The first electronic device encodes the first information into an optical signal and an acoustic signal, wherein the optical signal is either an image or a light source; The first electronic device displays the optical signal and plays the sound wave signal; The second electronic device acquires the optical signal and the acoustic signal; The second electronic device determines the first information based on the optical signal and the acoustic signal.

20. The method according to claim 19, characterized in that, The first information includes the file identifier information of the first file, and the method further includes: The second electronic device generates and sends a first request to the first electronic device based on the first information. The first request is used to request the first electronic device to send the first file, wherein the first electronic device and the second electronic device have established a connection. The first electronic device receives the first request; In response to the first request, the first electronic device sends the first file to the second electronic device; The second electronic device receives the first file.

21. An electronic device, characterized in that, It includes one or more processors; one or more memories; said one or more memories storing one or more computer programs, said one or more computer programs including instructions that, when executed by said one or more processors, cause the method of any one of claims 1 to 10 or the method of any one of claims 11 to 18 to be performed.

22. An information transmission system, comprising a first electronic device and a second electronic device, characterized in that, The first electronic device performs the method as described in any one of claims 1 to 10, and the second electronic device performs the method as described in any one of claims 11 to 18.

23. A chip, characterized in that, The chip includes a processor and a communication interface, the communication interface being used to receive signals and transmit the signals to the processor, the processor processing the signals such that the method as claimed in any one of claims 1 to 10 or the method as claimed in any one of claims 11 to 18 is executed.

24. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on a computer, cause the method as claimed in any one of claims 1 to 10 or the method as claimed in any one of claims 11 to 18 to be performed.

25. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as claimed in any one of claims 1 to 10 or the method as claimed in any one of claims 11 to 18.

Images