A method for processing a screen mirroring picture and a related device

Abstract

Provided are a method for processing a projection picture and related devices. The method comprises: determining a target device for calculating black border data from a first device and a second device based on a computing performance parameter of the first device and a received computing performance parameter of the second device, the second device being configured to display a projection picture from the first device, the computing performance parameter of the first device being configured to indicate a computing capability of the first device, and the computing performance parameter of the second device being configured to indicate a computing capability of the second device; in a case where the target device is the first device, the first device calculates the black border data based on a first projection picture, the first projection picture being the projection picture to be displayed by the second device, and the black border data being configured to indicate a black border area in the first projection picture; and the first device sends the first projection picture and the black border data to the second device. In this way, the detection time can be shortened, the detection efficiency can be improved, and the user experience can be improved.

Description

Technical Field

[0001] This application relates to the field of terminals, and in particular to a method and related apparatus for processing projected images. Background Technology

[0002] With the development of internet technology, the terminal devices used for playing videos in homes are becoming increasingly diverse, including but not limited to televisions, mobile phones, computers, and tablets. However, users face limitations when watching videos on smaller-screen devices like mobile phones. Therefore, screen mirroring technology is often used to project videos onto larger screens like televisions for viewing. For example, screen mirroring can be used to project videos from a mobile phone onto a television for playback, thus improving the user's viewing experience.

[0003] During screen mirroring, black borders may appear on the projected image, affecting the user's viewing experience. Therefore, large screens need to detect black borders, obtain black border data, and crop the black border areas to maximize the display of the projected image horizontally or vertically. However, in some cases, the black border detection process may be time-consuming and inefficient, resulting in a long wait for the projected image to display, leading to a poor user experience. Summary of the Invention

[0004] This application provides a method and related apparatus for processing projected images, aiming to shorten the black border detection time, improve the efficiency of black border detection, and thus improve the user experience.

[0005] The first aspect provides a method for processing the projected screen. This method can be executed by a first device, or by a component (such as a chip, chip system, etc.) configured in the first device, or by a logic module or software capable of implementing all or part of the functions of the first device. This application does not limit this method.

[0006] For example, the method includes: determining a target device for calculating black border data between the first device and the second device based on the computing performance parameters of the first device and the received computing performance parameters of the second device, wherein the second device is used to project a screen image from the first device, the computing performance parameters of the first device are used to indicate the computing power of the first device, and the computing performance parameters of the second device are used to indicate the computing power of the second device; if the target device is the first device, calculating black border data based on the first projected screen image, wherein the first projected screen image is the projected screen image to be displayed by the second device, and the black border data is used to indicate the black border area in the first projected screen image; and sending the first projected screen image and the black border data to the second device.

[0007] Based on the above technical solution, the first and second devices determine the target device for calculating black border data based on computational performance parameters, ensuring that the selected target device has relatively high computing power. This helps to shorten the calculation time for black border data and improve the efficiency of black border detection. For example, if the target device is the first device, the first device calculates the black border data based on the first projected screen and sends it to the second device for cropping. Compared to the second device directly performing black border detection on the first projected screen to obtain the black border data, this method shortens the calculation time, improves detection efficiency, and ultimately enhances the user experience.

[0008] In conjunction with the first aspect, in some possible implementations of the first aspect, sending the first projection screen and black border data to the second device includes: sending the first projection screen and black border data to the second device via a transport stream.

[0009] A black border data field can be added to the transport stream to store black border data. In other words, the transport stream can include the first screen projection image and the corresponding black border data. In this way, the second device can directly receive the first screen projection image and its corresponding black border data without having to perform black border detection on the first screen projection image. This greatly shortens the time it takes for the image to be displayed after being projected, and reduces the lag rate between the screen projection image displayed by the second device and the image displayed by the first device. In other words, it helps to improve the synchronization rate of the images displayed by the first and second devices, thereby improving the user experience.

[0010] In conjunction with the first aspect, in some possible implementations of the first aspect, the first projected image is obtained based on the display image of the first device; before calculating the black border data based on the first projected image, the above method further includes: performing black border detection on the display image to obtain a black border detection result, the black border detection result being used to indicate the black border area in the display image; and calculating the black border data based on the first projected image, including: calculating the black border data based on the black border detection result and the display ratio or resolution of the first projected image.

[0011] The first projected image is the image to be displayed on the second device. Due to the different resolutions of the first and second devices, a black border is added to the first device's display to ensure the aspect ratio of the image after the black border is added matches that of the second device. In this application, the first device can perform black border detection on its display before adjusting its resolution, obtain the black border detection result, and then combine it with the resolution of the second device to calculate the entire black border area of ​​the first projected image. In this way, the first device detects the display before the black border is added; that is, it only detects the black border area of ​​the first device's display itself, greatly reducing the detection area and improving detection efficiency.

[0012] In conjunction with the first aspect, in some possible implementations of the first aspect, before determining the target device for calculating black border data among the first device and the second device based on the computing performance parameters of the first device and the received computing performance parameters of the second device, the above method further includes: receiving first information from the second device, the first information indicating whether the second device has performed black border detection; and determining, based on the first information, that the second device has not performed black border detection.

[0013] The first device determines whether the second device has performed black border detection based on the received first information. In this way, when determining the target device, the computing power of the second device can be comprehensively considered. For example, if the second device has performed black border detection, the first device can directly perform black border detection, which helps to reduce the computing load of the second device and improve the detection efficiency. If the second device has not performed black border detection, the computing capabilities of the first and second devices can be further evaluated. This allows the device with better computing power to be selected for detection, which helps to improve the detection efficiency.

[0014] In conjunction with the first aspect, in some possible implementations of the first aspect, determining a target device for calculating black border data between the first device and the second device based on the computing performance parameters of the first device and the received computing performance parameters of the second device includes: if the computing power of the first device is higher than that of the second device, determining the first device as the target device; or if the computing power of the second device is higher than that of the first device, determining the second device as the target device.

[0015] If the second device does not perform black border detection, the computing power of the first and second devices can be further evaluated. Selecting the device with better computing power for detection is beneficial to improving detection efficiency and enabling the screen to be displayed faster after being projected. This also improves the synchronization rate of the screens displayed by the first and second devices, thereby enhancing the user experience.

[0016] In conjunction with the first aspect, in some possible implementations of the first aspect, when the target device is a second device, the above method further includes: sending the first projection screen to the second device without sending black border data.

[0017] When the target device is the second device, the first screen projection image is sent to the second device without sending black border data. That is, the second device performs black border detection on the first screen projection image. In this way, by comprehensively considering the computing power of the first device and the computing power of the second device, the second device with higher computing power is finally selected to perform black border detection, which can improve detection efficiency and thus improve user experience.

[0018] The second aspect provides a method for processing the projected screen. This method can be executed by a second device, or by a component (such as a chip, chip system, etc.) configured in the second device, or by a logic module or software capable of implementing all or part of the functions of the second device. This application does not limit this method.

[0019] For example, the method includes: determining a target device for calculating black border data between the first device and the second device based on the computing performance parameters of the second device and the received computing performance parameters of the first device; the second device is used to project a screen image from the first device; the computing performance parameters of the first device are used to indicate the computing power of the first device; and the computing performance parameters of the second device are used to indicate the computing power of the second device. If the target device is the first device, receiving a first screen image and black border data from the first device, wherein the black border data is calculated by the first device based on the first screen image, and the black border data is used to indicate the black border area in the first screen image, and the first screen image is the screen image to be displayed by the second device.

[0020] Based on the above technical solution, the first and second devices determine the target device for calculating black border data based on computational performance parameters, ensuring that the selected target device has relatively high computing power. This helps to shorten the calculation time for black border data and improve the efficiency of black border detection. For example, if the target device is the first device, the first device calculates the black border data based on the first projected screen and sends it to the second device. Correspondingly, the second device receives the first projected screen and the black border data so that it can perform cropping. Compared to the second device directly performing black border detection on the first projected screen to obtain the black border data, this method shortens the time, improves detection efficiency, and allows the screen to be displayed quickly after being projected. This also improves the synchronization rate of the screens displayed by the first and second devices, thereby improving the user experience.

[0021] In conjunction with the second aspect, in some possible implementations of the second aspect, receiving the first projected screen and black border data from the first device includes: receiving the first projected screen and black border data via a transport stream.

[0022] A black border data field can be added to the transport stream to store black border data. In other words, the transport stream can include the first screen projection image and the corresponding black border data. In this way, the second device can directly receive the first screen projection image and its corresponding black border data without having to perform black border detection on the first screen projection image. This greatly shortens the time it takes for the image to be displayed after being projected, thereby improving the synchronization rate of the images displayed by the first and second devices and improving the user experience.

[0023] In conjunction with the second aspect, in some possible implementations of the second aspect, the above method further includes: when the display screen of the second device includes multiple projected screens, determining whether the second device has performed black border detection; and determining a target device for calculating black border data between the first device and the second device based on the computing performance parameters of the second device and the received computing performance parameters of the first device, including: when it is determined that the second device has not performed black border detection, determining a target device for calculating black border data between the first device and the second device based on the computing performance parameters of the second device and the received computing performance parameters of the first device.

[0024] When the display screen of the second device includes multiple projected images, the second device can determine whether it has performed black border detection. If it has not performed black border detection, based on the computing performance parameters of the second device and the received computing performance parameters of the first device, it determines the target device for calculating black border data between the first and second devices. It is understood that whether the second device has performed black border detection affects its detection efficiency. For example, if the second device has already performed black border detection, performing black border detection again may result in lower detection efficiency. Therefore, in this application, when determining the target device, whether the second device has performed black border detection is comprehensively considered. For example, if the second device has performed black border detection, the first device can directly perform black border detection, which helps reduce the computing load on the second device and improves detection efficiency. If the second device has not performed black border detection, the computing capabilities of the first and second devices can be further evaluated, allowing the device with better computing capabilities to be selected for detection, which helps improve detection efficiency.

[0025] In conjunction with the second aspect, in some possible implementations of the second aspect, the above method further includes: sending first information to the first device, the first information being used to indicate whether the second device has performed black border detection.

[0026] In conjunction with the second aspect, in some possible implementations of the second aspect, based on the computing performance parameters of the second device and the received computing performance parameters of the first device, a target device for computing black border data is determined between the first device and the second device, including: if the computing power of the first device is higher than that of the second device, the first device is determined as the target device; or if the computing power of the second device is higher than that of the first device, the second device is determined as the target device.

[0027] Between the first and second devices, the device with higher computing power is selected as the target device, and the black border detection is performed by the target device. This helps to shorten the detection time, improve the detection efficiency, and thus improve the user experience.

[0028] In conjunction with the second aspect, in some possible implementations of the second aspect, when the target device is the second device, the above method further includes: receiving a first projection screen from the first device; performing black border detection on the first projection screen to obtain black border data.

[0029] When the target device is the second device, the first screen projection image from the first device is received, but the black border data is not received. That is, the second device performs black border detection on the first screen projection image. In this way, by comprehensively considering the computing power of the first device and the computing power of the second device, the second device with higher computing power is finally selected to perform black border detection, which can improve detection efficiency and thus improve user experience.

[0030] In conjunction with the second aspect, in some possible implementations of the second aspect, the above method further includes: cropping the first projection screen based on the black border data to obtain and display the cropped projection screen.

[0031] A third aspect provides a screen projection processing apparatus capable of implementing the methods of the first aspect and any possible implementation thereof, or implementing the methods of the second aspect and any possible implementation thereof. The apparatus includes corresponding modules for performing the aforementioned methods. These modules can be implemented in software and / or hardware.

[0032] A fourth aspect provides a screen projection processing apparatus, which includes a processor. The processor is coupled to a memory and can be used to execute a computer program in the memory to implement the methods of the first aspect and any possible implementation thereof, or to implement the methods of the second aspect and any possible implementation thereof.

[0033] Optionally, the device also includes a memory.

[0034] Optionally, the device also includes a communication interface, to which the processor is coupled.

[0035] The fifth aspect provides a computer-readable storage medium storing a computer program or instructions that, when executed, implement the methods of the first aspect and any possible implementation thereof, or implement the methods of the second aspect and any possible implementation thereof.

[0036] The sixth aspect provides a computer program product including instructions that, when executed, implement the methods of the first aspect and any possible implementation thereof, or implement the methods of the second aspect and any possible implementation thereof.

[0037] A seventh aspect provides a chip system including at least one processor for supporting the implementation of the functions involved in the first aspect and any possible implementation of the first aspect, or for supporting the implementation of the functions involved in the second aspect and any possible implementation of the second aspect, such as receiving or processing data involved in the above methods.

[0038] In one possible design, the chip system also includes a memory for storing program instructions and data, which may be located within or outside the processor.

[0039] The chip system can consist of chips or include chips and other discrete components.

[0040] It should be understood that the third to seventh aspects of the embodiments of this application correspond to the technical solutions of the first and second aspects of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be described again. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0042] Figure 2 This is a schematic diagram of the system architecture applicable to the methods provided in the embodiments of this application;

[0043] Figure 3 This is a schematic flowchart of an existing screen mirroring process provided in the embodiments of this application;

[0044] Figure 4 This is a schematic diagram of the module relationship between devices provided in the embodiments of this application;

[0045] Figure 5 This is a schematic flowchart illustrating a method for processing screen projection provided in an embodiment of this application;

[0046] Figure 6 This is a schematic diagram of a single-channel screen projection method provided in an embodiment of this application;

[0047] Figure 7 This is a schematic diagram of the dual-channel projection method provided in the embodiments of this application;

[0048] Figure 8 This is a flowchart illustrating the screen mirroring process provided in an embodiment of this application. Detailed Implementation

[0049] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0050] The methods provided in this application can be applied to electronic devices such as mobile phones, televisions, tablets, smartwatches, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, personal computers (PCs), ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and distributed devices. This application does not limit the specific type of electronic device.

[0051] Wearable devices, also known as wearable smart devices, are a general term for devices that utilize wearable technology to intelligently design and develop everyday wearables, such as smartwatches and smart bracelets. Wearable devices are portable devices worn directly on the body or integrated into the user's clothing or accessories. Wearable devices are not merely hardware devices; they achieve powerful functions through software support, data interaction, and cloud interaction. Broadly defined, wearable smart devices include those with comprehensive functions, large sizes, and the ability to perform complete or partial functions without relying on a smartphone, such as smartwatches or smart glasses. They also include devices focused on a specific application function that require the use of other devices, such as smart bracelets and smart jewelry for vital sign monitoring.

[0052] Furthermore, the methods provided in this application embodiment can support operating environments such as Linux operating system, Android operating system (Android OS), Harmony OS, Mac operating system, iOS, Windows, and lightweight operating systems (such as LiteOS). This application embodiment does not impose any limitations on these environments.

[0053] For example, Figure 1 A schematic diagram of the structure of the electronic device 100 is shown. For example... Figure 1As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0054] Processor 110 may include one or more processing units, such as application processor (AP), microcontroller unit (MCU), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0055] The application processor outputs sound signals through the audio module 170 (such as the speaker 170A) or displays images or videos through the display screen 194.

[0056] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0057] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0058] The processor 110 can execute different operations by executing instructions to achieve different functions. These instructions may be, for example, instructions pre-stored in the memory before the device leaves the factory, or instructions read from the application (APP) after the user installs a new application during use. This application embodiment does not limit them in any way.

[0059] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a secure digital input and output (SDIO) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a universal synchronous asynchronous receiver / transmitter (USART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0060] It is understood that the interface connection relationships between the modules illustrated in this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0061] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0062] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, external memory, display screen 194, camera 193, and wireless communication module 160. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0063] The wireless communication function of electronic device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor.

[0064] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0065] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0066] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0067] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs), such as wireless fidelity (Wi-Fi), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0068] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with networks and other devices through wireless communication technology.

[0069] Electronic device 100 can implement display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0070] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device 100 may include one or more display screens 194.

[0071] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0072] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0073] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP (Internet Service Provider) for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP (Digital Signal Processor) for processing. The DSP converts the digital image signal into image signals in standard formats such as RGB and YUV. In some embodiments, the electronic device 100 may include one or more cameras 193.

[0074] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0075] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats.

[0076] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0077] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, etc.

[0078] Electronic device 100 can implement audio functions through audio module 170, such as speaker 170A, receiver 170B, microphone 170C, and headphone jack 170D, as well as application processor, such as music playback and recording.

[0079] Audio module 170 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. Audio module 170 can also be used for encoding and decoding audio signals.

[0080] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0081] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0082] The microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals.

[0083] The 170D headphone jack is used to connect wired headphones.

[0084] Button 190 includes the power button, volume buttons, etc.

[0085] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback.

[0086] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0087] The SIM card interface 195 is used to connect a SIM card. The electronic device 100 interacts with the network through the SIM card to achieve functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from it.

[0088] It should be understood that the structure illustrated in this application does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0089] It should also be understood that the first device, second device, etc., involved in the embodiments of this application can all be devices with... Figure 1 The electronic device shown in the illustration, or an electronic device with more or fewer components than those shown, is not limited to any particular type of electronic device in this application.

[0090] Figure 2 This is a schematic diagram of the architecture of a system 200 applicable to the methods provided in the embodiments of this application. For example... Figure 2 As shown, the system 200 includes an electronic device 210 and an electronic device 220. The electronic device 220 is used to project a screen display from the electronic device 210. The electronic device 210 can be considered the source, and the electronic device 220 can be considered the receiver. The electronic devices 210 and 220 can be different types of electronic devices or the same type of electronic devices. For example, the electronic device 210 can be a mobile phone or other electronic devices not shown in the figure, such as a tablet; the electronic device 220 can be a television or other devices not shown in the figure. This application embodiment does not limit the specific types of the electronic devices 210 and 220. The electronic device 210 can project its own displayed screen to the electronic device 220 to improve the viewing experience. For example, the electronic device 210 can transmit its currently playing video to the electronic device 220 through a communication channel established with the electronic device.

[0091] It should be understood that Figure 2 The example shown is merely an illustration, depicting one source and one receiver, and should not be construed as limiting this application. There can be one or more devices. For example, multiple source devices can project the screen to the receiver, such as... Figure 2 The system shown also includes an electronic device, which and electronic device 210 project their own display screen onto electronic device 220.

[0092] It should also be understood that, in this application, "large screen" can refer to an electronic device with a large display screen, such as, but not limited to, televisions and tablet computers. In this application, the large screen is used to project the display interface from a first device (such as a mobile phone).

[0093] exist Figure 2 In the scenario shown, to improve the viewing experience of the projected image, black borders need to be detected and cropped, and the cropped image is then displayed by the electronic device 220. Currently, the electronic device 220 (such as a large screen) directly detects black borders and crops the black areas to maximize the display of the projected image horizontally or vertically.

[0094] Figure 3 This is a schematic flowchart illustrating an existing screen mirroring process provided in the embodiments of this application. The following will combine... Figure 3 The detailed process of screen mirroring is described. Figure 3 In the method shown, the black border detection is performed by the receiving end (taking a large screen as an example). The source end is, for example, a mobile phone. The source end refers to the provider of the screen projection image; in other words, the source end provides the image to be projected and displayed by the receiving end, and the receiving end projects and displays the screen projection image from the source end.

[0095] Step 310: The phone and the large screen negotiate the resolution.

[0096] Normally, mobile phones and large screens have different resolutions. According to standard screen mirroring protocols, mobile phones need to be adjusted and adapted to the resolution of the large screen. This is done by filling in the black border area to form a screen mirroring image of the corresponding proportion, so that the screen mirroring image received by the large screen is consistent with the resolution ratio of the large screen.

[0097] Alternatively, resolution can be replaced with display ratio. There is a corresponding relationship between resolution and display ratio. The mobile phone can also obtain the display ratio of the large screen and then adjust the mobile phone display interface to adapt to the display ratio of the large screen.

[0098] For example, the mobile phone and the large screen can negotiate the resolution. The large screen sends its own resolution to the mobile phone so that the mobile phone can adjust its display based on the large screen's resolution, such as filling in black borders to fit the screen's aspect ratio. The mobile phone sends its own resolution to the large screen so that the large screen can obtain the mobile phone's resolution. Furthermore, the mobile phone and the large screen can negotiate the type of data transmission protocol, such as Transmission Control Protocol (TCP) or User Datagram Protocol (UDP). The specific negotiation process can be found in known technologies and will not be detailed here.

[0099] Step 320: The mobile phone receives the displayed screen.

[0100] The above-mentioned display screen is the screen shown on the mobile phone, which is the screen that you want to project onto a large screen.

[0101] Step 330: The phone adds a black border area to the displayed screen according to the large screen resolution.

[0102] The phone adds a black border to its display based on the larger screen's resolution to adjust the phone's screen resolution and maintain the same aspect ratio as the larger screen. This black border adds a black border to the phone's display, which is then projected onto the larger screen, displaying a screen with the same aspect ratio as the larger screen.

[0103] It should be understood that even without an added black border, a black border may still exist on the phone's display. In other words, the projected image with an added black border includes both the added black border from the phone and the black border from the phone's display. For example, if the phone's display includes black border area 1, and due to the different resolutions of the phone and the large screen, a black border area 2 is added to the phone's display to adapt to the larger screen's resolution ratio. Therefore, the projected image with an added black border includes both black border area 1 and black border area 2.

[0104] Step 340: The phone sends the projected screen with the black border area added to it to the large screen.

[0105] Correspondingly, the large screen receives the projected image from the mobile phone after the black border area has been added.

[0106] Step 350: The large screen performs black border detection on the projected screen after the black border area is added.

[0107] Step 360: The large screen cropped the black borders on the projected image after adding the black border area.

[0108] Large screens can formulate appropriate black border cropping strategies based on actual application scenarios, that is, which black borders need to be cropped, the size of the black border area to be cropped, etc., and then perform black border cropping to obtain the cropped projection screen.

[0109] Step 370: The cropped projection screen is displayed on the large screen.

[0110] As can be seen from the above, black border detection can be performed directly on the large screen to obtain black border data. However, in some special scenarios, the above process is time-consuming and inefficient. This results in a high lag between the screen projection displayed on the second device and the screen displayed on the first device. In other words, the synchronization rate between the screens displayed on the first and second devices is low, which leads to a poor user experience.

[0111] To address the aforementioned issues, this application provides a method for processing projected images. Based on the computational performance parameters of a first device and a second device, a target device for calculating black border data is determined. Assuming the target device is the first device, the first device calculates the black border data based on the projected image to be displayed on the second device and sends it to the second device. Compared to directly selecting the second device for black border detection to obtain black border data, the solution provided in this application fully considers the different computational capabilities of the first and second devices when determining the target device for calculating black border data. This improves the efficiency of black border detection, shortens the time required to calculate black border data, and thus enhances the user experience. Especially when the second device's display includes multiple projected images, this method helps reduce the computational load on the second device and effectively avoids the problem of low detection efficiency due to a high load on the second device.

[0112] Before introducing the method provided in the embodiments of this application, the following points should be made first:

[0113] First, to facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with essentially the same function and purpose. For example, "first device" and "second device" are used to distinguish different devices and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" do not necessarily imply that they are different.

[0114] Second, the terms “comprising” and “having” and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product or device.

[0115] Third, in the embodiments of this application, "multiple" refers to two or more. "One or more of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, one or more of a, b, or c can represent: a, b, c; a and b; a and c; b and c; or a and b and c.

[0116] Before introducing the screen projection processing method provided in this application, we will combine... Figure 4 The functional modules included in the device applicable to the method provided in this application are described in detail.

[0117] Figure 4 This is a schematic diagram of the module relationship of the device provided in the embodiments of this application.

[0118] like Figure 4 As shown, the first device includes a capability negotiation module, a video transmission module, a video encoding module, and a black border detection module. The capability negotiation module can be used to negotiate black border detection capabilities, so as to determine which end should perform black border detection based on the computing capabilities of the first and second devices. It should be understood that the capability negotiation module can also be used for negotiating transmission protocols and resolutions, etc., which are not limited in this embodiment. The video transmission module is used to transmit the projected image and / or black border data. The video encoding module is used to encode the projected image. The black border detection module is used to detect black borders on the projected image.

[0119] The second device includes a capability negotiation module, a video transmission module, a video decoding module, and a black border cropping module. The capability negotiation module can be used to negotiate black border detection capabilities, so as to determine which end should perform black border detection based on the computing capabilities of the first and second devices. It should be understood that the capability negotiation module can also be used for negotiating transmission protocols and resolutions, etc., which are not limited in this embodiment. The video transmission module is used to transmit the projected image and / or black border data. The video encoding module is used to decode the received projected image and / or black border data to obtain the projected image and / or black border data. The black border cropping module is used to crop the projected image based on the black border data to facilitate the display of the cropped projected image.

[0120] It should be understood that Figure 4 The structures of the first and second devices shown are merely examples and should not be construed as limiting the embodiments of this application. For example, the second device may further include a black border detection module to detect black borders on the projected screen when the computing power of the second device is higher than that of the first device. As another example, the second device may further include a display module to display the cropped projected screen. Yet another example, the first device may further include a display module to display images, videos, etc. The embodiments of this application do not limit the specific structure of the mobile phone and the large screen.

[0121] The following will describe in detail, with reference to the accompanying drawings, a method for processing screen projection provided by an embodiment of this application.

[0122] It should be understood that the embodiments shown below describe the method from the perspective of the interaction between the first device and the second device. The first device (which can be understood as the source of the projected screen) could be, for example, a... Figure 2 The electronic device 210 shown (such as a mobile phone), and the second device (which can be understood as the receiving end of the screen projection, i.e., the device that ultimately displays the screen projection), could be, for example, an electronic device 210 (such as a mobile phone), a second device (which can be understood as the receiving end of the screen projection, i.e., the device that ultimately displays the screen projection), for example, a second device 210 (such as a mobile phone), a second device (which can be understood as the receiving end Figure 2 The electronic device 220 shown is an example of a large screen.

[0123] It should also be understood that although the embodiments shown below are described using the interaction between a first device and a second device as an example, they should not be construed as limiting the subject of execution of the method. The method provided in this application can be executed as long as a program containing the code of the method provided in the embodiments of this application is run. For example, the first device can be replaced by a component configured in the first device (e.g., a chip, a chip system, etc.), or other functional modules capable of calling and executing programs; similarly, the second device can be replaced by a component configured in the second device (e.g., a chip, a chip system, etc.), or other functional modules capable of calling and executing programs. This application does not limit this aspect.

[0124] Figure 5This is a schematic flowchart of a screen projection processing method 500 provided in an embodiment of this application. Figure 5 The method 500 shown may include steps 510 to 540, and the steps in method 500 are described in detail below.

[0125] Step 510: The first device and the second device negotiate the target device.

[0126] The second device is used to project and display the screen image from the first device. The target device is the device determined from the first and second devices for calculating the black border data.

[0127] One possible implementation is that the first device receives computing performance parameters from the second device, and based on the computing performance parameters of the first device and the received computing performance parameters of the second device, determines the target device for computing the black border data between the first device and the second device, wherein the computing performance parameters of the first device are used to indicate the computing power of the first device, and the computing performance parameters of the second device are used to indicate the computing power of the second device.

[0128] Another possible implementation is that the second device receives computing performance parameters from the first device, and based on the received computing performance parameters of the first device and the second device, determines the target device for computing the black border data between the first device and the second device, wherein the computing performance parameters of the first device are used to indicate the computing power of the first device, and the computing performance parameters of the second device are used to indicate the computing power of the second device.

[0129] Optionally, the computing performance parameters of the first device include, but are not limited to: whether the first device has the ability to detect black borders, the computing speed of the first device, etc.; the computing performance parameters of the second device include, but are not limited to: whether the second device has the ability to detect black borders, the computing speed of the second device, etc.

[0130] It is understandable that computational performance parameters can indicate computational capability. For example, if the first device has black border detection capability but the second device does not, then the computational capability of the first device is higher than that of the second device, and the target device is the first device. Similarly, if the first device does not have black border detection capability but the second device does, then the computational capability of the second device is higher than that of the first device, and the target device is the second device. As another example, if both the first and second devices have black border detection capability, and the computational speed of the first device is higher than that of the second device, then the computational capability of the first device is higher than that of the second device, and the target device is the first device. Likewise, if both the first and second devices have black border detection capability, and the computational speed of the second device is higher than that of the first device, then the computational capability of the second device is higher than that of the first device, meaning the target device is the second device.

[0131] It should be understood that, if the computing power of the first device and the computing power of the second device are the same, the first device can be prioritized as the target device.

[0132] It should also be understood that determining the target device based on the difference in computing power between the first device and the second device is merely an example and should not constitute any limitation on the embodiments of this application. For example, the first device may also determine the target device by the relationship between its computing performance parameters and a preset threshold. For instance, if the first device's computing performance parameters are greater than the preset threshold, the first device determines itself as the target device.

[0133] Optionally, the first device and the second device can also negotiate resolution, transmission protocol, and whether the second device has performed black border detection, etc. Whether the second device has performed black border detection can reflect the load level of the second device. For example, if the second device has performed black border detection, it can be considered to have a high load and is not suitable for being identified as a target device; if the second device has not performed black border detection, it can be considered to have a low load and can be further considered for being identified as a target device. Furthermore, this application embodiment does not limit the specific content negotiated by the first device and the second device. For example, the second device can send its own resolution, computational performance parameters, etc., to the first device, and the first device can send its own resolution, computational performance parameters, etc., to the second device.

[0134] Step 520: When the target device is the first device, the first device calculates the black border data based on the first projection screen.

[0135] The first screen projection image is the screen projection image to be displayed through the second device, and the aforementioned black border data is used to indicate the black border area in the first screen projection image.

[0136] The first and second devices negotiate the target device. One scenario is that the target device is the first device, where its computing power is higher than the second device's. The other scenario is that the target device is the second device, where its computing power is higher than the first device's. When the target device is the second device, it performs black border detection on the first projected screen to obtain black border data. For details, please refer to [link / document / reference]. Figure 3 The relevant descriptions will not be repeated here. The following will describe in detail the case where the target device is the first device.

[0137] One possible implementation is that, when the target device is a first device, the first device calculates the black border data based on the first projection screen. That is, the black border data is calculated based on the projection screen to be displayed by the second device. The first projection screen can be the projection screen after the first device adds a black border area to the display screen of the first device to adapt to the resolution or display ratio of the second device, or it can be the projection screen after the resolution (or display ratio) of the first device and the second device are the same, without adding a black border area to the display screen of the first device. This application embodiment does not limit this.

[0138] For example, when the target device is a first device, the first device performs black border detection on its display screen to obtain a black border detection result. This result indicates the black border area in the first device's display screen. Further, the first device calculates black border data based on the black border detection result and the display ratio or resolution of the first projected screen. For example, if the first device has a resolution of 800×480 and the second device has a resolution of 1920×1080, the first device performs black border detection on its display screen to obtain a black border detection result, such as the top and bottom borders of the black border area being 50 and the left and right borders being 100. Further, since the first device needs to adapt to the resolution of the second device, it needs to supplement the black border area. The first projected screen obtained after resolution adaptation includes the added black border area as well as the black border area already present in the first device's display screen. The final black border data indicates the black border area in the first projected screen, and the first device calculates the black border data according to the resolution of the second device. It should be understood that the size of the top and bottom borders and the left and right borders of the black border area are merely examples and should not constitute any limitation on the embodiments of this application. For example, the top and bottom borders may be different, and the left and right borders may also be different.

[0139] The following will demonstrate a specific method for black border detection.

[0140] For example, the width and height of the image are determined by its resolution, and then black border detection is performed on the image from the outside in. Assuming a resolution of 800×480, the coordinates of the top-left corner are (0, 0), the top-right corner are (800, 0), the bottom-left corner are (0, 480), and the bottom-right corner are (800, 480). Using x as the horizontal axis and y as the vertical axis, taking the detection of the top edge of the image as an example, the pixels in the row where y = 0 are first detected, including 800 pixels. If all 800 pixels are black, the row where y = 0 is considered a black border area. If one or more of the 800 pixels are not black, the row where y = 0 is not considered a black border area. If the row where y = 0 is a black border area, the pixels in the row where y = 1 are detected, and so on, until a row is no longer considered a black border area. For example, if one or more non-black pixels are detected in a row where y=11, then that row is not considered a black border region. In other words, the top border of the black border region of the image is 10. Similarly, the bottom border, left border, and right border of the black border region of the image can be derived.

[0141] Furthermore, to improve detection efficiency, after detecting the row with y=0, you can directly detect the row with y=10. If all pixels in the row with y=10 are black, meaning the row with y=10 is a black border area, you can continue detecting the row with y=20, and so on. If the row with y=10 is not a black border area, continue narrowing down the detection range. For example, if you are detecting the pixels in the row with y=5, and one or more pixels in the row with y=5 are not black, continue narrowing down the range until all pixels in a certain row are black. This greatly improves the efficiency of black border detection.

[0142] The first device obtains the following black border area values ​​for its display screen: top border 50, bottom border 40, left border 100, and right border 50. Based on the resolution of the second device, the black border area of ​​the interface to be displayed is calculated. For example, if the resolution of the second device is 1200×1000, the center point coordinates of the image are (600, 500). The border size of the black border area added directly above is 600-400=200. Combining this with the detected black border area of ​​the first device's display screen being 50, the final black border area is 50+200=250. The rest are similar and will not be calculated here.

[0143] In step 530, the first device sends the first projected screen and black border data to the second device. Correspondingly, the second device receives the first projected screen and black border data from the first device.

[0144] When the target device is the first device, after the first device calculates the black border data, it sends the first projection screen and its corresponding black border data to the second device so that the second device can crop the first projection screen.

[0145] For example, the first device can send the first projected screen and black border data to the second device via a transport stream; in other words, the transport stream carries the first projected screen and black border data. For instance, the first and second devices can establish a data communication channel for data transmission, and the black border data can be carried by adding a black border data field to the transport stream.

[0146] It is understandable that before sending the first screen projection image and black border data, the first device can encode the first screen projection image and send the encoded first screen projection image to the second device through a transport stream. The transport stream also includes a black border data field to carry the black border data. After receiving the transport stream (taking a video stream as an example), the second device decodes it to obtain the first screen projection image and black border data.

[0147] Step 540: The second device crops the first projection screen based on the black border data to obtain and display the cropped projection screen.

[0148] After the second device acquires the first projected screen (i.e., the projected screen to be displayed on the second device) and the black border data, it can crop the first projected screen based on the black border data. For example, the second device can formulate a reasonable cropping strategy and combine it with the black border data to crop the first projected screen. For instance, if the video includes subtitles, the second device reserves an area for displaying the subtitles and crops the remaining black border area. Alternatively, the second device can crop the entire black border area; this embodiment does not limit this approach. Furthermore, the second device's implementation of a reasonable cropping strategy also includes determining the black border area to be cropped based on the resolution ratio of the second device, so that the aspect ratio of the projected screen after cropping the black border area is consistent with the resolution ratio of the second device.

[0149] Based on the above technical solution, the first and second devices determine the target device for calculating black border data based on computing performance parameters, ensuring that the selected target device has relatively high computing power. This helps to shorten the calculation time for black border data and improve the efficiency of black border detection. For example, if the target device is the first device, the first device calculates the black border data based on the first projected screen and sends it to the second device for cropping. Compared to the second device directly performing black border detection on the first projected screen to obtain the black border data, this method shortens the calculation time, improves detection efficiency, and allows the screen to be displayed quickly after being projected, thereby improving the user experience.

[0150] It should be understood that the solution described above can also be applied to multi-screen projection scenarios, that is, the display screen of the second device includes multiple projection screens (these multiple projection screens can come from different devices). The solution for multi-screen projection scenarios will be described below.

[0151] In multi-screen projection scenarios, before determining the target device for calculating black border data between the first and second devices based on the computing performance parameters of the first device and the received computing performance parameters of the second device, Figure 5 The method further includes: a first device receiving first information from a second device, the first information indicating whether the second device has performed black border detection. In other words, the second device needs to determine whether it has performed black border detection and, through the first information, indicate to the first device whether it has performed black border detection.

[0152] One possible scenario is that the first device, based on the first information, determines that the second device has performed black border detection, and thus identifies the first device as the target device. For example, the display on the large screen includes multiple projected images. The first projected image originates from mobile phone 1, and the second projected image originates from mobile phone 2. The large screen determines that the projected image from mobile phone 1 is the one performing black border detection. Therefore, the large screen instructs mobile phone 2 via the first information that it has performed black border detection. Based on the first information, mobile phone 2 determines that the large screen has performed black border detection. Furthermore, mobile phone 2 identifies itself as the target device; that is, mobile phone 2 calculates the black border data based on the projected image from mobile phone 2 to the large screen. In other words, mobile phone 2 performs the black border detection.

[0153] Another possible scenario is that, based on the first information, the first device determines that the second device has not performed black border detection. Then, the first device further determines the target device for calculating the black border data from the first and second devices based on the computing performance parameters of both devices. For example, if the computing power of the first device is higher than that of the second device, the first device determines the first device as the target device; or if the computing power of the second device is higher than that of the first device, the first device determines the second device as the target device. Wherein, if the first device determines the second device as the target device, it can also notify the second device to perform black border detection.

[0154] For example, the display on the large screen includes multiple projected images, namely the projected image from mobile phone 1 and the projected image from mobile phone 2. The large screen determines that the projected image from mobile phone 1 is the one performing black border detection. Then, the large screen sends a first message to mobile phone 2 instructing that the large screen has not performed black border detection. Based on the first message, mobile phone 2 determines that the large screen has not performed black border detection. Furthermore, mobile phone 2 determines the target device based on the computing performance parameters of mobile phone 2 and the large screen. If the computing power of mobile phone 2 is higher than that of the large screen, then mobile phone 2 is determined to be the target device. If the computing power of the large screen is higher than that of the large screen, then the large screen is determined to be the target device.

[0155] For example, the display on the large screen includes multiple projected images, namely the projected image from mobile phone 1, the projected image from mobile phone 2, and the projected image from mobile phone 3. The large screen determines that the projected image from mobile phone 1 is the one whose black border detection is performed by mobile phone 1, and the projected image from mobile phone 2 is the one whose black border detection is performed by the large screen. Then, the large screen instructs mobile phone 3 through the first information that the large screen has performed black border detection. Based on the first information, mobile phone 3 determines that the large screen has performed black border detection, and further determines that mobile phone 3 is the target device. That is, mobile phone 3 calculates the black border data based on the projected image from mobile phone 3 to the large screen.

[0156] It should be understood that in either of the above two situations, the target device can also be determined by the second device. In other words, the second device determines whether it has performed black border detection. If it has not performed black border detection, it further determines the target device based on the computing performance parameters from the first device and its own computing performance parameters. If it has performed black border detection, it determines the first device as the target device. In this case, the second device can also notify the first device to perform black border detection.

[0157] It should also be understood that the second device can send the first information and the computing performance parameters together to the first device, that is, the first information and the computing performance parameters of the second device are carried in the same signaling, or the first information and the computing performance parameters can be sent separately, that is, the first information and the computing performance parameters of the second device are carried in different signaling. This application embodiment does not limit this.

[0158] Based on the above technical solution, in a multi-screen projection scenario, it is first determined whether the second device has performed black border detection. If not, the first and second devices determine the target device for calculating black border data based on computing performance parameters, ensuring that the selected target device has relatively high computing power. This helps to shorten the calculation time for black border data and improve the efficiency of black border detection. If black border detection has been performed, the first device performs the black border detection. This reduces the computing load on the second device, allowing for efficient use of the first device's computing power, thereby improving the efficiency of black border detection. Furthermore, this allows the screen to be displayed quickly after projection, improving the synchronization rate of the screens displayed by the first and second devices, and ultimately enhancing the user experience.

[0159] The following will combine Figure 6 The single-channel screen projection method provided in the embodiments of this application will be described in detail.

[0160] Figure 6 This is a schematic diagram of the screen mirroring method provided in an embodiment of this application. For example... Figure 6 As shown, the first device is a mobile phone, and the second device is a large screen.

[0161] The mobile phone (as shown in Figure a) projects its screen onto the large screen. Before projection, the phone and the large screen negotiate computing power. For example, the phone receives computing performance parameters from the large screen and compares them with its own. If the phone's computing power is higher than the large screen's, the phone performs black border detection; if the large screen's computing power is higher, the large screen performs black border detection. The process of black border detection by the large screen can refer to known technologies and will not be detailed here.

[0162] The following assumes that the black border detection is performed by a mobile phone. The mobile phone captures the screen display (e.g., ...). Figure 6 As shown in b) of the diagram, black border detection is performed on the current mobile phone display screen to obtain the black border detection result (as shown in b) of the diagram. Figure 6 (The black area shown in b) in the text.

[0163] in, Figure 6Figures b) and c) show the mobile phone display screen when it is not adapted to the large screen resolution (for example, in a scenario where both resolutions are the same). The projection screen after adding black borders to the mobile phone to adapt to the large screen resolution is not shown in the figures, but this should not be construed as limiting the embodiments of this application. If the mobile phone and the large screen have different resolutions, the mobile phone performs black border detection on the mobile phone display screen. After obtaining the black border detection result, it further adds black borders to the mobile phone display screen according to the resolution of the large screen to obtain a projection screen adapted to the large screen resolution (i.e., the screen to be displayed on the large screen). Based on the resolution of the projection screen and the black border detection result, black border data is calculated. This black border data is used to indicate the black border area of ​​the projection screen.

[0164] Before sending the screen projection, the mobile phone can encode the projection screen (for example, the projection screen with black borders added when the resolution is different, or the projection screen without black borders added when the resolution is the same) and send it to the large screen through the transport stream. At the same time, the transport stream also carries black border data, which is used to indicate the black border area of ​​the projection screen.

[0165] On the other hand, the large screen decodes the received projected image and black border data to obtain the black border data and projected image (such as...). Figure 6 As shown in c) in the figure, where, Figure 6 c) only shows the screen projection without black borders, and does not show the screen projection with black borders, but this should not be construed as limiting the embodiments of this application.

[0166] Furthermore, the large screen cropped the black borders of the projected image to obtain the cropped projected image (e.g., Figure 6 As shown in d), where, Figure 6 The cropped projection screen shown in d) is a projection screen with all black border areas cropped out, but this should not constitute any limitation on the embodiments of this application. For example, a large screen can also crop out part of the black border area to obtain a cropped projection screen, and the embodiments of this application do not limit this.

[0167] Finally, the cropped projection image will be displayed on the large screen (e.g., Figure 6 (as shown in e)).

[0168] Understandable. Figure 6 Although b) and c) are the same, Figure 6 b) shows the screen projection screen before encoding on the mobile phone side. Figure 6 c) shows the screen projection image after decoding on the large screen.

[0169] The following will combine Figure 7 The screen projection method provided in the embodiments of this application will be described in detail. Figure 7This refers to a dual-screen projection scenario, where the large screen displays multiple projected images, such as simultaneously projecting multiple mobile phone screens onto the same large screen.

[0170] Figure 7 This is a schematic diagram of the dual-channel projection method provided in an embodiment of this application. For example... Figure 7 As shown, the phone on the left is denoted as phone 1, and the phone on the right is denoted as phone 2. Assuming that phone 1, after negotiation with the large screen, determines that phone 1 will perform black border detection, then phone 2, after negotiation with the large screen, may have the large screen perform black border detection or phone 2 will perform black border detection. Let's assume that phone 2 will perform black border detection.

[0171] Figure 7 a) Figure 7 (b) Figure 7 c) Figure 7 d) and Figure 7 e) in the diagram illustrates the process of mobile phone 1 projecting its screen onto a large screen. For a detailed description, please refer to [link / reference needed]. Figure 6 This will not be elaborated upon here. Figure 7 f) Figure 7 g) Figure 7 h) Figure 7 k) and Figure 7 e) in the diagram illustrates the process of phone 2 projecting its screen onto a larger screen. For a more detailed description, please refer to [link / reference]. Figure 6 This will not be elaborated upon here.

[0172] It should be understood that, assuming that after negotiation between mobile phone 1 and the large screen, it is determined that the large screen will perform the black border detection, then after negotiation between mobile phone 2 and the large screen, since the large screen has already performed the black border detection of the screen projection of mobile phone 1, mobile phone 2 will perform the black border detection of the screen projection of mobile phone 2.

[0173] Figure 8 This is a flowchart illustrating the screen mirroring process provided in an embodiment of this application. Figure 8 In this example, the first device is a mobile phone, and the second device is a large screen.

[0174] like Figure 8 As shown in the figure, in this application, the process of projecting the mobile phone display screen onto a large screen includes the following steps:

[0175] Step 810: The mobile phone and the large screen negotiate the target device.

[0176] In other words, the device that negotiates with the phone and the large screen to perform black border detection.

[0177] Step 820: The phone displays the screen.

[0178] The aforementioned display screen is the screen shown on the mobile phone and is intended to be projected onto a large screen.

[0179] Step 830: The mobile phone performs black border detection on the displayed screen.

[0180] The above display is the screen shown on the mobile phone, that is, the screen before adding black borders based on the resolution of the large screen.

[0181] Step 840: The phone adds a black border area to the displayed screen according to the large screen resolution.

[0182] Step 850: The mobile phone sends the projected screen image and black border data after the black border area has been added to the large screen.

[0183] Accordingly, the large screen receives the projected image with the added black border area and the black border data from the mobile phone. The aforementioned black border data is used to indicate all black border areas in the projected image after the black border area is added, including the added black border area and the black border area that is already present in the mobile phone display screen.

[0184] Step 860: The large screen cropped the black borders of the projected image after adding the black border area.

[0185] Step 870: The cropped projection image is displayed on the large screen.

[0186] about Figure 8 The steps of the method shown can be found in [reference]. Figure 5 The relevant descriptions will not be repeated here.

[0187] In summary, the screen mirroring method provided in this application and known screen mirroring methods (see [link to known screen mirroring methods]) Figure 3 The main differences are as follows:

[0188] The first difference is that existing technologies directly perform black border detection on the large screen without consulting the mobile phone. In this application, the mobile phone and the large screen negotiate which end performs the black border detection. For example, both parties determine the device to perform the black border detection based on computing performance. This fully considers and utilizes the mobile phone's computing power, which is beneficial for effectively utilizing the mobile phone's computing power when the large screen's computing power is insufficient or low, thereby improving the efficiency of black border detection.

[0189] The second difference is that existing technologies perform black border detection on the large screen, and the black border detection is performed on the projected screen after adapting to the large screen resolution. In other words, the black border detection is performed on the projected screen after adding black border areas. In this application, the black border detection is performed on the mobile phone display screen, that is, the black border detection is performed on the projected screen without black border areas. This can reduce the area of ​​black border detection and help improve detection efficiency.

[0190] Difference 3: In known technologies, mobile phones do not need to send black border data to the large screen. In this application, mobile phones need to send black border data to the large screen so that the large screen can directly receive the black border data and crop it based on the black border data to obtain the final screen projection image.

[0191] Based on the above technical solution, the first and second devices determine the target device for calculating black border data based on computational performance parameters. This ensures that the selected target device has relatively high computing power, which helps to shorten the calculation time for black border data, improve the efficiency of black border detection, and further enable the screen to be displayed quickly after being projected, thereby improving the user experience. For example, if the target device is the first device, the first device calculates the black border data based on the first projected screen and sends it to the second device for cropping. Compared to the second device directly performing black border detection on the first projected screen to obtain the black border data, this method shortens the calculation time and improves detection efficiency. In addition, in multi-projection scenarios, prioritizing whether the second device has already performed black border detection helps to mitigate the problem of low detection efficiency caused by the second device performing black border detection on multiple projected screens. That is, fully utilizing the computing power of the first device helps to improve the efficiency of black border detection.

[0192] It should be understood that each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are not provided here.

[0193] This application also provides an electronic device, which may include a memory and a processor. The memory may be used to store computer programs; the processor may be used to invoke the computer programs stored in the memory, causing the electronic device to perform... Figures 5 to 8 The method executed by the first device or the method executed by the second device in any of the embodiments shown.

[0194] This application also provides a chip system, the chip system including at least one processor for implementing the above. Figures 5 to 8 The method described in any of the embodiments shown may, for example, receive or process the data and / or information involved in the above methods.

[0195] In one possible design, the chip system also includes a memory for storing program instructions and data, which may be located within or outside the processor.

[0196] The chip system can consist of chips or include chips and other discrete components.

[0197] This application also provides a computer program product, the computer program product comprising: a computer program (also referred to as code or instructions), which, when executed, causes an electronic device to perform... Figures 5 to 8 The method described in any one of the embodiments shown.

[0198] This application also provides a computer-readable storage medium storing a computer program (also referred to as code or instructions). When the computer program is executed, it causes an electronic device to perform... Figures 5 to 8 The method described in any one of the embodiments shown.

[0199] This application also provides a communication system, including the first device and the second device as described above.

[0200] It should be understood that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method embodiments can be completed by the integrated logic circuitry in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0201] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0202] The terms “unit”, “module”, etc., used in this specification may be used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution.

[0203] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. In the several embodiments provided in this application, it should be understood that the disclosed apparatus, devices, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0204] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0205] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0206] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs, DVDs), or semiconductor media (e.g., solid-state disks, SSDs), etc.

[0207] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0208] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for processing projected images, characterized in that, Applied to a first device, the method includes: Based on the computing performance parameters of the first device and the received computing performance parameters of the second device, a target device for computing black border data is determined between the first device and the second device. The second device is used to project and display the screen from the first device. The computing performance parameters of the first device are used to indicate the computing power of the first device, and the computing performance parameters of the second device are used to indicate the computing power of the second device. When the target device is the first device, black border data is calculated based on the first projection screen, where the first projection screen is the projection screen to be displayed through the second device, and the black border data is used to indicate the black border area in the first projection screen. The first projection screen and the black border data are sent to the second device.

2. The method as described in claim 1, characterized in that, Sending the first projection screen and the black border data to the second device includes: The first projection screen and the black border data are sent to the second device via a transmission stream.

3. The method as described in claim 1 or 2, characterized in that, The first projected image is obtained based on the display image of the first device; Before calculating the black border data based on the first projected screen, the method further includes: Perform black border detection on the displayed image to obtain black border detection results, which are used to indicate black border areas in the displayed image; and, The calculation of black border data based on the first projected screen includes: Based on the black border detection results and the display ratio or resolution of the first projected image, the black border data is calculated.

4. The method according to any one of claims 1 to 3, characterized in that, Before determining the target device for calculating the black border data among the first device and the second device based on the computing performance parameters of the first device and the received computing performance parameters of the second device, the method further includes: Receive first information from the second device, the first information being used to indicate whether the second device has performed black border detection; Based on the first information, it is determined that the second device did not perform black border detection.

5. The method according to any one of claims 1 to 4, characterized in that, The step of determining the target device for calculating black border data among the first device and the second device based on the computing performance parameters of the first device and the received computing performance parameters of the second device includes: If the computing power of the first device is higher than that of the second device, then the first device is determined to be the target device; or If the computing power of the second device is higher than that of the first device, the second device is determined to be the target device.

6. The method according to any one of claims 1 to 5, characterized in that, When the target device is the second device, the method further includes: The first screen projection image is sent to the second device, but the black border data is not sent.

7. A method for processing projected images, characterized in that, Applied to a second device, the method includes: Based on the computing performance parameters of the second device and the received computing performance parameters of the first device, a target device for computing black border data is determined between the first device and the second device. The second device is used to project and display the screen from the first device. The computing performance parameters of the first device are used to indicate the computing power of the first device, and the computing performance parameters of the second device are used to indicate the computing power of the second device. When the target device is the first device, a first projection screen and black border data are received from the first device. The black border data is calculated by the first device based on the first projection screen and is used to indicate the black border area in the first projection screen. The first projection screen is the projection screen to be displayed by the second device.

8. The method as described in claim 7, characterized in that, The receiving of the first projected image and black border data from the first device includes: The first projected screen and the black border data are received via a transmission stream.

9. The method as described in claim 7 or 8, characterized in that, The method further includes: If the display screen of the second device includes multiple projected screens, determine whether the second device has performed black border detection; The step of determining the target device for calculating black border data among the first device and the second device based on the computing performance parameters of the second device and the received computing performance parameters of the first device includes: If it is determined that the second device does not perform black border detection, a target device for calculating black border data is determined among the first device and the second device based on the computing performance parameters of the second device and the received computing performance parameters of the first device.

10. The method as described in claim 9, characterized in that, The method further includes: Send a first message to the first device, the first message being used to indicate whether the second device has performed black border detection.

11. The method according to any one of claims 7 to 10, characterized in that, The step of determining the target device for calculating black border data among the first device and the second device based on the computing performance parameters of the second device and the received computing performance parameters of the first device includes: If the computing power of the first device is higher than that of the second device, then the first device is determined to be the target device; or If the computing power of the second device is higher than that of the first device, the second device is determined to be the target device.

12. The method according to any one of claims 7 to 11, characterized in that, When the target device is the second device, the method further includes: Receive the first screen projection image from the first device; Black border detection is performed on the first projected screen to obtain black border data.

13. The method according to any one of claims 7 to 12, characterized in that, The method further includes: Based on the black border data, the first projection screen is cropped to obtain and display the cropped projection screen.

14. A device for processing projected images, characterized in that, Including processor and memory, among which, The memory is used to store computer programs; The processor is configured to invoke the computer program to cause the device to perform the method of any one of claims 1 to 6, or to cause the device to perform the method of any one of claims 7 to 13.

15. A computer-readable storage medium, characterized in that, The storage medium stores a computer program or instructions, which, when executed by a computer, implement the method as described in any one of claims 1 to 6, or the method as described in any one of claims 7 to 13.

16. A computer program product, characterized in that, The computer program product includes instructions that, when executed by a computer, implement the method as described in any one of claims 1 to 6, or implement the method as described in any one of claims 7 to 13.

Images