Video data processing method, chip, electronic device and readable storage medium

Abstract

This application provides a video data processing method, relating to the field of terminals. A first electronic device encodes image frames projected to a second electronic device based on a first number of segments, obtaining a first number of slices. The first electronic device processes the slices through multiple parallel threads, and sends the processed data to the second electronic device. The first electronic device obtains the processing time of each thread processing the slices, and updates the first number of segments based on the processing time of each thread. This application also provides an electronic device, a chip, and a computer-readable storage medium. This application uses slices as the basic unit for processing, and processes slices through parallel threads, reducing slice processing time. Simultaneously, it can update the number of image frame segments based on the monitored slice processing time, maximizing thread processing efficiency and reducing projection latency.

Description

Technical Field

[0001] This application relates to the field of terminals, and more particularly to a video data processing method, a chip, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Digital video technology has been increasingly widely used in the fields of communications and broadcasting. With the rapid development of the Internet and mobile communications, the processing and transmission of video information in the Internet or mobile networks has become a hot technology in my country's informatization. Video data is large in volume, requiring solutions to the compression coding problem. The mainstream solution today is to use coding standards such as H.264 and H.265 to encode video data.

[0003] Most existing mobile phones support screen mirroring protocols. After the mobile phone (source device) and the device being mirrored (destination device) are connected to the same Wi-Fi hotspot or via Wi-Fi Direct, the mobile phone projects its interface onto the destination device (TV, computer, etc.) using the screen mirroring protocol. Currently, mobile phone screen mirroring processes images frame by frame. The mobile phone uses H.264 or H.265 encoding standards to encode, encrypt, and package image frames, then sends the packaged result to the destination device. The destination device decrypts, unpacks, and decodes the received packaged result, and then renders and displays the decoded image frame. If the encoding time of the image frame is too long, it will cause delays in subsequent processing steps such as packaging, encryption, and transmission (the next process can only proceed after the previous process has completed processing the image frame), resulting in significant screen mirroring latency and impacting the user's viewing experience. Summary of the Invention

[0004] In view of this, it is necessary to provide a video data processing method that processes data in strips as the basic unit, which can reduce projection latency.

[0005] The first aspect of this application discloses a video data processing method applied to a first electronic device, which projects several image frames onto a second electronic device for display. The video data processing method includes: encoding each image frame in the plurality of image frames based on a first number of segments to obtain a first number of stripes corresponding to each image frame; wherein the first electronic device includes multiple threads running in parallel for processing the stripes; sending the processed data obtained from processing the stripes to the second electronic device; obtaining the processing time of each thread processing the stripes; and updating the first number of segments based on the processing time of each thread processing the stripes.

[0006] Using this technical solution, the first electronic device encodes the image frames to be projected into several strips, processes them using strips as the basic unit, and processes the strips through parallel threads, thereby reducing the processing time of the strips. At the same time, by monitoring the processing time of each thread's strips and determining the optimal number of image frames to be segmented based on the monitored processing time, the image frames are then encoded based on the optimal number of segments, which can maximize the processing efficiency of the threads and reduce the projection latency.

[0007] In some embodiments, the plurality of image frames includes a first image frame, and the video data processing method further includes: obtaining the processing time of each of the plurality of threads processing the first image frame, and calculating a first segmentation number based on the processing time of each thread processing the first image frame.

[0008] Using this technical solution, the initial screen projection display can be processed using image frames as the basic unit. The first number of segments is calculated by estimating the processing time of each thread on the image frame. Then, the image frames are encoded based on the first number of segments to obtain several strips, which are then processed using the strips as the basic unit.

[0009] In some embodiments, updating the first segmentation number based on the processing time of each thread for the stripe includes: calculating a second segmentation number based on the processing time of each thread for processing the same stripe; obtaining multiple stripes processed within a first preset time, and multiple second segmentation numbers corresponding to the multiple stripes; obtaining a third segmentation number based on the multiple second segmentation numbers, and updating the first segmentation number to the third segmentation number; and encoding unencoded image frames in a plurality of image frames based on the third segmentation number.

[0010] This technical solution allows for the periodic execution of processing based on the current stripe to obtain the optimal number of segments corresponding to the current stripe. It can also calculate multiple optimal number of segments within a preset time period, and obtain the current optimal number of segments based on these multiple optimal number of segments. The current optimal number of segments is then used to replace the number of segments used previously, thereby periodically attempting to update the encoding rules of the image frames and minimizing projection latency.

[0011] In some embodiments, obtaining a third segmentation number based on a plurality of second segmentation numbers includes: averaging the plurality of second segmentation numbers and rounding the calculated average up or down to obtain the third segmentation number; or using the median of the plurality of second segmentation numbers as the third segmentation number.

[0012] Using this technical solution, the optimal number of segments can be calculated from multiple second segmentation numbers based on preset calculation rules. For example, the mean or median of multiple second segmentation numbers can be used as the optimal number of segments.

[0013] In some embodiments, the multiple threads include an encoding thread, an encryption thread, and a packaging thread. Calculating the second number of segments based on the processing time of each thread processing the same stripe includes: obtaining the encoding processing time of the encoding thread encoding any one of the image frames into a first number of stripes; obtaining the encryption processing time and packaging processing time of the encryption thread and the packaging thread for a stripe in any one image frame; and calculating the second number of segments based on the smaller of the encoding processing time and the encryption processing time and the packaging processing time.

[0014] By adopting this technical solution, the second number of segments is calculated based on the smaller of the total encoding time consumed by encoding an image frame into multiple stripes and the encryption processing time and packing processing time of the stripes. This ensures that each thread does not hinder the data processing of the next thread, thereby maximizing the processing efficiency of the threads.

[0015] In some embodiments, obtaining multiple stripes processed within a first preset time includes: when the calculated second number of segments is greater than or less than the first number of segments, obtaining multiple stripes processed within the first preset time.

[0016] This technical solution aims to update the number of the first segment when the number of the second segment is not equal to the number of the first segment, thereby maximizing the processing efficiency of the thread.

[0017] In some embodiments, the first electronic device includes a video decoder, which encodes each image frame in a plurality of image frames based on a first number of segments, including: querying whether the video decoder has stripe coding capability; querying whether the second electronic device has at least one of stripe decoding capability and stripe packetization capability; when the video decoder has stripe coding capability and the second electronic device has stripe decoding capability or stripe packetization capability, encoding each image frame in the plurality of image frames based on the first number of segments.

[0018] This technical solution allows image frames to be encoded into several stripes when both the source and destination devices have stripe processing capabilities, thus preventing screen projection errors caused by either the source or destination device being unable to process the stripes.

[0019] Secondly, embodiments of this application provide a video data processing method applied to a second electronic device, wherein a first electronic device projects several image frames onto the second electronic device for display. The video data processing method includes: receiving processing data sent by the first electronic device; processing the processing data to obtain stripes corresponding to the processing data; splicing the multiple stripes obtained from the processing to obtain encoded image frames corresponding to the multiple stripes; and decoding and displaying the encoded image frames.

[0020] Using this technical solution, the image frames displayed on the screen are encoded into several slices, and processed using slices as the basic unit, which reduces the screen projection latency. By assembling and decoding the slices, the image frames displayed on the screen can be realized.

[0021] In some embodiments, splicing multiple processed stripes includes: detecting whether the processed stripe contains an end marker; when it is detected that the processed stripe does not contain an end marker, caching the processed stripe; when it is detected that the processed stripe contains an end marker, splicing the stripe containing the end marker with the previously cached stripe.

[0022] By using this technical solution, it is possible to determine whether a strip is the last strip of an image frame by detecting whether the strip contains an end marker. When the last strip of an image frame is detected, the strip is spliced ​​with the previously cached strips to obtain a complete encoded image frame.

[0023] In some embodiments, the second electronic device includes multiple parallel threads for processing stripes, and splicing the processed stripes together, including splicing the processed stripes together when the second electronic device has stripe splicing capability but no stripe decoding capability.

[0024] Using this technical solution, for destination devices that do not support stripe decoding, a complete encoded image frame is obtained by splicing the stripes, enabling the destination device to decode the encoded image frame.

[0025] In some embodiments, the video data processing method further includes: when the second electronic device has stripe decoding capability, decoding and displaying the processed stripe.

[0026] Using this technical solution, for target devices that support stripe decoding, the stripes obtained from decryption and unpacking can be decoded directly without stripe repackaging.

[0027] In some embodiments, decoding and displaying the processed stripe includes: detecting whether the processed stripe contains an end marker; when the processed stripe does not contain the end marker, decoding the processed stripe and caching the decoding result; when the processed stripe contains an end marker, decoding the stripe containing the end marker and displaying the decoding result of the stripe containing the end marker along with the decoding results of all previously cached stripes.

[0028] Using this technical solution, it is possible to determine whether a strip is the last strip of an image frame by detecting whether the strip contains an end marker. When the decoding of the last strip of the image frame is completed, the decoding result of the last strip is sent to the display along with the decoding results of all previously cached strips.

[0029] Thirdly, embodiments of this application provide a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the video data processing method as described in the first or second aspect.

[0030] Fourthly, embodiments of this application provide an electronic device, which includes a processor and a memory. The memory is used to store instructions, and the processor is used to call the instructions in the memory, causing the electronic device to perform the video data processing method as described in the first or second aspect.

[0031] Fifthly, embodiments of this application provide a computer program product that, when run on a computer, causes the computer to perform the video data processing method as described in the first or second aspect.

[0032] Sixthly, a device (such as a chip) is provided that has the function of implementing the behavior of a first electronic device in the method provided in the first aspect or the function of implementing the behavior of a second electronic device in the method provided in the second aspect. The function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described functions.

[0033] It is understood that the computer-readable storage medium described in the third aspect, the electronic device described in the fourth aspect, the computer program product described in the fifth aspect, and the device described in the sixth aspect all correspond to the methods described in the first or second aspects above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application;

[0035] Figure 2 A schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0036] Figure 3 This is a schematic diagram illustrating an application scenario of the video data processing method provided in an embodiment of this application;

[0037] Figure 4 A schematic diagram illustrating screen projection image frame processing between a first electronic device and a second electronic device according to an embodiment of this application;

[0038] Figure 5 A schematic diagram of the processing flow of a first electronic device provided in an embodiment of this application, using strips as the basic processing unit;

[0039] Figure 6 This is a schematic diagram illustrating the correspondence between the number of image frame segments and the encoding time in an embodiment of this application.

[0040] Figure 7 This is a schematic diagram of data interaction between a first electronic device and a second electronic device provided in an embodiment of this application;

[0041] Figure 8 This is a schematic diagram of data interaction between a first electronic device and a second electronic device provided in another embodiment of this application;

[0042] Figure 9 A schematic diagram comparing the data processing time with and without striping in a screen projection scenario provided in an embodiment of this application;

[0043] Figure 10 A schematic diagram of the processing flow of a second electronic device provided in an embodiment of this application, which uses stripes as the basic processing unit when it does not have stripe decoding capability;

[0044] Figure 11 This is a schematic diagram of data interaction between a first electronic device and a second electronic device, provided in yet another embodiment of this application.

[0045] Figure 12 A schematic diagram illustrating strip splicing of a second electronic device provided in an embodiment of this application;

[0046] Figure 13 A schematic diagram of the processing flow of a second electronic device provided in an embodiment of this application, which has stripe decoding capability and uses stripes as the basic processing unit;

[0047] Figure 14 A schematic flowchart illustrating a video data processing method provided in an embodiment of this application;

[0048] Figure 15 A flowchart illustrating a video data processing method provided in another embodiment of this application;

[0049] Figure 16 This is a schematic diagram of the structure of a possible first electronic device provided in an embodiment of this application;

[0050] Figure 17 This is a schematic diagram of the structure of a possible second electronic device provided in an embodiment of this application. Detailed Implementation

[0051] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or sequence.

[0052] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0053] For ease of understanding, some concepts related to the embodiments of this application are illustrated and explained by way of example for reference.

[0054] Video is a series of image frames displayed at a given frequency. When the continuous image changes exceed 24 frames per second, the human eye can no longer distinguish individual static images, and the visual effect appears smooth and continuous. Video encoding is essentially encoding each image frame of the video. For example, encoding standards such as H.264 or H.265 can be used to encode image frames. The H.264 encoding standard's bitstream structure can be divided into two layers: the Network Abstraction Layer (NAL) and the Video Coding Layer (VCL). In the NAL, the H.264 format bitstream is represented as a series of Network Abstraction Layer Units (NALUs), with different NALUs containing different types of syntax elements. The portion of the video containing image data is stored in the VCL's NALUs; this data is called a slice in the bitstream. A slice contains part or all of the data from one frame of image data, and one frame of image data can be encoded into one or more slices. A slice contains at least one macroblock and at most the data of the entire image frame.

[0055] An image (video is essentially composed of consecutive images) can be encoded into a frame. An image can be divided into several macroblocks, each of which can use a different predictive coding type; that is, the predictive coding types of different macroblocks in an image can be different. A macroblock is the basic unit of coding processing, typically 16x16 pixels or 8x8 pixels in size. In the H.264 or H.265 coding standards, an image can be encoded into one or more slices, each slice consisting of macroblocks. Slices limit the spread and transmission of errors, keeping the encoded data independent. After encoding, a slice can be packed into a NALU. Besides holding slices, a NALU can also hold other data. Decoding operations between different slices are also independent. The data referenced during the decoding process of a particular slice (e.g., predictive coding) cannot cross the slice's boundaries.

[0056] Electronic devices can communicate with other electronic devices or servers via a communication network. Electronic devices may include at least one of the following: mobile phones, foldable electronic devices, tablets, personal computers (PCs), laptops, handheld computers, ultra-mobile personal computers (UMPCs), netbooks, cellular phones, personal digital assistants (PDAs), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, in-vehicle devices, smart home devices, or smart city devices. This application does not impose any special limitations on the specific type of electronic device. The communication network can be a wired network or a wireless network. For example, the communication network can be a local area network (LAN) or a wide area network (WAN), such as the Internet. When the communication network is a LAN, for example, it can be a Wi-Fi hotspot network, a Wi-Fi P2P network, a Bluetooth network, a Zigbee network, or a near field communication (NFC) network, etc. When the communication network is a wide area network, for example, the communication network can be a third-generation wireless telephone technology (3G) network, a fourth-generation mobile communication technology (4G) network, a fifth-generation mobile communication technology (5G) network, a future evolved public land mobile network (PLMN) or the Internet, etc.

[0057] Electronic devices can install one or more applications. An application, often simply called an app, is a software program that performs one or more specific functions. Examples include instant messaging apps, video apps, audio apps, image capture apps, cloud desktop apps, and so on. Instant messaging apps, for example, can include SMS apps, WhatsApp Photo sharing Kakao Applications that capture images, such as camera apps (system camera or third-party camera apps). Applications that capture videos, such as... And so on. Audio applications, for example, may include... QQ Etc. The applications mentioned in the following embodiments may be system applications that are pre-installed when the electronic device leaves the factory, or third-party applications that the user downloads from the network or obtains from other electronic devices during the use of the electronic device.

[0058] Electronic devices including but not limited to those equipped with Windows Or other operating systems.

[0059] Figure 1 This diagram illustrates the structure of an electronic device 10.

[0060] Electronic device 10 may include processor 110, external memory interface 120, internal memory 121, antenna 1, antenna 2, mobile communication module 130, wireless communication module 140, audio module 150, sensor module 160, camera module 170, display screen 180, etc.

[0061] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 10. In other embodiments of this application, the electronic device 10 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0062] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0063] The processor can generate operation control signals based on the instruction opcode and timing signals to control the instruction fetching and execution.

[0064] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 may be a cache memory. This memory can store instructions or data that the processor 110 has used or that are used frequently. If the processor 110 needs to use the instruction or data, it can directly retrieve it from this memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0065] In some embodiments, the processor 110 may include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc. The processor 110 can connect to modules such as an audio module, a wireless communication module, a display, and a camera through at least one of these interfaces.

[0066] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 10. In other embodiments of this application, the electronic device 10 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0067] The wireless communication function of electronic device 10 can be implemented through antenna 1, antenna 2, mobile communication module 130, wireless communication module 140, modem processor, and baseband processor.

[0068] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 10 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0069] The mobile communication module 130 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 10. The mobile communication module 130 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 130 can receive electromagnetic waves via the antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to the modem processor for demodulation. The mobile communication module 130 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via the antenna 1. In some embodiments, at least some functional modules of the mobile communication module 130 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 130 and at least some modules of the processor 110 may be housed in the same device.

[0070] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device or displays an image or video through the display screen 180. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 130 or other functional modules.

[0071] The wireless communication module 140 can provide solutions for wireless communication applications on the electronic device 10, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), Bluetooth Low Energy (BLE), ultra-wideband (UWB), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 140 can be one or more devices integrating at least one communication processing module. The wireless communication module 140 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 140 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0072] In some embodiments, antenna 1 of electronic device 10 is coupled to mobile communication module 130, and antenna 2 is coupled to wireless communication module 140, enabling electronic device 10 to communicate with networks and other electronic devices via wireless communication technology. This wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0073] Electronic device 10 can implement display functions through a GPU, a display screen 180, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 180 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0074] The sensor module includes touch sensors, pressure sensors, fingerprint sensors, etc. The camera module 170 includes a camera. The display screen 180 is used to display images, videos, etc. The display screen 180 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 10 may include one or more displays 180.

[0075] Electronic device 10 can realize camera function through camera module 170, ISP, video codec, GPU, display screen 180, application processor AP, neural network processor NPU, etc.

[0076] A digital signal processor (DSP) is used to process digital signals, and can also process other digital signals. For example, when the electronic device 10 selects a frequency point, the DSP is used to perform Fourier transforms on the frequency energy, etc.

[0077] Video codecs are used to compress or decompress digital video. Electronic device 10 may support one or more video codecs. Thus, electronic device 10 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0078] An NPU (Neural Processing Unit) is a neural network (NN) computing processor that, by borrowing from the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, rapidly processes input information and can continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0079] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 10. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, it can save music, video, and other files to the external memory card, or transfer music, video, and other files from the electronic device to the external memory card.

[0080] Internal memory 121 can be used to store computer executable program code, including instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc. The data storage area may store data created during the use of electronic device 10 (such as audio data, phone book, etc.). In addition, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional methods or data processing of electronic device 10 by running instructions stored in internal memory 121 and / or instructions stored in memory disposed in the processor.

[0081] The audio module 150 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 150 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 150 may be located in the processor 110, or some functional modules of the audio module 150 may be located in the processor 110.

[0082] The software system of electronic device 10 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 10.

[0083] Figure 2 This is a software structure block diagram of the electronic device 10 according to an embodiment of this application.

[0084] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: the application layer, the application framework layer, the Android runtime (ART) and native C / C++ libraries, the Hardware Abstraction Layer (HAL), and the kernel layer.

[0085] The application layer can include a series of application packages.

[0086] like Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0087] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0088] like Figure 2 As shown, the application framework layer may include a window manager, content provider, view system, resource manager, notification manager, activity manager, input manager, etc.

[0089] The window manager provides Window Manager Service (WMS), which can be used for window management, window animation management, surface management, and as a relay station for the input system.

[0090] Content providers store and retrieve data, making that data accessible to applications. This data can include videos, images, audio, phone calls made and received, browsing history and bookmarks, phone books, etc.

[0091] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0092] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0093] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0094] The Activity Manager Service (AMS) can be used to start, switch, and schedule system components (such as activities, services, content providers, and broadcast receivers), as well as manage and schedule application processes.

[0095] The Input Manager Service (IMS) provides input management services, which can be used to manage system inputs such as touchscreen input, keypad input, and sensor input. IMS retrieves events from input device nodes and, through interaction with the WMS (Windows Management System), distributes these events to appropriate windows.

[0096] The Android runtime consists of the core libraries and the Android runtime itself. The Android runtime is responsible for converting source code into machine code. The Android runtime primarily employs ahead-of-time (AOT) compilation and just-in-time (JIT) compilation techniques.

[0097] The core library primarily provides basic Java class library functionalities, such as libraries for fundamental data structures, mathematics, I / O, tools, databases, and networking. It also provides APIs for users to develop Android applications.

[0098] Native C / C++ libraries can include multiple functional modules. Examples include: surface manager, media framework, libc, OpenGL ES, SQLite, Webkit, etc.

[0099] The Surface Manager manages the display subsystem and provides 2D and 3D layer blending for multiple applications. The Media Framework supports playback and recording of various common audio and video formats, as well as still image files. The Media Library supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. OpenGL ES provides drawing and manipulation of 2D and 3D graphics in applications. SQLite provides a lightweight relational database for applications on the electronic device 10.

[0100] The Hardware Abstraction Layer (HAL) runs in user space, encapsulates kernel-level drivers, and provides calling interfaces to the upper layers.

[0101] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0102] The following example illustrates the workflow of the software and hardware of electronic device 10 in the context of capturing and photographing a scene.

[0103] When the display receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, etc.). The raw input event is stored in the kernel layer. The application framework layer retrieves the raw input event from the kernel layer and identifies the control corresponding to the input event. Taking a single touch operation as an example, where the corresponding control is the camera application icon, the camera application calls the application framework layer's interface to launch the camera application, and then calls the kernel layer to launch the camera driver, capturing still images or videos through the camera.

[0104] The following is combined with Figure 3 An exemplary illustration of an application scenario of the video data processing method provided by an embodiment of the present invention is shown below.

[0105] This embodiment may include two electronic devices (e.g., such as...) Figure 1 The two electronic devices shown are 10 and 200, respectively. For ease of distinction, they are hereinafter referred to as the first electronic device 100 and the second electronic device 200. The first electronic device 100 and the second electronic device 200 can be located on the same local area network (LAN). For example, they can establish a communication connection via Wi-Fi. The first electronic device 100 includes a first display screen 1001, and the second electronic device 200 includes a second display screen 2001. The first electronic device 100 is the source device, and the second electronic device 200 is the destination device. An application running on the first electronic device 100 can be projected onto the second electronic device 200. This application can be a desktop application, a gallery application, a map application, a music application, a video application, etc.

[0106] The first electronic device 100 and the second electronic device 200 include, but are not limited to, mobile phones, tablets, PCs, laptops, televisions, smart screens, and other devices. Figure 3 The following example illustrates the use of a mobile phone as the first electronic device 100 and a television as the second electronic device 200.

[0107] For example, the first electronic device 100 can open the video application APP1. When the first electronic device 100 projects the video application APP1 onto the second electronic device 200, the second electronic device 200 can generate an application window corresponding to the video application APP1, enabling video playback on the second electronic device 200. Users can control the projected video application APP1 on the second electronic device; for example, users can control the video application APP1 using the remote control of the second electronic device 200.

[0108] like Figure 4 As shown, when the video application APP1 runs on the first electronic device 100, the image frames output by SurfaceFlinger can be encoded by a video encoder, and then the encoded data is encrypted and packaged to generate a video data stream. The first electronic device 100 can send the video data stream to the second electronic device 200 via a Wi-Fi channel. The second electronic device 200 can decrypt and unpack the received video data stream, and then decode the data obtained from the decryption and unpacking by a video decoder. The video decoder can send the decoding result to the display, for example, to the second display screen 2001, so that the application window of the video application APP1 can be drawn on the second electronic device 200.

[0109] Assume the first electronic device 100 includes a first processor. In a screen mirroring scenario, the first electronic device 100 processes image frames for the screen mirroring application, which may include three processing flows: encoding, encryption / packaging, and transmission. The first processor of the first electronic device 100 can execute these three processing flows in parallel using multiple threads, maximizing processor utilization and reducing screen mirroring latency. For example, one processing flow corresponds to one thread, and the first processor executes the encoding, encryption / packaging, and transmission flows in parallel using three threads.

[0110] In some embodiments, the processing of image frames for the screen projection application by the first electronic device 100 can be divided into more than three processing steps or fewer than three processing steps. This application does not limit this; the number of threads can be set according to the number of processing steps. For example, the processing of image frames for the screen projection application by the first electronic device 100 can be divided into four processing steps: encoding, encryption, packaging, and sending. To achieve parallel processing of image frames, each image frame can be encoded into multiple slices, and multi-threaded parallel processing of slices and their intermediate results can be used to further reduce the latency of screen projection.

[0111] For example, an image frame can be divided into multiple macroblocks. A predetermined number of macroblocks can be encoded into slices, thus enabling the image frame to be encoded into multiple slices. Assuming an image frame contains 280 macroblocks, the first 100 macroblocks can be encoded into the first slice, the middle 100 macroblocks into the second slice, and the last 80 macroblocks into the third slice. When an image frame is encoded into multiple slices, the last slice can carry an end-of-image (EOF) marker. The EOF marker indicates that this slice is the last slice of the image frame; for example, the third slice carries an EOF marker.

[0112] In the prior art, the projection of the first electronic device 100 is processed with image frames as the basic unit. In the H.264 or H.265 encoding standard, image frames can be encoded into one or more slices. This application can utilize the slice capability of the video encoder to process the projection of the first electronic device 100 with slices as the basic unit.

[0113] In some embodiments, the video encoder can first encode and segment the image frame based on a default slicing rule to obtain multiple slices. For example, the default slicing rule is to encode the image frame into 3 slices. The first processor can obtain the processing time of the slice in each processing flow, and calculate the optimal number of slices for the video encoder to encode the image frame based on the processing time of the processing flow. Then, the video encoder can encode the image frame based on the optimal number of slices to achieve maximum optimization of projection latency.

[0114] In some embodiments, the video encoder may not first encode and segment the image frames. The first processor can obtain the processing time of the image frame in each processing flow, and then calculate the optimal number of segments for the video encoder to encode and segment the image frame based on the processing time of the processing flow. For example, the first processor can substitute the processing time of each processing flow into a preset segmentation calculation algorithm to obtain the optimal number of segments for the video encoder to encode and segment the image frame.

[0115] like Figure 5 As shown, assuming the first electronic device 100 divides the processing of image frames for screen projection applications into four processing flows: encoding, encryption, packaging, and transmission, the first processor can execute these four processing flows in parallel using four threads. The image frame is encoded and divided into three slices: the first slice, the second slice, and the third slice, where the third slice carries an EOF flag. The optimal image frame segmentation scheme in this scenario satisfies the following conditions: each processing flow does not hinder the data processing of the next processing flow, and for a relatively long period in between, most processing flows are processing data in parallel.

[0116] During the t0 time period, the image frame is encoded to generate the first slice;

[0117] During time period t1, the image frame is encoded to generate the second slice, and the first slice enters the encryption process to be encrypted.

[0118] During time period t2, the image frame is encoded to generate the third slice. The first slice enters the packaging process to package the encrypted result of the first slice. The second slice enters the encryption process to encrypt the second slice.

[0119] That is, the encryption processing time t of the slice needs to be reduced. 加密 Less than or equal to the encoding processing time t 编码 (t 加密 <= t 编码 If t 加密 >t 编码 This can occur when the encoding process obtains the second slice, but the encryption process has not yet finished processing the first slice. Consequently, the second slice must wait for the encryption processing of the first slice to complete before it can enter the encryption process. Similarly, the slice packaging processing time t... 打包 Less than or equal to encryption processing time t 加密 The slice sending and processing time t 发送 Less than or equal to the packaging processing time t 打包 The optimal partitioning scheme satisfies the following conditions: t 发送 <= t 打包 <= t 加密 <= t 编码 The sending process is an instantaneous sending action with a short duration, t. 发送 It will generally be less than t. 打包 t 加密 and t 编码 However, the transmission time fluctuates with network fluctuations.

[0120] Therefore, the optimal number of image frame segments N slice It can be obtained from formula i:

[0121] Where, N slice For T 总编码 / t 打包 The floor value, T 总编码 The time T takes to encode an image frame 总编码 This can be obtained by summing the encoding processing time of each slice in the image frame. In some embodiments, N slice It can also be T 总编码 / t 打包 The rounded-up value is not limited in this application.

[0122] In some embodiments, when equation t 打包 <= t 加密 If it is not true, then t in expression i can be removed. 打包 Revised to t 加密 t 打包 The minimum value in the equation can be used to transform equation i into the following equation ii:

[0123]

[0124] In some embodiments, it is assumed that the first electronic device processes the encoded slice through four steps: Step 1, Step 2, Step 3, and Step 4. The optimal number of image frame segments N is... slice It can also be obtained through the following formula:

[0125] In some embodiments, assuming the first electronic device 100 is a mobile phone, the time spent by the first electronic device 100 to encrypt a frame of image is generally less than 2ms, the time spent to package a frame of image is also generally less than 2ms, and the sending process can even send the data out in less than 1ms, i.e., t 加密 With t 打包 All times are less than 2ms. Based on formula ii, the optimal number of image frames to be segmented for mobile screen mirroring can be obtained through the following formula iii:

[0126] T 总编码 <=3, N slice =1…(iii);

[0127]

[0128]

[0129] The optimal number of image frame segments N obtained based on formula iii slice With the encoding time T of the image frame 总编码 The relationship between them is as follows Figure 6 As shown.

[0130] In some embodiments, the first electronic device 100 can calculate the initial number of segments based on the estimated time required to encode one frame of an image and the time required to encrypt or package one frame of an image. The video encoder encodes and segments the image frame based on the initial number of segments, and then obtains the optimal number of segments N based on the above formula iii. slice .

[0131] In some embodiments, the first electronic device 100 may periodically attempt to modify the optimal number of segments N. slice For example, the first electronic device 100 can calculate the optimal number of segments every first preset time interval. When the first electronic device 100 calculates N at a certain time t', slice The N currently used in the first electronic device 100 slice When they are not equal, the first electronic device 100 can count all N values ​​within a second preset time period (between time t' and time t") starting from time t'. slice The value of N, for example, can be these N sliceThe mean or median can be used as the new optimal number of segments, or other calculation rules can be applied to these N segments. slice The new optimal number of segments is calculated, but this application does not limit this number. The first preset time and the second preset time can be set according to actual needs, and this application does not limit this number. The first preset time is shorter than the second preset time, for example, the first preset time is 10 seconds and the second preset time is 2 minutes.

[0132] The following is combined with Figure 7 This paper presents an exemplary schematic diagram illustrating the data interaction between a first electronic device 100 and a second electronic device 200 in a screen projection scenario, according to an embodiment of the present invention.

[0133] For ease of description below, in the application scenario described below, the first electronic device 100 projects the video application APP1 onto the second electronic device 200 for display. The example provided is that the first electronic device 100 includes a first display screen, a video encoder, a first software framework (FW), and a first wireless communication chip, while the second electronic device 200 includes a second wireless communication chip, a second FW, a video decoder, and a second display screen. The first and second FWs can be built-in software functional architectures of the electronic device's operating system, possessing encryption / packaging and decryption / unpacking functions. The first FW can be executed by the first processor of the first electronic device 100, and the second FW can be executed by the second processor of the second electronic device 200.

[0134] 70. In response to the screen projection command generated by the user's touch operation, the video encoder encodes the image frames of the video application APP1.

[0135] In some embodiments, when a user performs a screen mirroring operation on the first display screen (e.g., clicks the screen mirroring function icon and selects the second electronic device 200 as the target device for screen mirroring), the video encoder can respond to the screen mirroring command generated by the user's operation and encode the image frames of the video application APP1.

[0136] 71. The video encoder sends the encoded results of the image frames to the first FW.

[0137] In some embodiments, when the video encoder completes the encoding of a frame or obtains a slice, the video encoder can output the encoding result to the first FW.

[0138] 72. The first FW encrypts and packages the received encoding result to obtain the first data.

[0139] In some embodiments, the encoding result can refer to the encoding result of a complete image frame or a slice. The first FW encrypts the received encoding result to prevent it from being altered or stolen. For example, the encoding result may contain the device's or user's private data; encrypting the encoding result can prevent the leakage of this private data. The first FW packages the encrypted encoding result, enabling reliable and fast transmission of image frame data in a network environment, while also maintaining synchronized audio and video transmission.

[0140] 73. The first FW sends the first data to the first wireless communication chip.

[0141] In some embodiments, the first wireless communication chip may be a Wi-Fi chip, a Bluetooth chip, or other wireless communication chip.

[0142] 74. The first wireless communication chip sends the first data to the second wireless communication chip.

[0143] In some embodiments, the first wireless communication chip can transmit first data to the second wireless communication chip via a preset communication protocol. The second wireless communication chip can be a Wi-Fi chip, a Bluetooth chip, or other wireless communication chips. For example, if both the first and second wireless communication chips are Wi-Fi chips, the first wireless communication chip can transmit the first data to the second wireless communication chip via the Wi-Fi communication protocol.

[0144] 75. The second wireless communication chip sends the first data to the second FW.

[0145] 76. The second FW decrypts and unpacks the first data to obtain the second data.

[0146] In some embodiments, the second FW decrypts and unpacks the first data to obtain second data that the decoder can recognize.

[0147] 77. The second FW sends the second data to the video decoder.

[0148] 78. The video decoder decodes the second data.

[0149] In some embodiments, the video decoder decodes the second data, and the resulting decoded result (image frame) can be sent to the second display screen.

[0150] 79. The video decoder sends the decoding results to the second display screen.

[0151] In some embodiments, the second display screen can render and display the received image frames, thereby displaying the image frames sent by the first electronic device 100 on the second display screen.

[0152] The following is combined with Figure 8 This paper provides an exemplary illustration of a data interaction diagram in which a first electronic device 100 and a second electronic device 200 perform image frame processing using multi-threading, according to an embodiment of the present invention.

[0153] For ease of description below, in the application scenario described below, the first electronic device 100 projects the video application APP1 onto the second electronic device 200 for display. The first electronic device 100 includes a video encoder, and the second electronic device 200 includes a video decoder. The first electronic device 100 processes image frames through multiple processing steps to transmit them to the second electronic device 200. Each processing step corresponds to one thread, maximizing thread utilization and reducing projection latency.

[0154] 81. The first electronic device 100 queries whether the video encoder has slice encoding capability.

[0155] 82. The first electronic device 100 sends a first query message to the second electronic device 200 to query whether the second electronic device 200 has at least one of the following: slice decoding capability and slice packet assembly capability.

[0156] 83. The second electronic device 200 queries whether the video decoder has slice decoding capability based on the first query information, and queries whether the second FW has slice assembly capability.

[0157] 84. The second electronic device 200 sends the query result information to the first electronic device 100.

[0158] 85. When the first electronic device 100 determines that the second electronic device 200 has slice decoding capability and / or slice packet assembly capability, the first electronic device 100 processes the slice (such as decryption and packaging) and monitors the processing time of each processing step in order to periodically attempt to modify the number of image frames encoded into slices.

[0159] 86. The first electronic device 100 sends the processed (e.g., encrypted and packaged) slice data to the second electronic device.

[0160] 87. The second electronic device 200 decrypts, unpacks, and decodes the received slice data and then displays it.

[0161] like Figure 9As shown, projection scenario 1 represents the time consumed by this application when processing using slices as the basic unit and employing multi-threading. Projection scenario 2 represents the time consumed by the existing solution when processing image frames as the basic unit and employing single-threading. The encoding time for image frames in projection scenarios 1 and 2 is approximately the same. After encoding, projection scenario 1 outputs a slice, while projection scenario 2 outputs a complete encoded image frame. Each processing step in projection scenario 1 processes a slice of data, while each processing step in projection scenario 2 processes a complete frame of encoded image data, such as... Figure 9 As shown, the time required to process one frame of image in screen projection scenario 1 is significantly lower than the time required to process one frame of image in screen projection scenario 2, thus screen projection scenario 1 can reduce screen projection latency.

[0162] The following is combined with Figure 10-12 This paper provides an exemplary schematic diagram of data interaction between a first electronic device 100 and a second electronic device 200, which employs multi-threading for image frame processing, according to another embodiment of the present invention.

[0163] For ease of description below, in the application scenario described below, the first electronic device 100 projects the video application APP1 onto the second electronic device 200 for display. The second electronic device 200 does not support slice decoding. The second electronic device 200 needs to assemble the decrypted and unpacked slices into a complete encoded image frame before decoding. The first electronic device 100 includes a first display screen, a video encoder, a first firmware (FW), and a first wireless communication chip. The second electronic device 200 includes a second wireless communication chip, a second firmware (FW), a video decoder, and a second display screen. The processing of the image frame of the projected application by the first electronic device 100 can be divided into three processing flows: encoding, encryption and packaging, and transmission. The first processor can execute these three processing flows in parallel using three threads. Taking the encoding and segmentation of the image frame into three slices as an example: the first slice, the second slice, and the third slice, where the third slice carries an EOF flag. The processing of image frames by the second electronic device 200 before display can be divided into four processing flows: receiving flow, decryption and unpacking flow, packet assembly flow, and decoding flow. The second processor can execute these four processing flows in parallel through four threads.

[0164] like Figure 10As shown, on the first electronic device 100 side, the encoding process, encryption and packaging process, and sending process can each correspond to a thread, such as an encoding thread, an encryption and packaging thread, and a sending thread. On the second electronic device 200 side, the receiving process, decryption and unpacking process, packet assembly process, and decoding process can each correspond to a thread, such as a receiving thread, a decryption and unpacking thread, a packet assembly thread, and a decoding thread. Multiple threads can process the intermediate results of the slice from the previous thread in parallel, maximizing thread utilization and reducing screen projection latency.

[0165] At time t_0, the video encoder encodes the image frame, and the first slice is in the encoding thread (the first slice has not yet been encoded);

[0166] At time t_1, the first slice is encoded and enters the encryption and packaging thread, which performs encryption and data encapsulation processing on the first slice (e.g., encapsulation can be performed based on the transmission protocol between the first electronic device 100 and the second electronic device 200). At this time, the encoding thread can continue to encode the image frame, and the second slice is in the encoding thread.

[0167] At time t_2, the second slice is encoded, the first slice is encrypted and packaged, and the first slice enters the sending thread. Since the first slice no longer occupies the encryption and packaging thread, the second slice can enter the encryption and packaging thread, that is, the second slice is encrypted and data encapsulated. At this time, the encoding thread can continue to encode the image frame, and the third slice is in the encoding thread.

[0168] At time t_3, the third slice is encoded, the first slice is sent, and the sending thread becomes idle. At this time, the first slice arrives at the second electronic device 200 and enters the receiving thread. The second slice is encrypted and packaged. Since the sending thread is idle, the second slice enters the sending thread. At the same time, since the second slice is encrypted and packaged, the encryption and packaging thread becomes idle. The third slice enters the encryption and packaging thread, that is, the third slice is encrypted and encapsulated.

[0169] At time t_4, the first slice is received and enters the decryption and unpacking thread, which decrypts and unpacks the encrypted packaging result of the first slice. The second slice is sent and arrives at the second electronic device 200. The second slice enters the receiving thread. The third slice is encrypted and packaged and enters the sending thread.

[0170] At time t_5, the first slice is decrypted and unpacked, and enters the packet assembly thread to check if the first slice carries an EOF flag. If it is confirmed that the first slice does not carry an EOF flag, the first slice is cached. The second slice is received and enters the decryption and unpacking thread, which decrypts and unpacks the encrypted packet of the second slice. The third slice is sent and arrives at the second electronic device 200. The third slice enters the receiving thread.

[0171] At time t_6, the second slice is decrypted and unpacked. The second slice enters the packet assembly thread to continue checking whether the second slice carries the EOF flag. When it is confirmed that the second slice does not carry the EOF flag, the first slice and the second slice are concatenated and the concatenated data is buffered. The third slice is received and enters the decryption and unpacking thread, that is, the encrypted packetization result of the third slice is decrypted and unpacked.

[0172] At time t_7, the third slice is decrypted and unpacked. The third slice enters the packet assembly thread and continues to check whether the third slice carries the EOF flag. When it is confirmed that the third slice carries the EOF flag, it is considered that the third slice is the last slice of the current image frame. The third slice is then assembled with the splicing data (first slice + second slice) cached at time t_6 to obtain a complete encoded image frame.

[0173] At time t_8, the complete encoded image frame en_frame obtained by splicing enters the decoding thread for decoding processing, and the decoding result can be sent for display.

[0174] In some embodiments, the second display 2001 can display the decoding result sent by the decoder, and the user can see the image frame when the next vertical synchronization signal arrives.

[0175] In some embodiments, at time t_6, the first slice and the second slice may not be combined into a single packet; the second slice can be cached. At time t_7, the first slice, the second slice, and the third slice are then combined into a single packet to obtain a complete encoded image frame.

[0176] like Figure 11As shown, the video encoder can encode and segment image frame 1 into a first slice_1, a second slice_2, and a third slice_3, with the third slice_3 carrying an EOF flag. The video encoder can then send the encoding results to the first framework (FW). For example, the video encoder encodes image frame 1 sequentially to obtain the first slice_1, the second slice_2, and the third slice_3. It first sends the first slice_1 to the first FW, then the second slice_2, and finally the third slice_3.

[0177] The first FW can encrypt and package the received slices. For example, when the first FW receives the first slice_1 sent by the video encoder, it encrypts and packages the first slice_1 to obtain a first data packet f1s1_ts, which can then be sent to the first wireless communication chip. When the first FW receives the second slice_2 sent by the video encoder, it encrypts and packages the second slice_2 to obtain a second data packet f1s2_ts, which can then be sent to the first wireless communication chip. When the first FW receives the third slice_3 sent by the video encoder, it encrypts and packages the third slice_3 to obtain a third data packet f1s3_ts, which can then be sent to the first wireless communication chip.

[0178] Upon receiving a first data packet f1s1_ts, the first wireless communication chip forwards the first data packet f1s1_ts to the second wireless communication chip, thereby enabling the second wireless communication chip to receive the first data packet f1s1_ts. Similarly, upon receiving a second data packet f1s2_ts, the first wireless communication chip forwards the second data packet f1s2_ts to the second wireless communication chip, thereby enabling the second wireless communication chip to receive the second data packet f1s2_ts. Likewise, upon receiving a third data packet f1s3_ts, the first wireless communication chip forwards the third data packet f1s3_ts to the second wireless communication chip, thereby enabling the second wireless communication chip to receive the third data packet f1s3_ts.

[0179] The second wireless communication chip can forward the received first data packet f1s1_ts to the second FW. The second FW can decrypt and unpack the first data packet f1s1_ts to obtain the first slice slice_1. The second FW can detect whether the first slice slice_1 carries the EOF flag. The second wireless communication chip can also forward the received second data packet f1s2_ts to the second FW. The second FW can decrypt and unpack the second data packet f1s2_ts to obtain the second slice slice_2. The second FW can detect whether the second slice slice_2 carries the EOF flag. The second wireless communication chip can also forward the received third data packet f1s3_ts to the second FW. The second FW can decrypt and unpack the third data packet f1s3_ts to obtain the third slice slice_3. The second FW can detect whether the third slice slice_3 carries the EOF flag. When the second FW determines that the third slice_3 carries the eof flag, the second FW concatenates the first slice_1, the second slice_2, and the third slice_3 to obtain a complete encoded image frame frame1_enc.

[0180] The second FW can send the complete encoded image frame1_enc to the video decoder. The video decoder decodes the encoded image frame1_enc to obtain the image frame1. The image frame1 obtained by the video decoder can be sent to the second display screen for rendering and display.

[0181] like Figure 12 As shown, the second FW can determine whether the slice of the current image frame has been completely received by detecting whether the first slice_1, the second slice_2, and the third slice_3 obtained by decryption and unpacking carry the EOF flag. When it is determined that the third slice_3 carries the EOF flag, the second FW can reassemble the first slice_1, the second slice_2, and the third slice_3 to obtain a complete encoded image frame frame1_enc, and then send the encoded image frame1_enc to the video decoder for decoding to obtain the image frame frame1.

[0182] The following is combined with Figure 13 This paper provides an exemplary schematic diagram of data interaction between a first electronic device 100 and a second electronic device 200, which employs multi-threading for image frame processing, according to another embodiment of the present invention.

[0183] For ease of description below, in the application scenario described below, the first electronic device 100 projects the video application APP1 onto the second electronic device 200 for display. The second electronic device 200 supports slice decoding and can decode the decrypted and unpacked slices without needing to reassemble them. The first electronic device 100 includes a first display screen, a video encoder, a first firmware (FW), and a first wireless communication chip. The second electronic device 200 includes a second wireless communication chip, a second firmware (FW), a video decoder, and a second display screen. The processing of image frames from the projected application by the first electronic device 100 can be divided into three processing flows: encoding, encryption and packaging, and transmission. The first processor can execute these three processing flows in parallel using three threads. Taking the encoding and segmentation of an image frame into three slices as an example: the first slice, the second slice, and the third slice, where the third slice carries an EOF flag. The processing of image frames by the second electronic device 200 before display can be divided into three processing flows: receiving flow, decryption and unpacking flow, and decoding flow. The second processor of the second electronic device 200 can execute these three processing flows in parallel through three threads.

[0184] like Figure 13 As shown, on the first electronic device 100 side, the encoding process, encryption and packaging process, and sending process can each correspond to a thread, such as an encoding thread, an encryption and packaging thread, and a sending thread. On the second electronic device 200 side, the receiving process, decryption and unpacking process, and decoding process can each correspond to a thread, such as a receiving thread, a decryption and unpacking thread, and a decoding thread. Multiple threads can process the intermediate results of the slice from the previous thread in parallel, maximizing thread utilization and reducing screen projection latency.

[0185] At time t_0, the video encoder encodes the image frame, and the first slice is in the encoding thread (the first slice has not yet been encoded);

[0186] At time t_1, the first slice is encoded and enters the encryption and packaging thread, which performs encryption and data encapsulation processing on the first slice (e.g., encapsulation can be performed based on the transmission protocol between the first electronic device 100 and the second electronic device 200). At this time, the encoding thread can continue to encode the image frame, and the second slice is in the encoding thread.

[0187] At time t_2, the second slice is encoded, the first slice is encrypted and packaged, and the first slice enters the sending thread. Since the first slice no longer occupies the encryption and packaging thread, the second slice can enter the encryption and packaging thread, that is, the second slice is encrypted and data encapsulated. At this time, the encoding thread can continue to encode the image frame, and the third slice is in the encoding thread.

[0188] At time t_3, the third slice is encoded, the first slice is sent, and the sending thread becomes idle. At this time, the first slice arrives at the second electronic device 200 and enters the receiving thread. The second slice is encrypted and packaged. Since the sending thread is idle, the second slice enters the sending thread. At the same time, since the second slice is encrypted and packaged, the encryption and packaging thread becomes idle. The third slice enters the encryption and packaging thread, that is, the third slice is encrypted and encapsulated.

[0189] At time t_4, the first slice is received and enters the decryption and unpacking thread, which decrypts and unpacks the encrypted packaging result of the first slice. The second slice is sent and arrives at the second electronic device 200. The second slice enters the receiving thread. The third slice is encrypted and packaged and enters the sending thread.

[0190] At time t_5, the first slice is decrypted and unpacked, and enters the decoding thread to check if it carries an EOF flag. If it is confirmed that the first slice does not carry an EOF flag, the decoding result of the first slice is cached. The second slice is received and enters the decryption and unpacking thread, which decrypts and unpacks the encrypted and packaged result of the second slice. The third slice is sent and arrives at the second electronic device 200. The third slice enters the receiving thread.

[0191] At time t_6, the second slice is decrypted and unpacked. The second slice enters the decoding thread to continue checking whether the second slice carries the EOF flag. When it is confirmed that the second slice does not carry the EOF flag, the decoding result of the second slice is cached. The third slice is received and enters the decryption and unpacking thread, that is, the encrypted and packaged result of the third slice is decrypted and unpacked.

[0192] At time t_7, the third slice is decrypted and unpacked. The third slice enters the decoding thread and continues to check whether the third slice carries the EOF flag. When it is confirmed that the third slice carries the EOF flag, it is considered that the third slice is the last slice of the current image frame. All the decoded data cached before the third slice are sent to the display, that is, the decoding results of the first slice, the second slice, and the third slice are sent to the display.

[0193] In some embodiments, the second display 2001 may display an image frame when the next vertical synchronization signal arrives.

[0194] In some embodiments, at time t_6, the first slice and the second slice may not be combined into a single packet; the second slice can be cached. At time t_7, the first slice, the second slice, and the third slice are then combined into a single packet to obtain a complete encoded image frame.

[0195] Reference Figure 14 As shown, this application provides a video data processing method applied to a first electronic device 100. The first electronic device 100 can project several image frames onto a second electronic device 200 for display. In this embodiment, the video data processing method may include:

[0196] 141. Encode each image frame in a number of image frames based on the first number of slices to obtain the first number of slices.

[0197] In some embodiments, the first electronic device 100 may encode each image frame based on a first number of segments to obtain a first number of slices. For example, if the first number of segments is 3, the first electronic device 100 may encode each image frame to obtain 3 slices: a first slice, a second slice, and a third slice, wherein the third slice carries an EOF tag.

[0198] In some embodiments, the first electronic device 100 may be configured with multiple parallel threads for processing the slice. Examples include an encoding thread, an encryption thread, a packaging thread, and a sending thread.

[0199] The first segmentation number can be a default number preset before encoding the image frame; alternatively, it can be calculated based on the processing time of each thread processing the image frame over a period of time, such as dividing the average encoding time of the image frame by the average packing time or average encryption time (rounding up or down is possible). The first segmentation number can also refer to the segmentation number updated using the scheme of this application.

[0200] In some embodiments, before encoding an image frame into several slices, the first electronic device 100 may query whether its own video decoder has slice encoding capability, and whether the second electronic device 200 has at least one of slice decoding capability or slice assembly capability. If the video decoder has slice encoding capability, and the second electronic device 200 has slice decoding capability or slice assembly capability, the first electronic device 100 encodes each image frame into several slices based on a first number of segments. Otherwise, the first electronic device 100 encodes each image frame but does not encode each image frame into several slices.

[0201] 142. The processed data obtained from processing the slice is sent to the second electronic device 200.

[0202] In some embodiments, the processed data may refer to the data obtained by encrypting and packaging the slice, and the first electronic device 100 may send the processed data to the second electronic device 200 through a sending thread.

[0203] 143. Obtain the processing time of each thread in multiple threads when processing a slice.

[0204] In some embodiments, since the sending thread is an instantaneous sending action, the processing time of the sending thread can be ignored. The processing time of each of the multiple threads for processing the slice may include encoding time, encryption time, and packaging time. The first electronic device 100 can obtain the processing time of the encoding thread, the processing time of the encryption thread, and the processing time of the packaging thread by monitoring the slice output time of the video encoder and the encryption time and packaging time of the slice.

[0205] 144. Update the number of the first slice based on the processing time of each thread on the slice.

[0206] In some embodiments, the first electronic device 100 may periodically attempt to update the first segmentation quantity. For example, the first electronic device 100 may calculate the segmentation quantity every first preset time interval. At a certain time t', the first electronic device 100 may calculate a second segmentation quantity based on the processing time of each thread processing the same slice. When the second segmentation quantity is not equal to the first segmentation quantity, the first electronic device 100 may further obtain multiple slices processed within a second preset time interval starting from that time t', and multiple second segmentation quantities corresponding to the multiple slices (e.g., calculating the segmentation quantity every first preset time interval within the second preset time interval). The first electronic device 100 may obtain a third segmentation quantity based on the multiple second segmentation quantities, and update the first segmentation quantity to the third segmentation quantity, so that unencoded image frames in a number of image frames are subsequently encoded and segmented using the third segmentation quantity. When the second segmentation quantity is equal to the first segmentation quantity, it indicates that the first segmentation quantity is the optimal segmentation quantity, and there is no need to attempt to update the first segmentation quantity.

[0207] For example, the first electronic device 100 can obtain the encoding processing time consumed by the encoding thread in encoding an image frame (the image frame processed by the first electronic device 100 at time t') into a first number of slices, and obtain the encryption processing time consumed by the encryption thread and the packing thread in encrypting a slice in the image frame and the packing processing time consumed in packing. The first electronic device 100 can calculate a second number of slices based on the encoding processing time and the smaller of the encryption processing time and the packing processing time.

[0208] In some embodiments, the first electronic device 100 may perform an average calculation on a plurality of second segmentation quantities, and round up or down the calculated average value to obtain a third segmentation quantity. The first electronic device 100 may also use the median value among the plurality of second segmentation quantities as the third segmentation quantity.

[0209] The above video data processing method encodes the image frames displayed on the screen into several slices, processes them using slices as the basic unit, and reduces the processing time of slices by processing slices through parallel threads. At the same time, by monitoring the slice processing time of each thread and determining the optimal number of image frame segments based on the monitored processing time, the image frames are encoded based on the optimal number of segments, which can maximize the processing efficiency of threads and reduce the screen projection latency.

[0210] Reference Figure 15 As shown in the figure, this application provides a video data processing method applied to a second electronic device 200.

[0211] In this embodiment, the video data processing method may include:

[0212] 151. Receive processing data sent by the first electronic device 100.

[0213] In some embodiments, the second electronic device 200 may be configured with multiple parallel threads for processing the slice. These may include a receiving thread, a decryption thread, an unpacking thread, and a decoding thread. The processed data received by the second electronic device 200 may refer to data obtained by encrypting and packaging the slice.

[0214] 152. Process the data to obtain the slice corresponding to the processed data.

[0215] In some embodiments, the second electronic device 200 can decrypt and unpack the processed data to obtain a slice corresponding to the processed data.

[0216] 153. The processed slices are spliced ​​together to obtain the encoded image frame corresponding to the slices.

[0217] In some embodiments, when the second electronic device 200 lacks slice decoding capability but possesses slice splicing capability, the second electronic device 200 can first splice the processed multiple slices to obtain a complete encoded image frame, and then transmit the encoded image frame to the video decoder for decoding processing. When the second electronic device 200 processes a slice, it can detect whether the processed slice contains an EOF tag. If the slice does not contain an EOF tag, it indicates that the slice is not the last slice of the image frame, and the slice is cached. If the slice contains an EOF tag, it indicates that the slice is the last slice of the image frame, and the second electronic device 200 can splice the slice containing the EOF tag with all previously cached slices to obtain a complete encoded image frame.

[0218] 154. Decode and display the encoded image frames.

[0219] In some embodiments, when a complete encoded image frame is obtained by splicing, the encoded image frame can be sent to a video decoder for decoding processing, and the decoded image frame can be displayed on a second display screen 2001.

[0220] In some embodiments, when the video decoder of the second electronic device has slice decoding capability, there is no need to splice the slices; the processed slices can be sent to the video decoder for decoding.

[0221] In some embodiments, the second electronic device can detect whether the processed slice contains an EOF flag. When it is detected that the processed slice does not contain an EOF flag, it indicates that the slice is not the last slice of the image frame, and the slice can be decoded and the decoding result can be cached. When it is detected that the processed slice contains an EOF flag, it indicates that the slice is the last slice of the image frame, and the slice can be decoded, and the decoding result of the slice is displayed along with the decoding results of all previously cached slices.

[0222] The above video data processing method encodes the image frames displayed on the screen into several slices, processes them using slices as the basic unit, and processes the slices through parallel threads, thereby reducing the processing time of the slices and thus reducing the screen projection latency. At the same time, it can choose to perform packet decoding or direct decoding on the slices according to the decoding capability of the electronic device.

[0223] refer to Figure 16 This is a schematic diagram of the hardware structure of the first electronic device 100 provided in an embodiment of this application. Figure 16 As shown, the first electronic device 100 may include a first display screen 1001, a first processor 1002, a first memory 1003, and a first communication bus 1004. The first memory 1003 stores one or more first computer programs 1005. The one or more first computer programs 1005 are configured to be executed by the first processor 1002. The one or more first computer programs 1005 include instructions that can be used to perform actions such as... in the first electronic device 100. Figure 14 The video data processing method described above.

[0224] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the first electronic device 100. In other embodiments, the first electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements.

[0225] refer to Figure 17 This is a schematic diagram of the hardware structure of the second electronic device 200 provided in an embodiment of this application. Figure 17As shown, the second electronic device 200 may include a second display screen 2001, a second processor 2002, a second memory 2003, and a second communication bus 2004. The second memory 2003 stores one or more second computer programs 2005. The one or more first and second computer programs 2005 are configured to be executed by the second processor 2002. The one or more second computer programs 2005 include instructions that can be used to implement actions such as... in the second electronic device 200. Figure 15 The video data processing method described above.

[0226] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the second electronic device 200. In other embodiments, the second electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements.

[0227] This embodiment also provides a computer storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the video data processing method described above.

[0228] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the video data processing method described in the above embodiment.

[0229] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component, or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the video data processing methods in the above-described method embodiments.

[0230] In this embodiment, the first electronic device, the second electronic device, the computer storage medium, the computer program product, or the chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0231] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0232] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are illustrative. For instance, the division of modules or units is a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0233] The unit described as a separate component may or may not be physically separate. The component shown as a unit can be one physical unit or multiple physical units, that is, it can be located in one place or distributed in multiple different places. Some or all of the units can be selected to achieve the purpose of the solution in this embodiment according to actual needs.

[0234] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0235] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0236] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be covered within the scope of protection of this application.

Claims

1. A video data processing method, applied to a first electronic device, wherein the first electronic device projects several image frames onto a second electronic device for display, characterized in that, The video data processing method includes: The first image frame in the plurality of image frames is encoded based on the first number of segments to obtain a first number of stripes corresponding to the first image frame, wherein the first electronic device includes a plurality of parallel threads for processing the stripes; The processed data obtained from processing the strip is sent to the second electronic device; Obtain the processing time of each of the plurality of threads for processing the stripe; The first segmentation number is updated based on the processing time of each thread for the stripe.

2. The video data processing method as described in claim 1, characterized in that, The plurality of image frames further includes a second image frame, and the video data processing method further includes: The processing time of each of the plurality of threads for processing the second image frame is obtained, and the first number of segments is calculated based on the processing time of each thread for processing the second image frame.

3. The video data processing method as described in claim 1, characterized in that, The step of updating the first segmentation number based on the processing time of each thread for the stripe includes: The second number of segments is calculated based on the processing time of each thread processing the same stripe; Obtain multiple strips processed within a first preset time period, and multiple second segmentation numbers corresponding to the multiple strips; A third segmentation number is obtained based on the plurality of second segmentation numbers, and the first segmentation number is updated to the third segmentation number; The unencoded image frames in the plurality of image frames are encoded based on the third segmentation number.

4. The video data processing method as described in claim 3, characterized in that, The process of obtaining the third segmentation number based on the plurality of second segmentation numbers includes: The average value of the plurality of second segmentation quantities is calculated, and the calculated average value is rounded up or down to obtain the third segmentation quantity; or The median of the plurality of second segmentation numbers is taken as the third segmentation number.

5. The video data processing method as described in claim 3 or 4, characterized in that, The multiple threads include an encoding thread, an encryption thread, and a packaging thread. The calculation of the second number of segments based on the processing time of each thread processing the same stripe includes: The encoding processing time for the encoding thread to encode any one of the plurality of image frames into the first number of stripes is obtained. Obtain the encryption processing time and packaging processing time of the encryption thread and the packaging thread for a strip in any image frame; The second number of segments is calculated based on the smaller of the encoding processing time, the encryption processing time, and the packaging processing time.

6. The video data processing method as described in claim 3 or 4, characterized in that, The acquisition of multiple strips processed within a first preset time period includes: When the calculated second segmentation number is greater than or less than the first segmentation number, multiple strips processed within the first preset time period are obtained.

7. The video data processing method according to any one of claims 1 to 4, characterized in that, The first electronic device includes a video encoder, and the encoding of the first image frame among the plurality of image frames based on a first segmentation number includes: Check whether the video encoder has stripe encoding capability; Check whether the second electronic device has at least one of the following: stripe decoding capability and stripe packet assembly capability; When the video encoder has the strip encoding capability and the second electronic device has the strip decoding capability or the strip packetization capability, the first image frame among the plurality of image frames is encoded based on the first number of segments.

8. A video data processing method, applied to a first electronic device and a second electronic device, wherein the first electronic device projects several image frames onto the second electronic device for display, characterized in that, The video data processing method includes: The first electronic device encodes a first image frame among the plurality of image frames based on a first number of segments to obtain a first number of stripes corresponding to the first image frame, wherein the first electronic device includes a plurality of parallel-running first threads for processing the stripes; The first electronic device sends the processed data obtained from processing the strip to the second electronic device; The first electronic device acquires the processing time of each of the plurality of first threads for processing the strip; The first electronic device updates the first segmentation number based on the processing time of each first thread for the strip; The second electronic device receives the processed data sent by the first electronic device; The second electronic device processes the processed data to obtain strips corresponding to the processed data; The second electronic device stitches together the processed multiple stripes to obtain coded image frames corresponding to the multiple stripes; The second electronic device decodes and displays the encoded image frame.

9. The video data processing method as described in claim 8, characterized in that, The process of splicing together the multiple strips obtained from the processing includes: Detect whether the strips obtained from the processing contain end markers; When it is detected that the stripe obtained by the process does not contain the end marker, the stripe obtained by the process is cached; When it is detected that the stripe obtained by the processing contains the end mark, the stripe containing the end mark is concatenated with all previously cached stripes.

10. The video data processing method as described in claim 8 or 9, characterized in that, The second electronic device includes multiple parallel-running second threads for processing the strips, wherein splicing the processed strips includes: When the second electronic device has strip splicing capability but not strip decoding capability, it splices the multiple strips obtained from the processing.

11. The video data processing method as described in claim 8 or 9, characterized in that, The video data processing method further includes: When the second electronic device has stripe decoding capability, it decodes and displays the processed stripe.

12. The video data processing method as described in claim 11, characterized in that, The decoding and display processing of the obtained stripes includes: Detect whether the strips obtained from the processing contain end markers; When it is detected that the stripe obtained by the processing does not contain the end marker, the stripe obtained by the processing is decoded and the decoding result is cached; When it is detected that the processed stripe contains the end marker, the stripe containing the end marker is decoded, and the decoding result of the stripe containing the end marker is displayed together with the decoding results of all previously cached stripes.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on an electronic device, cause the electronic device to perform the video data processing method as described in any one of claims 1 to 7.

14. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory being used to store instructions, and the processor being used to invoke the instructions in the memory, causing the electronic device to perform the video data processing method as described in any one of claims 1 to 7.

15. A chip coupled to a memory in an electronic device, characterized in that, The chip is used to control the electronic device to perform the video data processing method as described in any one of claims 1 to 7.

Images