Method and system for expanding computing power for television end application based on mobile phone computing power

Abstract

The invention provides a method and a system for expanding computing power for a television end application based on mobile phone computing power, and relates to the technical field of intelligent equipment, the method comprises the following steps: establishing wireless communication connection between a television and a mobile phone, the wireless communication connection comprising WiFi direct connection automatically searched and established by the application, or a local area network connection established by manually inputting the IP address of the equipment by a user; when a television runs an application, each task in the running of the application is identified and screened, the attribute feature of each task is extracted through a task feature matching algorithm, and the attribute feature of each task is compared with a preset high-computing-power task feature library to obtain a feature matching result, and the computing power consumption scale of each task is predicted. And identifying tasks which are difficult to process by the television and need to be supported by high computing power through the matching result and the consumption scale, wherein the tasks which need to be supported by the high computing power comprise 3D picture rendering or high-definition video decoding. According to the invention, the smooth operation of the high-computing-power application of the television end is realized.

Description

Technical Field

[0001] This invention relates to the field of smart device technology, and in particular to a method and system for expanding the computing power of television applications based on the computing power of mobile phones. Background Technology

[0002] As smart TVs upgrade to become the central hub for home intelligence, high-performance applications such as 3D games and 8K high-definition video are becoming increasingly popular. However, the traditional development logic of TVs, which prioritizes display over computation, has resulted in a significant lag in chip performance. The CPU and GPU performance of mainstream low-to-mid-range TVs is only one-third to one-fifth that of flagship mobile phones. Furthermore, storage technology largely uses the eMMC standard discarded by mobile phones, which is insufficient to support high computing power demands. Currently, the industry is trying to alleviate this problem through hardware upgrades, application simplification, or cloud services: hardware upgrades require users to purchase high-end TVs or high-performance boxes, which are costly; application simplification requires removing special effects and reducing model accuracy, sacrificing the core experience; and cloud services rely on stable... Low-latency networks, especially home Wi-Fi, are susceptible to channel interference and wall obstructions, leading to screen freezes and operational delays. For example, when a user runs a TV version on an entry-level smart TV equipped with a G31 GPU, even after developers have reduced the number of triangles in the 3D model and disabled lighting effects, the frame rate still drops below the original frame rate during team battles. Switching to cloud gaming mode, even with Wi-Fi enabled, network fluctuations during peak hours still cause significant delays in input commands, resulting in a severe disconnect between skill releases and visual feedback, completely failing to meet the demands of game interaction. Solutions relying on hardware upgrades or cloud computing power either raise the bar for users or are limited by network conditions, failing to fundamentally resolve the compatibility issue between low-configuration TVs and high-computing applications. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method and system for expanding the computing power of TV applications based on the computing power of mobile phones, so as to realize the smooth operation of high computing power applications on TVs.

[0004] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: In a first aspect, a method for expanding the computing power of TV applications based on mobile phone computing power, the method comprising: Establish a wireless communication connection between the TV and the mobile phone. The wireless communication connection includes a WiFi Direct connection that is automatically searched and established by the application, or a local area network connection that is established by the user manually entering the device's IP address. When the TV is running the application, the various tasks in the application are identified and filtered. The attribute features of each task are extracted by the task feature matching algorithm, and the attribute features of each task are compared with the preset high computing power task feature library to obtain the feature matching results. The computing power consumption scale of each task is predicted. Based on the matching results and consumption scale, tasks that are difficult for the TV to process and require high computing power support are identified. Tasks that require high computing power support include 3D image rendering or high-definition video decoding. Tasks requiring high computing power are distributed from the TV to the mobile phone, which then performs the computation and processing to obtain 3D image rendering results. The 3D image rendering results are then encoded and compressed to obtain a compressed video stream. Based on compressed video streams, mobile phones transmit compressed video streams to TVs via wireless communication connections. The TVs receive and decode the compressed video streams to obtain the picture, and then display the picture to the user. Based on the television display, the system receives user commands on the television and transmits these commands to the mobile phone via wireless communication. The application status is updated and the calculation process is re-executed based on the operation instructions received by the mobile phone. The new screen result is encoded and compressed to obtain a new compressed video stream, which is then transmitted to the TV to complete the two-way interaction between the user and the TV application.

[0005] Furthermore, a wireless communication connection is established between the TV and the mobile phone. This wireless communication connection includes a WiFi Direct connection automatically searched and established by the application, or a local area network connection established by the user manually entering the device's IP address, including: Based on the TV application and the mobile application being launched and discovering each other within the local area network, the TV or mobile phone initiates a connection request based on mutual discovery, and in response to the connection request, authentication and authorization are performed between the TV and the mobile phone. Once authentication and authorization are successful, a wireless communication connection session is established between the TV and the mobile phone for transmitting compressed video streams and operation commands.

[0006] Furthermore, during the application's operation on the television, various tasks are identified and filtered. Attribute features of each task are extracted using a task feature matching algorithm, and these features are compared with a pre-defined high-computing-power task feature library to obtain feature matching results. The computational power consumption of each task is predicted, and tasks that are difficult for the television to process and require high computational power are identified based on the matching results and consumption scale. These tasks include 3D image rendering or high-definition video decoding, including: Step 2.1: When running the application on the TV, monitor and extract the attribute characteristics of each task to be processed in the application in real time. The attribute characteristics include task type identifier, amount of data to be processed, and real-time priority. Step 2.2: Match the extracted attribute features with a preset high-computing-power task feature library. The high-computing-power task feature library stores benchmark features related to 3D image rendering and high-definition video decoding tasks. For tasks that are successfully matched by the high-computing-power task feature library, predict the computing power consumption required for execution based on the attribute features. Step 2.4: By combining the matching results with the predicted computing power consumption scale and comparing it with the current remaining computing power resources on the TV, tasks that require high computing power support are identified.

[0007] Furthermore, tasks requiring high computing power are distributed from the television to the mobile phone, where the computation is performed to obtain 3D image rendering results. These 3D image rendering results are then encoded and compressed to obtain a compressed video stream, including: The task data and processing instructions for the tasks to be assigned are encapsulated into a task request by the television and sent to the mobile phone via wireless communication connection. Based on the task request received by the mobile phone, the task data and processing instructions are parsed out; the mobile phone calls local computing resources to perform calculations and processing on the task data to obtain the original 3D image rendering result. Based on the original 3D image rendering results, the mobile phone uses a hardware encoder to encode and compress the original 3D image rendering results in real time to obtain a compressed video stream.

[0008] Furthermore, based on the compressed video stream, the mobile phone transmits the compressed video stream to the television via a wireless communication connection. The television receives and decodes the compressed video stream to obtain the picture, which is then displayed to the user, including: Based on the obtained compressed video stream, the mobile phone encapsulates the compressed video stream into real-time transmission protocol data packets and sends them through the established wireless communication connection session. Based on the real-time transmission protocol data packets received by the television, the received data packets are reassembled to obtain complete compressed video stream data; Based on the obtained complete compressed video stream data, the TV calls the video decoding module to decode the complete compressed video stream data and restore the original picture frame sequence; Based on the original frame sequence, the original frame sequence is submitted to the graphics rendering pipeline for rendering processing via the television, and finally the rendered image is output to the display device to be presented to the user.

[0009] Furthermore, based on the television display, the system receives user commands on the television and transmits these commands to a mobile phone via wireless communication, including: During the process of displaying the image on the television, the user's input commands are captured through the input device interface on the television. The operation instructions are encapsulated into low-latency control data packets, which contain operation type and parameter information. These low-latency control data packets are transmitted to the mobile phone in real time via an established wireless communication connection. Furthermore, based on the operation instructions received from the mobile phone, the application state is updated and the calculation process is re-executed. The resulting new screen image is encoded and compressed to obtain a new compressed video stream, which is then transmitted to the television, completing the two-way interaction between the user and the television application. This includes: Based on the parsed operation type and parameter information, the mobile phone updates the internal logic state of the currently running application; Based on the updated internal logic state, the corresponding graphics calculations and rendering processes are re-executed on the mobile phone to obtain a new original image result; The new original image result is encoded and compressed in real time by a mobile phone to obtain a new compressed video stream; The mobile phone transmits a new compressed video stream to the TV via a wireless communication connection session, triggering an update of the TV screen and completing a full two-way interactive loop between the user and the TV application.

[0010] Secondly, a method and system for expanding the computing power of TV applications based on mobile phone computing power includes: The acquisition module is used to establish a wireless communication connection between the TV and the mobile phone. The wireless communication connection includes a WiFi direct connection automatically searched and established by the application, or a local area network connection established by the user manually entering the device IP address. When the application is running on the TV, the module identifies and filters the various tasks running in the application, extracts the attribute features of each task through a task feature matching algorithm, and compares the attribute features of each task with a preset high computing power task feature library to obtain feature matching results. The module predicts the computing power consumption scale of each task and identifies tasks that are difficult for the TV to process and require high computing power support based on the matching results and consumption scale. Tasks requiring high computing power support include 3D image rendering or high-definition video decoding. The compression module is used to distribute tasks requiring high computing power from the TV to the mobile phone, where the mobile phone performs the computation and processing to obtain 3D image rendering results. The 3D image rendering results are then encoded and compressed to obtain a compressed video stream. The transmission module is used to transmit compressed video streams from mobile phones to televisions via wireless communication connections. The televisions receive and decode the compressed video streams to obtain the picture and display the picture to the user. Based on the television display, the system receives user commands on the television and transmits these commands to the mobile phone via wireless communication. The processing module updates the application status and re-executes the calculation processing based on the operation instructions received from the mobile phone. It encodes and compresses the new image result to obtain a new compressed video stream, and transmits the new compressed video stream to the TV, thus completing the two-way interaction between the user and the TV application.

[0011] Thirdly, a computing device includes: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the method.

[0012] Fourthly, a computer-readable storage medium storing a program that, when executed by a processor, implements the method.

[0013] The above-described solution of the present invention has at least the following beneficial effects: Because it adopts a wireless communication connection between the TV and the mobile phone via direct WiFi connection or manual input of the device IP, simple tasks such as UI rendering and receiving remote control operation commands are left to the TV when the application is running. High-computing tasks such as 3D image rendering and high-definition video decoding are assigned to the mobile phone. The mobile phone uses local computing resources to complete the calculation, encodes and compresses the image results, and transmits the compressed video stream to the TV through real-time transmission protocols. At the same time, the TV captures user operation commands and encapsulates them into low-latency control data packets and sends them back to the mobile phone. The mobile phone updates the application status according to the operation commands, recalculates, encodes, and transmits the new image to achieve two-way interaction. Therefore, it overcomes the technical problems of existing hardware upgrade solutions that require additional equipment purchases, resulting in high costs; application simplification solutions that reduce image quality and special effects, resulting in poor user experience; and cloud service solutions that rely on stable high-bandwidth networks and increase bandwidth costs for developers. It achieves the following: no need for users to purchase additional equipment to lower the usage threshold; ensures high-definition image quality and smooth operation experience of applications; and fully utilizes the idle computing power of mobile phones to improve resource utilization efficiency. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating a method for extending the computing power of TV applications based on mobile phone computing power, as provided in an embodiment of the present invention.

[0015] Figure 2 This is a schematic diagram of a system that extends the computing power of a TV application based on the computing power of a mobile phone, according to an embodiment of the present invention.

[0016] Figure 3 It is an interactive flowchart that expands the computing power of TV applications based on the computing power of mobile phones. Detailed Implementation

[0017] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0018] like Figure 1 As shown, an embodiment of the present invention proposes a method for expanding the computing power of TV applications based on mobile phone computing power, the method comprising the following steps: Step 1: Establish a wireless communication connection between the TV and the mobile phone. The wireless communication connection includes a WiFi Direct connection that is automatically searched and established by the application, or a local area network connection that is established by the user manually entering the device's IP address. Step 2: When the application is running on the TV, the various tasks in the application are identified and filtered. The attribute features of each task are extracted by the task feature matching algorithm, and the attribute features of each task are compared with the preset high computing power task feature library to obtain the feature matching results. The computing power consumption scale of each task is predicted. Based on the matching results and consumption scale, tasks that are difficult for the TV to process and require high computing power support are identified. Tasks that require high computing power support include 3D image rendering or high-definition video decoding. Step 3: Distribute tasks requiring high computing power from the TV to the mobile phone, where the mobile phone performs the computation and processing to obtain the 3D image rendering result. Then, encode and compress the 3D image rendering result to obtain the compressed video stream. Step 4: Based on the compressed video stream, the mobile phone transmits the compressed video stream to the TV via wireless communication connection. The TV receives and decodes the compressed video stream to obtain the picture and displays the picture to the user. Step 5: Based on the TV display, receive the user's operation commands on the TV and transmit the operation commands to the mobile phone via wireless communication connection; Step 6: Based on the operation instructions received by the mobile phone, update the application status and re-execute the calculation process. Encode and compress the new screen result to obtain a new compressed video stream. Transmit the new compressed video stream to the TV to complete the two-way interaction between the user and the TV application.

[0019] In this embodiment of the invention, a wireless communication connection between the TV and the mobile phone is established by first automatically searching for and establishing a WiFi direct connection through the application or by the user manually entering the device IP to establish a local area network connection. Then, when the TV is running an application, high-computing tasks that are difficult for the TV to handle, such as 3D image rendering and high-definition video decoding, are identified and assigned to the mobile phone. After the mobile phone completes the calculation and processing, it encodes and compresses the image result to obtain a compressed video stream. Then, the mobile phone transmits the compressed video stream to the TV for decoding and display through a wireless connection. At the same time, the TV receives user operation commands and sends them back to the mobile phone. The mobile phone updates the application status according to the commands and recalculates, encodes and compresses a new video stream, and transmits it to the TV to complete two-way interaction. Therefore, it overcomes the technical problems of existing hardware upgrade solutions that require additional equipment purchases, resulting in high costs; application simplification solutions that reduce image quality and special effects, resulting in poor user experience; and cloud service solutions that rely on stable and high-bandwidth networks. Thus, it achieves the goal of not requiring users to purchase additional equipment to lower the usage threshold, ensuring high-definition image quality and smooth operation experience for TV applications, and fully utilizing the idle computing power of the mobile phone to improve resource utilization efficiency.

[0020] In a preferred embodiment of the present invention, step 1 above may include: Step 1.1: Based on the launch and mutual discovery of the TV and mobile applications within the local area network, the TV or mobile phone initiates a connection request. In response to the connection request, authentication and authorization are performed between the TV and the mobile phone. Specifically, this includes: first, connecting the TV and mobile phone to the same local area network environment; then, launching the computing power collaboration application on both the TV and mobile phone. After the TV application launches, it continuously scans for devices within the local area network that conform to the computing power collaboration protocol. Simultaneously, after the mobile application launches, it actively sends its own device identification signal to the local area network. When the TV application receives the device identification signal from the mobile application, or the mobile application receives the scan signal from the TV application, the two... Once mutual discovery is complete, if the TV has a high-computing-power application running requirement, the TV application will initiate a connection request to the mobile application. If the mobile application detects insufficient computing power on the TV, it will initiate a connection request to the TV application. After responding to the request, the TV will send information containing its own device model and user authorization identifier to the mobile application, and the mobile application will also send information containing its own device model and user authorization identifier to the TV. Both parties will verify the received device model and authorization identifier to confirm that the device model meets the computing power collaboration requirements and that the authorization identifier is legal and valid, thus completing the authentication and authorization process between the TV and the mobile application.

[0021] Step 1.2 After successful authentication and authorization, a wireless communication connection session is established between the TV and the mobile phone for transmitting compressed video streams and operation commands. Specifically, after the authentication and authorization process between the TV and the mobile phone is fully completed, the TV application and the mobile application work together to create an independent wireless communication connection session for both. The session will pre-set data transmission type restrictions, allowing only compressed video streams and operation commands to be transmitted through the session. The TV and the mobile phone work together to configure the transmission channel of the session, prioritizing the allocation of network bandwidth resources within the local area network to the session to prevent network data from other devices from occupying the transmission bandwidth of the session. A data transmission verification mechanism will also be set in the session to ensure that the compressed video stream will not be lost or damaged during transmission, and that the operation commands can be transmitted accurately, thereby ensuring that the transmission of compressed video streams and operation commands between the TV and the mobile phone can be stable and efficient.

[0022] In this embodiment of the invention, because the TV application and the mobile application discover each other after starting in the local area network, and the TV or mobile phone initiates a connection request and performs authentication and authorization between the two, and then establishes a wireless communication connection session dedicated to transmitting compressed video streams and operation commands after the authentication and authorization are successful, the technical means overcome the insufficient connection security caused by the lack of authentication and authorization in the existing device connection process, as well as the data transmission chaos and mutual interference caused by the lack of dedicated transmission sessions for critical data. This achieves the goal of ensuring the connection security between the TV and the mobile phone, while providing a dedicated channel for core data transmission, avoiding transmission interference, and ensuring the stability and efficiency of data transmission.

[0023] In a preferred embodiment of the present invention, step 2 above may include: Step 2.1: When the application is running on the TV, the attribute characteristics of each pending task in the application are monitored and extracted in real time. The attribute characteristics include task type identifier, the amount of data to be processed, and real-time priority. Specifically, when the application is launched and running on the TV, the TV background automatically starts the task monitoring function. The function will follow the task execution process of the application in real time, dynamically track and collect information on all pending tasks in the application. For each pending task, its task type identifier will be identified and extracted. If the task is to obtain scene images for a 3D game, the task type identifier will be determined as 3D image rendering. If the task is to parse 8K video data for playback, the task type identifier will be determined as high-definition video decoding. The system calculates the amount of data required for each task: In 3D rendering tasks, it calculates the volume of model data and texture size in the current scene to be rendered. In high-definition video decoding tasks, it calculates the data volume of a single 8K video frame and the number of video frames to be decoded per unit time. Combining the user's current actions and application logic, the system determines the real-time priority of each task: For example, when a user is playing a 3D game and entering a team battle scene, the 3D rendering task directly affects the user's real-time viewing experience, so its real-time priority is set to the highest. If the user does not fast forward or rewind during high-definition video playback and only watches normally, the real-time priority of the corresponding high-definition video decoding task is set to medium.

[0024] Step 2.2: Match the extracted attribute features with a preset high-computing-power task feature library. The high-computing-power task feature library pre-stores benchmark features related to 3D image rendering and high-definition video decoding tasks. For tasks that are successfully matched by the high-computing-power task feature library, predict the computing power consumption required for execution based on the attribute features. Specifically, the TV system calls the preset high-computing-power task feature library, which pre-stores various benchmark features related to 3D image rendering and high-definition video decoding. For example, the benchmark features for 3D image rendering tasks include specific model complexity ranges and texture accuracy standards, while the benchmark features for high-definition video decoding tasks include thresholds for the amount of decoding data corresponding to different resolution videos. The attribute features of each extracted task to be processed are compared and matched one by one with the benchmark features in the high-computing-power task feature library. If the attribute features of a task to be processed completely match the benchmark features of 3D image rendering or high-definition video decoding in the library, the task is determined to be a successful match. For successfully matched tasks, the required computing power consumption is predicted based on the data volume information in its attribute features. For example, for a successfully matched 3D image rendering task, if the number of model triangle faces and the texture precision in the data volume to be processed are large, the predicted task will require high graphics processing computing power. For a successfully matched high-definition video decoding task, if the data volume to be processed is 8K resolution and the number of frames decoded per unit time is large, the predicted task will require high central processing unit computing power.

[0025] Step 2.3: By combining the matching results with the predicted computing power consumption scale and comparing it with the current remaining computing power resources on the TV, tasks requiring high computing power support are identified. Specifically, the TV system first obtains the current remaining computing power resources on the TV by reading data such as the real-time utilization rate of the central processing unit, the idle computing power of the graphics processor, the remaining available space of memory, and the read / write speed of the storage device. The remaining computing power that the TV can currently call upon is then calculated. Subsequently, the matching results are combined with the predicted computing power consumption scale. The predicted computing power consumption scale is compared with the current remaining computing power resources on the TV. If a task matches the characteristics of a high computing power task and its predicted computing power consumption scale exceeds the carrying capacity limit of the current remaining computing power resources on the TV, such as the predicted computing power requirement for 3D image rendering exceeding the idle computing power of the TV's graphics processor, or the predicted computing power requirement for high-definition video decoding exceeding the remaining computing power of the TV's central processing unit, and considering that TV storage mostly uses the eMMC standard and has limited data processing speed, it is determined that the task cannot be processed independently and efficiently by the TV. Finally, it is identified as a task requiring high computing power support.

[0026] In this embodiment of the invention, the TV monitors the load parameters of its own processor and graphics processor in real time while running applications, and then compares the monitored load parameters with a preset load threshold. When the load parameters exceed the preset load threshold, it is determined that the currently running task is a task that the TV cannot handle and requires high computing power. The task is then marked as a task to be assigned to the mobile phone. This overcomes the technical problem of traditional TVs being unable to accurately identify high computing power tasks that they cannot handle. They either force the TV to handle high computing power tasks, resulting in application lag and screen unplayability, or blindly assign tasks, resulting in resource waste. This achieves the technical effect of accurately locating tasks that the TV needs to rely on external computing power for processing, providing an accurate basis for the subsequent reasonable allocation of tasks to the mobile phone, avoiding TV hardware overload, and ensuring the basic operational stability of TV applications.

[0027] In a preferred embodiment of the present invention, step 3 above may include: Step 3.1: The TV encapsulates the task data and processing instructions of the task to be assigned into a task request and sends it to the mobile phone via wireless communication. Specifically, when the TV determines the task to be assigned, such as a 3D game rendering task or an 8K high-definition video decoding task, the TV first collects the task data corresponding to the task to be assigned. If the task is 3D game rendering, the task data includes the character model data of the current game scene, scene terrain parameters, and lighting effect configuration information; if the task is 8K high-definition video decoding, the task data includes the 8K video frame data to be decoded and the video encoding format information. The TV will receive corresponding processing instructions, which clearly indicate the task type and processing requirements. For example, it may be to perform single-frame rendering of the current 3D game scene data or decode 8K video frame data according to the specified encoding format. The TV will integrate the collected task data with the generated processing instructions and encapsulate them into a complete task request. During the encapsulation process, it will ensure that the data and instructions correspond and are related without omission. After the encapsulation is completed, the TV will send the task request to the authenticated and authorized mobile phone through the previously established wireless communication connection, avoiding the impact of home WiFi channel interference, wall obstruction, etc. due to reliance on external cloud services.

[0028] Step 3.2: Based on the task request received by the mobile phone, the task data and processing instructions are parsed out. The mobile phone calls its local computing resources to perform calculations on the task data and obtain the original 3D image rendering result. Specifically, after the mobile phone receives the task request sent by the TV through its own wireless communication module, it first deconstructs and analyzes the task request, separating the task data and processing instructions from the request, clarifying whether the current task type to be processed is 3D image rendering or 8K video decoding, and the specific processing requirements. Since the CPU and GPU performance of the mobile phone is much higher than that of mid-to-low-end TVs, the mobile phone will call the corresponding local computing resources according to the task type. If it is a 3D image rendering task, it calls its own high-performance GPU and related graphics processing modules to load the character model and scene parameters in the task data, calculate the character movement trajectory and scene lighting effects according to the processing instructions, and obtain the complete original image frame. If it is an 8K video decoding task, it calls its own dedicated video decoding module, combines the video encoding format information in the task data, and performs decoding calculations on the 8K video frame data to obtain a clear original video image. During the calculation and processing, the mobile phone makes full use of its own idle high computing power resources to avoid the TV from experiencing task processing lag or failure due to insufficient hardware performance.

[0029] Step 3.3: Based on the original 3D image rendering result, the mobile phone uses a hardware encoder to perform real-time encoding and data compression on the original 3D image rendering result to obtain a compressed video stream. Specifically, after the mobile phone completes the task calculation and processing to obtain the original 3D image rendering result, in order to avoid excessive network bandwidth consumption due to excessive data volume during transmission, which could lead to issues such as image freezing and operation delays similar to those in cloud service solutions, the mobile phone will activate its own hardware encoder. The hardware encoder selects a suitable encoding format based on the characteristics of the original image result, performs real-time encoding processing on the original image, and compresses redundant information in the image data. While ensuring that the image quality is not significantly degraded, it greatly reduces the data volume, ultimately obtaining a compressed video stream suitable for transmission via home wireless communication connections. The entire encoding and compression process is efficiently completed by the mobile phone's hardware encoder, and the processing speed matches the task calculation rhythm, avoiding situations where image generation and encoding are out of sync.

[0030] In this embodiment of the invention, the television encapsulates the task data and processing instructions of the task to be assigned into a task request and sends it to the mobile phone via a wireless communication connection. After receiving the task request, the mobile phone parses the task data and processing instructions, calls local computing resources to perform computational processing to obtain the original 3D image rendering result, and then the mobile phone uses a hardware encoder to encode and compress the original image result in real time to obtain a compressed video stream. Therefore, it overcomes the limitations of traditional televisions, whose CPU and GPU performance is only one-third to one-fifth of that of flagship mobile phones, making it difficult to handle high-computing tasks such as 3D image rendering or high-definition video decoding. Furthermore, existing hardware upgrade solutions are costly, application simplification solutions sacrifice user experience, and cloud service solutions rely on the network. This invention fully utilizes the idle high-computing resources of the mobile phone to make up for the television's computing power shortcomings, efficiently processes high-computing tasks, and reduces the amount of video stream data through real-time encoding and compression to reduce subsequent transmission pressure, thus ensuring the efficiency of high-computing application processing and data transmission on the television.

[0031] In a preferred embodiment of the present invention, step 4 above may include: Step 4.1: Based on the obtained compressed video stream, the mobile phone encapsulates the compressed video stream into Real-Time Transmission Protocol (RTP) data packets and sends them through the established wireless communication connection session. Specifically, after obtaining the compressed video stream, the mobile phone encapsulates the compressed video stream according to the RTP specifications based on the previously established wireless communication connection session with the TV. During encapsulation, a unique sequence number and timestamp are added to each segment of the compressed video stream data. The sequence number ensures that the TV can clearly identify the data transmission order when receiving it, and the timestamp is used to synchronize the display sequence of the screen, avoiding data transmission order disorder caused by channel interference or wall obstruction that may exist in home WiFi. After encapsulation, the mobile phone continuously and stably sends these encapsulated RTP data packets to the TV through the established wireless communication connection session, ensuring the real-time nature of data transmission and avoiding transmission delay issues caused by network fluctuations during peak cloud service periods.

[0032] Step 4.2 involves the TV receiving Real-Time Transmission Protocol (RTP) data packets and reassembling the received packets to obtain a complete compressed video stream. Specifically, the TV continuously receives RTP data packets from the mobile phone via its wireless communication receiving module. During reception, the TV first verifies the sequence number and timestamp of each data packet to check for lost packets or incorrect transmission order. If a lost packet is detected, the TV sends a brief retransmission request to the mobile phone. If the data packets are found to be out of order, their sequence number is rearranged. Once it is confirmed that all data packets required to form a complete compressed video stream have been received and are in the correct order, the TV integrates and splices these data packets to form a complete compressed video stream. This prevents image freezing due to incomplete or disordered data packets, thus resolving the image interruption problem caused by network instability in traditional cloud service solutions.

[0033] Step 4.3: Based on the obtained complete compressed video stream data, the TV calls the video decoding module to decode the complete compressed video stream data and restore the original frame sequence. Specifically, after the TV obtains the complete compressed video stream data, since the core design of the TV focuses on display functions, it is equipped with a dedicated video decoding module, which the TV will directly call. The decoding module will first identify the encoding format of the compressed video stream, and then decode the complete compressed video stream data according to the decoding rules corresponding to the encoding format, restoring the compressed video data to the uncompressed original frame sequence. The entire decoding process only consumes a small amount of the TV's computing resources, avoiding decoding stuttering due to the performance lag of the TV chip, and the original frame sequence can be output stably and continuously.

[0034] Step 4.4: Based on the original frame sequence, the TV submits the original frame sequence to the graphics rendering pipeline for rendering processing. Finally, the rendered image is output to the display device for the user. Specifically, after receiving the original frame sequence, the TV leverages its superior display processing capabilities to submit these frames to its own graphics rendering pipeline. The pipeline performs only basic display adaptation processing on the original frame sequence, adjusting the screen resolution to match the TV screen size and optimizing color and brightness to suit the TV's display characteristics. This avoids complex computational calculations and conserves the TV's limited computing resources. After rendering, the TV directly outputs the final optimized image to its display device for the user. 3D games maintain smooth frame rates, and 8K high-definition videos deliver clear image quality without the need for simplified solutions that cut effects or reduce model precision, thus ensuring a superior user experience when using high-performance applications.

[0035] In this embodiment of the invention, because the mobile phone encapsulates the obtained compressed video stream into real-time transmission protocol data packets and sends them through an established wireless communication connection session, the TV receives the real-time transmission protocol data packets, reassembles the received data packets to obtain complete compressed video stream data, and then calls its own video decoding module to decode the complete compressed video stream data to restore the original frame sequence. Finally, the original frame sequence is submitted to the graphics rendering pipeline for rendering and output to the display device. Therefore, this invention overcomes the technical problems of existing cloud service solutions, such as reliance on stable low-latency networks, susceptibility to interference from home WiFi channels and wall obstructions leading to image freezing, and the easy loss of data packets during compressed video stream transmission, as well as the difficulty for TVs to efficiently complete data transmission processing and decoding rendering due to insufficient computing power. This invention ensures the real-time performance and integrity of compressed video stream transmission, allows the TV to smoothly complete the decoding and rendering process, fully leverages the TV's display advantages, and ultimately presents users with a clear and smooth picture, avoiding image interruption or delay, and improving the user experience of high-computing applications on the TV.

[0036] In a preferred embodiment of the present invention, step 5 above may include: Step 5.1: During the TV display process, the user's input commands are captured through the TV's input device interface. Specifically, while the TV is continuously displaying the image processed by the mobile phone, it is equipped with a dedicated input device interface that is compatible with commonly used input devices such as remote controls. When the user interacts with the remote control, such as controlling a character to move, jump, or release skills in a 3D game, or adjusting the playback progress, switching chapters, or adjusting the volume in an 8K high-definition video, the TV's input device interface will capture these user input commands in real time. The entire capture process only calls the TV's basic input processing functions and does not require the TV's relatively weak CPU or GPU resources, thus avoiding delays or omissions in capturing operation commands due to insufficient computing power of the TV chip.

[0037] Step 5.2 encapsulates the operation command into a low-latency control data packet. The control data packet contains the operation type and parameter information. Through the established wireless communication connection session, the low-latency control data packet is transmitted to the mobile phone in real time. Specifically, after the TV captures the user's operation command, in order to solve the problem of high transmission latency of operation commands in traditional cloud service solutions, the operation command is simplified and encapsulated to obtain a low-latency control data packet. The data packet only contains key information, namely the operation type and corresponding parameter information. For example, when the operation type is character movement, the parameter information includes the movement direction and movement speed; when the operation type is video fast forward, the parameter information includes the fast forward duration. After encapsulation, the TV does not re-establish a new communication connection, but directly transmits the low-latency control data packet to the mobile phone quickly through the previously established wireless communication connection session. Due to the small size of the data packet and the reliance on the existing stable communication session, it can effectively avoid the transmission latency caused by channel interference and wall obstruction in home WiFi, and avoid the situation where the operation command latency exceeds the standard during peak cloud service periods. The operation command is transmitted to the mobile phone in real time.

[0038] In this embodiment of the invention, by employing a technique that captures user input commands through the input device interface of the TV during the TV display process, encapsulates the commands into low-latency control data packets containing operation type and parameter information, and transmits the low-latency control data packets to the mobile phone in real time through an established wireless communication connection session, the invention overcomes the high latency in operation command transmission caused by reliance on home WiFi, which is susceptible to channel interference and wall obstruction, resulting in a serious disconnect between user operation and screen feedback. Furthermore, if the TV processes the command transmission itself, its insufficient computing power can further slow down the interaction rhythm. This invention ensures that user operation commands can be quickly and accurately transmitted to the mobile phone, providing a safeguard for the mobile phone to update the application status and recalculate the screen in a timely manner based on the operation, effectively avoiding interaction delays and ensuring the smooth two-way interaction of the TV application.

[0039] In a preferred embodiment of the present invention, step 6 above may include: Step 6.1: Based on the parsed operation type and parameter information, the mobile phone updates the internal logic state of the currently running application. Specifically, after receiving the low-latency control data packet transmitted by the TV, the mobile phone first parses the operation type and parameter information in the data packet to clarify the user's specific operation requirements. For example, if the user is running a 3D game on the TV and issues a character movement command via remote control, the mobile phone will parse the operation type as character movement, and the parameter information will include the movement direction and movement speed. If the user issues a fast-forward command while watching an 8K high-definition video, the mobile phone will parse the operation type as video fast-forward, and the parameter information will include the fast-forward duration. Based on the parsing results, the mobile phone updates the internal logic state of the currently running application. For example, for 3D games, it updates the character's coordinate position in the game scene, current movement state, and related skill cooldown progress; for 8K high-definition videos, it updates the current playback time node of the video. The application's internal logic is consistent with the user's operation requirements, and the entire parsing and updating process is completed quickly by relying on the mobile phone's strong CPU performance, avoiding latency.

[0040] Step 6.2: Based on the updated internal logic state, the phone re-executes the corresponding graphics calculations and rendering processes to obtain a new original image result. Specifically, after updating the application's internal logic state, the phone fully utilizes its high-performance GPU resources to re-execute the corresponding graphics calculations and rendering processes based on the updated internal logic state. If the current application is a 3D game, the phone will call the GPU to calculate the updated motion trajectory of the character, the changes in light and shadow corresponding to the character's position in the game scene, and the real-time state of other game elements. Then, through rendering processing, a new original image result containing complete special effects and clear models will be generated. If the current application is an 8K high-definition video, the phone will calculate and generate the corresponding 8K high-definition original video frame based on the updated playback time point, ensuring complete image details without sacrificing clarity. The entire calculation and rendering process does not rely on the TV's weak computing power and is not affected by external network fluctuations like cloud services, ensuring a stable output of high-quality original image results.

[0041] Step 6.3 involves performing real-time encoding and data compression on the new original image result via the mobile phone to obtain a new compressed video stream. Specifically, after receiving the new original image result, the mobile phone will activate its own hardware encoder to avoid excessive data volume consuming too much home WiFi bandwidth during subsequent transmission. The hardware encoder will select an appropriate and efficient encoding format based on the characteristics of the original image result, such as the dynamic frame rate of 3D game graphics or the high resolution of 8K video graphics, and perform real-time encoding processing on the new original image result. During the encoding process, redundant data in the image will be intelligently compressed, significantly reducing the data size while ensuring that the image quality is not degraded. The final result is a new compressed video stream suitable for transmission via home wireless communication connection, and the encoding speed is synchronized with the image acquisition speed to avoid image stacking delays.

[0042] Step 6.4: The mobile phone transmits the new compressed video stream to the TV via a wireless communication connection session, triggering an update of the TV screen and completing a full two-way interactive loop between the user and the TV application. Specifically, this includes: after the mobile phone generates a new compressed video stream, it can quickly transmit the new compressed video stream to the TV directly through the previously established wireless communication connection without re-establishing a communication connection. Upon receiving the new compressed video stream, the TV calls its dedicated video decoding module to quickly decode it, restore the original frame sequence, and submit it to the graphics rendering pipeline for display adaptation processing. The processed image is then output to the TV screen, completing the update of the TV's display. For example, when a user issues a skill release command in a 3D game, the mobile phone quickly processes and transmits the new image, and the TV can promptly display the skill release effects and feedback, without any significant disconnect between skill release and image feedback. Similarly, when a user adjusts the progress of an 8K video, the TV can quickly display the corresponding progress. From the user initiating an operation on the TV, to the mobile phone processing and providing feedback on the new image, and then to the TV updating the display, a complete two-way interactive loop between the user and the TV application is formed. This eliminates the need for users to purchase additional high-end TVs or performance boxes and removes the dependence on stable, low-latency networks, ensuring the core experience of the application.

[0043] In this embodiment of the invention, the mobile phone updates the internal logic state of the currently running application based on the parsed operation type and parameter information, and then re-executes the corresponding graphics calculation and rendering processing based on the updated internal logic state to obtain a new original screen result. Real-time encoding and data compression are then performed on the new original screen result to obtain a new compressed video stream. This new compressed video stream is transmitted to the TV through an established wireless communication connection session to trigger a screen update on the TV, thus completing a complete two-way interactive closed loop between the user and the TV application. This overcomes the technical problems of existing cloud service solutions where interference from home WiFi channels and network fluctuations cause serious disconnect between operation commands and screen feedback, and low-configuration TVs, whose CPU and GPU performance is only one-third to one-fifth that of flagship mobile phones, lack sufficient computing power to respond to operations and update the screen in a timely manner, thus failing to meet user interaction needs. This invention achieves rapid response to user operations based on the mobile phone's sufficient computing power, ensuring real-time synchronization between operation commands and screen updates, avoiding interaction delays such as disconnect between skill release and screen feedback, and fully realizing a smooth two-way interactive closed loop. Furthermore, it eliminates the need for users to purchase additional high-end TVs or performance boxes, and eliminates the need to rely on stable, low-latency networks, ensuring a high-quality user experience for high-computing applications such as 3D games and 8K high-definition videos on the TV.

[0044] like Figure 2 As shown, embodiments of the present invention also provide a system for extending the computing power of television applications based on mobile phone computing power, comprising: The acquisition module is used to establish a wireless communication connection between the TV and the mobile phone. The wireless communication connection includes a WiFi direct connection automatically searched and established by the application, or a local area network connection established by the user manually entering the device IP address. When the application is running on the TV, the module identifies and filters the various tasks running in the application, extracts the attribute features of each task through a task feature matching algorithm, and compares the attribute features of each task with a preset high computing power task feature library to obtain feature matching results. The module predicts the computing power consumption scale of each task and identifies tasks that are difficult for the TV to process and require high computing power support based on the matching results and consumption scale. Tasks requiring high computing power support include 3D image rendering or high-definition video decoding. The compression module is used to distribute tasks requiring high computing power from the TV to the mobile phone, where the mobile phone performs the computation and processing to obtain 3D image rendering results. The 3D image rendering results are then encoded and compressed to obtain a compressed video stream. The transmission module is used to transmit compressed video streams from mobile phones to televisions via wireless communication connections. The televisions receive and decode the compressed video streams to obtain the picture and display the picture to the user. Based on the television display, the system receives user commands on the television and transmits these commands to the mobile phone via wireless communication. The processing module updates the application status and re-executes the calculation processing based on the operation instructions received from the mobile phone. It encodes and compresses the new image result to obtain a new compressed video stream, and transmits the new compressed video stream to the TV, thus completing the two-way interaction between the user and the TV application.

[0045] This invention primarily addresses the problem of sluggish application performance and poor user experience caused by low hardware configurations in smart TVs. By enabling collaborative work between a mobile phone and the TV, the mobile phone assists the TV in completing complex computational tasks, thereby improving the user experience of TV applications. It is applicable to various applications running on TVs. The core idea is to leverage the powerful computing capabilities of a user's commonly used mobile phone to help a low-configuration TV complete complex tasks in applications that rely on computing power. The specific process is as follows: Connecting Devices: Open the TV app or mobile app. The app will automatically search for nearby connectable devices and attempt to establish a connection via Wi-Fi Direct. If the connection fails, the app will display the target device's local area network IP address, which the user can manually enter to complete the connection between the TV and the mobile phone.

[0046] Task allocation: When a user uses an application on the TV, the system will determine which tasks the TV cannot handle based on the required computing power, such as complex scene rendering in games or high-definition video decoding. Then, these tasks will be allocated to the mobile phone to complete, while the TV will only be responsible for handling some simple tasks, such as UI rendering and receiving remote control operation commands.

[0047] Video transmission: After the mobile phone completes the task processing, it will obtain a suitable video stream for transmission and perform format conversion, transcoding, compression, etc. Then, the compressed video stream will be quickly transmitted to the TV so that the TV can play these images smoothly.

[0048] Two-way interaction: Users operate applications using the TV remote control, such as controlling character movement while playing games or adjusting the progress while watching videos. This operation information is transmitted back to the mobile phone in real time. The mobile phone processes the operation accordingly and then transmits the new picture back to the TV, resulting in a complete interactive process, as if the TV itself has powerful computing power and runs the application smoothly.

[0049] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for extending the computing power of a television application based on the computing power of a mobile phone, characterized in that the method comprises: establishing a wireless communication connection between the television and the mobile phone, including WiFi direct connection automatically searched and established by the application, or local area network connection established by the user manually inputting the device IP address; when the television runs the application, identifying and screening each task in the running of the application, extracting the attribute features of each task through a task feature matching algorithm, and comparing the attribute features of each task with a pre-set high-computing-power task feature library to obtain a feature matching result, predict the computing power consumption scale of each task, and identify the tasks that the television is difficult to handle and require high-computing-power support through the matching result and the consumption scale, including 3D picture rendering or high-definition video decoding; allocating the tasks requiring high-computing-power support from the television to the mobile phone for computing and processing by the mobile phone to obtain a 3D picture rendering result, encoding and compressing the 3D picture rendering result to obtain a compressed video stream; based on the compressed video stream, the mobile phone transmits the compressed video stream to the television through the wireless communication connection, receives and decodes the compressed video stream through the television to obtain a picture, and displays the picture to the user; based on the television displaying the picture, receiving the operation instructions of the user on the television end, and transmitting the operation instructions to the mobile phone through the wireless communication connection; updating the application state and re-executing the computing and processing through the operation instructions received by the mobile phone, encoding and compressing the new picture result obtained by the processing to obtain a new compressed video stream, transmitting the new compressed video stream to the television, and completing the bidirectional interaction between the user and the television application.

2. The method for extending the computing power of a television application based on the computing power of a mobile phone according to claim 1, characterized in that establishing a wireless communication connection between the television and the mobile phone, including WiFi direct connection automatically searched and established by the application, or local area network connection established by the user manually inputting the device IP address, comprises: based on the television end application and the mobile phone end application starting and discovering each other in the local area network, initiating a connection request by the television or the mobile phone based on the mutual discovery, and responding to the connection request to perform authentication and authorization between the television and the mobile phone; after the authentication and authorization, establishing a wireless communication connection session between the television and the mobile phone for transmitting the compressed video stream and the operation instructions.

3. The method for extending the computing power of a television application based on the computing power of a mobile phone according to claim 2, characterized in that when the television runs the application, identifying and screening each task in the running of the application, extracting the attribute features of each task through a task feature matching algorithm, and comparing the attribute features of each task with a pre-set high-computing-power task feature library to obtain a feature matching result, predict the computing power consumption scale of each task, and identify the tasks that the television is difficult to handle and require high-computing-power support through the matching result and the consumption scale, including 3D picture rendering or high-definition video decoding, comprising: When the application is running on the television side, the attributes of each task to be processed in the application are monitored and extracted in real time, including task type identification, data volume required for processing, and real-time priority; The extracted attribute features are matched with a preset high-computing-power task feature library, which pre-stores benchmark features related to 3D picture rendering and high-definition video decoding tasks; for tasks that match the high-computing-power task feature library, the computing power consumption scale required for execution is predicted based on the attribute features; By combining the matching result and the predicted computing power consumption scale, and comparing them with the current remaining computing power resources of the television side, tasks requiring high computing power support are identified.

4. The method of claim 3, wherein tasks requiring high computing power support are allocated from the television to the mobile phone, and the mobile phone performs calculation and processing to obtain 3D picture rendering results, and the 3D picture rendering results are encoded and compressed to obtain compressed video streams, including: The task data and processing instructions of the task to be allocated are packaged into a task request by the television and sent to the mobile phone through a wireless communication connection; Based on the task request received by the mobile phone, the task data and processing instructions are parsed; The mobile phone calls local computing resources to perform calculation and processing on the task data to obtain original 3D picture rendering results; Based on the original 3D picture rendering results, the mobile phone performs real-time encoding and data compression on the original 3D picture rendering results through a hardware encoder to obtain compressed video streams.

5. The method of claim 4, wherein based on the compressed video streams, the mobile phone transmits the compressed video streams to the television through a wireless communication connection, receives and decodes the compressed video streams by the television to obtain pictures, and displays the pictures to the user, including: Based on the obtained compressed video streams, the mobile phone sends the compressed video streams as real-time transport protocol data packets through an established wireless communication connection session; Based on the television receiving the real-time transport protocol data packets, the received data packets are reorganized to obtain complete compressed video stream data; Based on the complete compressed video stream data, the television calls a video decoding module to decode the complete compressed video stream data to restore the original picture frame sequence; Based on the original picture frame sequence, the television submits the original picture frame sequence to a graphics rendering pipeline for rendering processing, and finally outputs the rendered pictures to a display device to present to the user.

6. The method of claim 5, wherein based on the television displaying the pictures, operation instructions of the user on the television side are received and transmitted to the mobile phone through a wireless communication connection, including: During the process of displaying the pictures by the television, the operation instructions input by the user are captured through an input device interface of the television side; The operation instructions are packaged into low-latency control data packets, which contain operation type and parameter information, and the low-latency control data packets are transmitted to the mobile phone in real time through an established wireless communication connection session.

7. The method of claim 6, wherein the operation instruction received by the mobile phone is used to update the application state and re-execute the calculation process, the new picture result obtained by the processing is encoded and compressed to obtain a new compressed video stream, and the new compressed video stream is transmitted to the television to complete the bidirectional interaction between the user and the television application, including: the mobile phone updates the internal logic state of the current running application based on the operation type and parameter information obtained by the analysis; the mobile phone re-executes the corresponding graphic calculation and rendering process based on the updated internal logic state to obtain a new original picture result; the mobile phone performs real-time encoding and data compression on the new original picture result to obtain a new compressed video stream; and the mobile phone transmits the new compressed video stream to the television through a wireless communication connection session to trigger the update of the television picture and complete a complete bidirectional interaction loop between the user and the television application.

8. A system for extending the computing power of a television application based on the computing power of a mobile phone, the system implementing the method of any one of claims 1 to 7, and comprising: an acquisition module configured to establish a wireless communication connection between the television and the mobile phone, the wireless communication connection including a WiFi direct connection automatically searched and established by the application or a local area network connection manually input by the user; identify and filter each task in the running application when the application is running on the television, extract the attribute features of each task through a task feature matching algorithm, and compare the attribute features of each task with a preset high-computing-power task feature library to obtain a feature matching result, predict the computing power consumption scale of each task, and identify the tasks that are difficult to process by the television and require high computing power support through the matching result and the consumption scale, the tasks requiring high computing power support including 3D picture rendering or high-definition video decoding; a compression module configured to distribute the tasks requiring high computing power support from the television to the mobile phone, execute the calculation process by the mobile phone to obtain a 3D picture rendering result, encode and compress the 3D picture rendering result to obtain a compressed video stream; a transmission module configured to transmit the compressed video stream from the mobile phone to the television through the wireless communication connection based on the compressed video stream, receive and decode the compressed video stream by the television to obtain a picture, and display the picture to the user; receive the operation instruction of the user on the television based on the picture displayed on the television, and transmit the operation instruction to the mobile phone through the wireless communication connection; and a processing module configured to update the application state and re-execute the calculation process through the operation instruction received by the mobile phone, encode and compress the new picture result obtained by the processing to obtain a new compressed video stream, and transmit the new compressed video stream to the television to complete the bidirectional interaction between the user and the television application.

9. A computing device, comprising: one or more processors; and a storage device storing one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1 to 7. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ 10. A computer readable storage medium having stored therein a program, which, when executed by a processor, implements the method of any one of claims 1 to 7.

Images