A method and system for UI interaction of a smart television based on a Wi-Fi module

Abstract

The application discloses a kind of based on Wi-Fi module's intelligent television UI interaction method and system, belong to intelligent internet of things technical field.To solve the problem that existing intelligent television voice interaction only supports fixed instruction recognition and cannot be associated with real-time UI picture semantic analysis, the present application takes television Wi-Fi module as core, collects original voice instruction and current UI interface original image, generates thumbnail after with voice data is transmitted to remote server end by Wi-Fi wireless;Remote server end is jointly coded to voice text and image by multi-modal UI analysis large model, generates initial result with confidence, confidence meets standard and is directly issued, and is requested to issue after secondary analysis of original image, and Wi-Fi module analysis result is synthesized and played reply voice and executes UI control instruction.The present application relies on Wi-Fi module transmission high-definition UI image and synchronously collects user voice, completes visual voice joint semantic understanding by remote large model, and improves the intelligent level of intelligent television UI voice interaction.

Description

Technical Field

[0001] This invention belongs to the field of smart Internet of Things technology, specifically relating to a smart TV UI interaction method and system based on a Wi-Fi module. Background Technology

[0002] With the rapid development of smart home and IoT technologies, smart TVs have evolved from simple video playback devices into core terminals for home smart entertainment and IoT control, with their intelligence and interactive experience continuously improving. Among these advancements, voice interaction, with its advantages of being contactless and easy to operate, is gradually replacing traditional infrared remote controls and becoming the standard interaction method for smart TVs, widely used in scenarios such as device control, content search, and information retrieval.

[0003] Currently, most mainstream voice interaction solutions for smart TVs are based on pure voice recognition technology. These solutions use a speech-to-text module on the device or a remote server to convert the user's speech into text, which is then matched with a pre-set set of commands to execute fixed functions such as power on / off, volume adjustment, channel switching, and video / movie resource searching. This type of solution only performs semantic parsing of the spoken text and does not involve any perception or understanding of the UI currently displayed on the TV; it is an independent voice control mode detached from the user interface.

[0004] The aforementioned existing technologies have obvious technical defects: voice commands cannot be semantically associated with real-time UI screens, users can only issue fixed-format commands preset by the system, and cannot issue natural language operation commands based on the current interface. For example, operation requirements that are strongly related to interface elements, such as "select the third recommended card" or "open the settings entry in the upper right corner", cannot be realized, the interaction scenarios are limited, and the user's intelligent experience is insufficient. Summary of the Invention

[0005] To solve the above-mentioned technical problems, the present invention adopts the following solution:

[0006] On one hand, the present invention provides a smart TV UI interaction method based on a Wi-Fi module, comprising the following steps:

[0007] S1. Collect the user's original voice command data and the original image data of the UI interface currently displayed on the smart TV through the built-in Wi-Fi module of the smart TV.

[0008] S2. The Wi-Fi module generates a thumbnail of the original image data of the UI interface, and forwards the thumbnail and the original voice command data to the remote server via Wi-Fi wireless local area network.

[0009] S3. The remote server transcribes the original voice command data into voice text, performs joint semantic encoding on the voice text and the thumbnail input pre-trained multimodal UI parsing big model, and outputs the initial response result.

[0010] S4. The remote server determines whether the confidence level of the initial response meets the preset requirements; if it does, the initial response is sent to the Wi-Fi module as the final response; if it does not, the original image data of the UI interface is obtained and combined with the voice and text to re-parse and generate the final response before sending it.

[0011] S5, the Wi-Fi module performs structured parsing on the final response result, splits it into response voice text and UI control commands, synthesizes the response voice text into speech and plays it, and executes corresponding interface operations through UI control commands.

[0012] In step S1, the original voice command data is time-domain audio data obtained by the Wi-Fi module after preprocessing the collected audio signal through echo cancellation and environmental noise reduction.

[0013] In step S2, the resolution of the thumbnail is 1 / 16 to 1 / 4 of the original image resolution. The Wi-Fi module generates the thumbnail by bilinear downsampling the original image data of the UI interface and carries the resolution information of the original image in the file header of the thumbnail.

[0014] The multimodal UI parsing large model in step S3 includes a first visual encoder, a second visual encoder, a spatial text feature projector, a text encoder, and a large language model; the first visual encoder is adapted to the thumbnail resolution and is used to extract the UI spatial visual features of the thumbnail; the second visual encoder is adapted to the original high-definition image resolution and is used to extract the fine visual features of the original UI interface image.

[0015] In step S3, the initial response result is in a structured data format, including a confidence score, the response voice text, and UI control instructions; the UI control instructions include an operation type field, a target element identifier field, and a target location field.

[0016] Specifically, step S4 includes the following steps:

[0017] S41. The remote server performs structured parsing on the initial response results and extracts the confidence scores.

[0018] S42. If the confidence score is greater than or equal to a preset threshold, the initial response result is sent to the Wi-Fi module as the final response result.

[0019] S43. If the confidence score is less than the preset threshold, the remote server sends a request to the Wi-Fi module to obtain the original image of the UI interface.

[0020] S44. After receiving the original UI image data returned by the Wi-Fi module, the remote server inputs the voice text and the original UI image into the multimodal UI parsing model to generate the final response result and send it to the Wi-Fi module.

[0021] Specifically, step S5, which involves executing the corresponding interface operation via UI control instructions, means that after the Wi-Fi module parses and obtains the UI control instructions, it calls the system interface operation interface of the smart TV through the UI interface, locates the corresponding control by combining the target element identifier and the target position, and executes the interface operation corresponding to the operation type field.

[0022] On the other hand, the present invention provides a smart TV UI interaction system based on a Wi-Fi module, including a Wi-Fi module installed on the smart TV and a remote server deployed in the cloud.

[0023] The Wi-Fi module is used to collect the user's original voice command data and the original image data of the current UI interface of the smart TV, generate a UI interface thumbnail and upload it to the remote server along with the original voice command data. It is also used to receive the final response result sent by the remote server, parse the response voice and UI control command and execute them respectively.

[0024] The remote server has a built-in multimodal UI parsing model, which is used to perform joint semantic parsing on the received thumbnail and speech-to-text, generate an initial response result with a confidence score, and choose to directly send the initial response result or request the Wi-Fi module to send back the original image for secondary parsing before sending the final response result based on the confidence score.

[0025] Furthermore, the Wi-Fi module includes a central processing unit, a voice interface unit, a UI interface unit, a wireless transmission unit, and a data storage unit; the voice interface unit is used to interface with audio acquisition hardware to obtain raw voice command data; the UI interface unit is used to interface with the frame buffer interface of the smart TV's main control chip to obtain raw UI interface image data; the central processing unit is used to generate UI interface thumbnails and parse the final response results to obtain UI control commands; the wireless transmission unit is used to establish a wireless local area network data transmission link with a remote server; and the data storage unit is used to cache image data and command data.

[0026] Compared with the prior art, the present invention has the following significant advantages:

[0027] This invention leverages the high bandwidth of Wi-Fi modules to achieve efficient wireless transmission of high-definition UI interface images from smart TVs. Simultaneously, it collects user voice commands, and a remote server-side multimodal UI parsing model completes visual-voice joint semantic encoding and understanding. This breaks through the limitations of traditional voice control, which only supports a fixed set of commands. It allows users to issue natural language control commands based on the current interface scene, accurately locates UI controls, and executes corresponding operations. This achieves precise voice control that integrates UI image semantics, significantly improving the intelligence level of smart TV UI voice interaction. Attached Figure Description

[0028] Figure 1 This is a flowchart of the overall method steps in Embodiment 1 of the present invention;

[0029] Figure 2 This is a schematic diagram of the internal structure of the large multimodal UI parsing model in Embodiment 1 of the present invention;

[0030] Figure 3 This is a flowchart of step S4 in Embodiment 1 of the present invention;

[0031] Figure 4 This is a block diagram of the system components in Embodiment 2 of the present invention; Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] Furthermore, for clarity and brevity, descriptions of well-known structures, functions, and configurations may have been omitted. Those skilled in the art will recognize that various changes and modifications can be made to the examples described herein without departing from the spirit and scope of this disclosure.

[0034] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0035] Example 1

[0036] This embodiment provides a UI interaction method for smart TVs based on Wi-Fi modules.

[0037] The aforementioned UI interaction method for a smart TV based on a Wi-Fi module, such as Figure 1 As shown, it includes the following steps:

[0038] S1. Collect the user's original voice command data and the original image data of the UI interface currently displayed on the smart TV through the built-in Wi-Fi module of the smart TV.

[0039] S2. The Wi-Fi module generates a thumbnail of the original image data of the UI interface, and forwards the thumbnail and the original voice command data to the remote server via Wi-Fi wireless local area network.

[0040] S3. The remote server transcribes the original voice command data into voice text, performs joint semantic encoding on the voice text and the thumbnail input pre-trained multimodal UI parsing big model, and outputs the initial response result.

[0041] S4. The remote server determines whether the confidence level of the initial response meets the preset requirements; if it does, the initial response is sent to the Wi-Fi module as the final response; if it does not, the original image data of the UI interface is obtained and combined with the voice and text to re-parse and generate the final response before sending it.

[0042] S5, the Wi-Fi module performs structured parsing on the final response result, splits it into response voice text and UI control commands, synthesizes the response voice text into speech and plays it, and executes corresponding interface operations through UI control commands.

[0043] In this embodiment, the original voice command data mentioned in step S1 is time-domain audio data obtained by the Wi-Fi module after echo cancellation and environmental noise reduction preprocessing of the collected audio signal.

[0044] Specifically, the AEC echo cancellation algorithm is used to cancel the audio echo played by the TV itself. Then, adaptive filtering and spectral subtraction are used to complete environmental noise reduction, and output mono time-domain audio data with a sampling rate of 16kHz and 16bit quantization is output to filter out background noise and reverberation interference, ensuring the accuracy of subsequent speech transcription.

[0045] In this embodiment, the resolution of the thumbnail in step S2 is 1 / 16 to 1 / 4 of the original image resolution. The Wi-Fi module generates the thumbnail by performing bilinear downsampling on the original image data of the UI interface, and carries the resolution information of the original image in the file header of the thumbnail.

[0046] Taking the original 4K TV UI image of 3840×2160 as an example, a 960×540 thumbnail is generated by bilinear downsampling at a 1 / 4 ratio, falling within the 1 / 16 to 1 / 4 range. Bilinear downsampling merges adjacent pixel grayscale interpolation to avoid blurry text in the thumbnail controls. The file header retains the original 3840 and 2160 resolutions for easy subsequent coordinate mapping and image restoration.

[0047] In this embodiment, the multimodal UI parsing within the large model described in step S3, such as... Figure 2 As shown, it includes a first visual encoder, a second visual encoder, a spatial text feature projector, a text encoder, and a large language model; the first visual encoder is adapted to the thumbnail resolution and is used to extract the UI spatial visual features of the thumbnail; the second visual encoder is adapted to the original high-definition image resolution and is used to extract the fine visual features of the original UI interface image.

[0048] Specifically, the first visual encoder is a lightweight visual Transformer structure, adapted to low-resolution thumbnail input, extracting global coarse-grained spatial visual features of the UI interface with low computational overhead. The second visual encoder is a high-resolution visual Transformer structure, adapted to raw high-definition image input, extracting fine visual features such as control text content, icon texture, and element boundaries. The spatial text feature projector uses a two-layer MLP to map visual spatial features into a feature sequence of the same dimension as the text embedding. The text encoder is paired with a large language model to complete word segmentation and embedding mapping for speech-to-text transcription. The large language model adopts current mainstream base models (such as GLM or Qwen), receiving concatenated input of visual feature sequences and text feature sequences, and outputting text response results.

[0049] This multimodal UI parsing model is built for smart TV UI interaction scenarios. The training dataset covers all categories of UI interfaces, including TV desktop, video-on-demand, and system settings, and annotates control types, pixel coordinates, semantic labels, and corresponding natural language command samples. The model input consists of UI interface thumbnails, raw image data, and user speech-to-text. The output is structured data including confidence scores, response text, and UI control commands. Training is divided into two stages: the first stage pre-trains a dual-visual encoder using UI interface data with a learning rate of 2e-4; the second stage fine-tunes the model using voice-UI paired labeled samples with a learning rate of 1e-5, a batch size of 8, and 20 training epochs. The loss is a fusion of semantic cross-entropy and control localization regression loss.

[0050] In this embodiment, the initial response result in step S3 is in a structured data format, including a confidence score, response voice text, and UI control instructions; the UI control instructions include an operation type field, a target element identifier field, and a target location field.

[0051] Specifically, the response results adopt a JSON structured format, with a confidence score ranging from 0 to 1; the operation types in the UI control instructions cover click, swipe, focus, and jump, the target element is identified by the control's unique ID, and the target position stores the pixel coordinates of the control's top left and bottom right corners.

[0052] In this embodiment, step S4, as follows Figure 3 As shown, it specifically includes the following steps:

[0053] S41. The remote server performs structured parsing on the initial response results and extracts the confidence scores.

[0054] S42. If the confidence score is greater than or equal to a preset threshold, the initial response result is sent to the Wi-Fi module as the final response result.

[0055] S43. If the confidence score is less than the preset threshold, the remote server sends a request to the Wi-Fi module to obtain the original image of the UI interface.

[0056] S44. After receiving the original UI image data returned by the Wi-Fi module, the remote server inputs the voice text and the original UI image into the multimodal UI parsing model to generate the final response result and send it to the Wi-Fi module.

[0057] Specifically, the remote server first parses the initial response JSON message, extracts the confidence score field, and verifies the format of the UI control command field, eliminating invalid results with missing fields or incorrect formats. If the confidence score meets the standard, the server encapsulates the initial result into a downlink control message and sends it to the TV Wi-Fi module via the established Wi-Fi long connection. If the confidence score does not meet the standard, a request for obtaining the original image with a unique session identifier, labeled image encoding format, and transmission priority is generated and sent to the module. After receiving the returned high-definition original image, the second visual encoder is called to extract fine visual features, and semantic reasoning is re-integrated with voice and text to generate a high-precision final result before sending the control message.

[0058] In this embodiment, the execution of the corresponding interface operation through UI control instructions in step S5 specifically means: after the Wi-Fi module parses and obtains the UI control instructions, it calls the system interface operation interface of the smart TV through the UI interface, and combines the target element identifier and the target position to locate the corresponding control and execute the interface operation corresponding to the operation type field.

[0059] Example 2

[0060] This embodiment provides a smart TV UI interaction system based on a Wi-Fi module, such as... Figure 4As shown, it includes a Wi-Fi module installed on the smart TV and a remote server deployed in the cloud.

[0061] In this embodiment, the Wi-Fi module is used to collect the user's original voice command data and the original image data of the current UI interface of the smart TV, generate a UI interface thumbnail and upload it to the remote server along with the original voice command data. It is also used to receive the final response result sent by the remote server, parse the response voice and UI control command and execute them respectively.

[0062] In this embodiment, the remote server has a built-in multimodal UI parsing model, which is used to perform joint semantic parsing on the received thumbnail and speech-to-text, generate an initial response result with a confidence score, and choose to directly send the initial response result or request the Wi-Fi module to send back the original image for secondary parsing before sending the final response result based on the confidence score.

[0063] Specifically, the remote server adopts a distributed microservice architecture, deploying speech transcription service, multimodal inference service and confidence determination service respectively, supporting multiple concurrent accesses, and dynamically scheduling inference computing resources according to real-time load.

[0064] In this embodiment, the Wi-Fi module, such as Figure 4 As shown, it includes a central processing unit, a voice interface unit, a UI interface unit, a wireless transmission unit, and a data storage unit. The voice interface unit is used to interface with audio acquisition hardware to obtain raw voice command data. The UI interface unit is used to interface with the frame buffer interface of the smart TV's main control chip to obtain raw UI interface image data. The central processing unit is used to generate UI interface thumbnails and parse the final response results to obtain UI control commands. The wireless transmission unit is used to establish a wireless local area network data transmission link with a remote server. The data storage unit is used to cache image data and command data.

[0065] Specifically, the central processing unit uses a SoC chip, and the data storage unit is equipped with Flash to cache temporary data. The wireless transmission unit supports the IEEE 802.11a / b / g / n / ac / ax Wi-Fi standard, covering the 2.4GHz and 5GHz dual-band, and is equipped with a MIMO metal antenna, enabling concurrent dual-band Wi-Fi transmission.

[0066] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Based on the technical essence of the present invention, any simple modifications, equivalent substitutions, and improvements made to the above embodiments within the spirit and principles of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A UI interaction method for a smart TV based on a Wi-Fi module, characterized in that, Includes the following steps: S1. Collect the user's original voice command data and the original image data of the UI interface currently displayed on the smart TV through the built-in Wi-Fi module of the smart TV. S2. The Wi-Fi module generates a thumbnail of the original image data of the UI interface, and forwards the thumbnail and the original voice command data to the remote server via Wi-Fi through a wireless local area network. S3. The remote server transcribes the original voice command data into voice text, performs joint semantic encoding on the voice text and the thumbnail input pre-trained multimodal UI parsing big model, and outputs the initial response result. S4. The remote server determines whether the confidence level of the initial response meets the threshold requirement; if it does, the initial response is sent to the Wi-Fi module as the final response. If the conditions are not met, the original image data of the UI interface will be obtained and combined with the voice and text to re-parse and generate the final response result before it is sent out. S5, the Wi-Fi module performs structured parsing on the final response result, splits it into response voice text and UI control commands, synthesizes the response voice text into speech and plays it, and executes corresponding interface operations through UI control commands.

2. The UI interaction method for a smart TV based on a Wi-Fi module according to claim 1, characterized in that, The original voice command data mentioned in step S1 is time-domain audio data obtained by the Wi-Fi module after echo cancellation and environmental noise reduction preprocessing of the collected audio signal.

3. The UI interaction method for a smart TV based on a Wi-Fi module according to claim 1, characterized in that, The resolution of the thumbnail in step S2 is 1 / 16 to 1 / 4 of the original image resolution. The Wi-Fi module generates the thumbnail by bilinear downsampling the original image data of the UI interface, and carries the resolution information of the original image in the file header of the thumbnail.

4. The UI interaction method for a smart TV based on a Wi-Fi module according to claim 1, characterized in that, The Wi-Fi method described in step S2 uses a multi-link concurrent transmission method for image data and a transmission method that carries a low-latency quality of service priority identifier for voice data.

5. The UI interaction method for a smart TV based on a Wi-Fi module according to claim 1, characterized in that, The multimodal UI parsing large model described in step S3 includes a first visual encoder, a second visual encoder, a spatial text feature projector, a text encoder, and a large language model; the first visual encoder adapts to the thumbnail resolution and is used to extract the UI spatial visual features of the thumbnail. The second visual encoder is adapted to the original high-definition image resolution and is used to extract fine visual features from the original UI interface image.

6. The UI interaction method for a smart TV based on a Wi-Fi module according to claim 1, characterized in that, The initial response result in step S3 is in a structured data format, including a confidence score, response voice text, and UI control instructions; the UI control instructions include an operation type field, a target element identifier field, and a target location field.

7. A smart TV UI interaction method based on a Wi-Fi module according to claim 5, characterized in that, Step S4 specifically includes the following steps: S41. The remote server performs structured parsing on the initial response results and extracts the confidence scores. S42. If the confidence score is greater than or equal to a preset threshold, the initial response result is sent to the Wi-Fi module as the final response result. S43. If the confidence score is less than the preset threshold, the remote server sends a request to the Wi-Fi module to obtain the original image of the UI interface. S44. After receiving the original UI image data returned by the Wi-Fi module, the remote server inputs the voice text and the original UI image into the multimodal UI parsing model to generate the final response result and send it to the Wi-Fi module.

8. A smart TV UI interaction method based on a Wi-Fi module according to claim 6, characterized in that, The step S5, which involves executing the corresponding interface operation via UI control instructions, specifically involves the Wi-Fi module parsing the UI control instructions, calling the smart TV's system interface operation interface through the UI interface, locating the corresponding control by combining the target element identifier and the target position, and executing the interface operation corresponding to the operation type field.

9. A smart TV UI interaction system based on a Wi-Fi module, characterized in that, This includes a Wi-Fi module installed inside the smart TV and a remote server deployed in the cloud. The Wi-Fi module is used to collect the user's original voice command data and the original image data of the current UI interface of the smart TV, generate a UI interface thumbnail, and upload it to the remote server along with the original voice command data. It is also used to receive the final response result sent by the remote server, parse the response voice and UI control commands, and execute them respectively. The remote server has a built-in multimodal UI parsing model, which is used to perform joint semantic parsing on the received thumbnail and speech-to-text, generate an initial response result with a confidence score, and choose to directly send the initial response result or request the Wi-Fi module to send back the original image for secondary parsing before sending the final response result based on the confidence score.

10. The smart TV UI interaction system based on a Wi-Fi module according to claim 9, characterized in that, The Wi-Fi module includes a central processing unit, a voice interface unit, a UI interface unit, a wireless transmission unit, and a data storage unit; the voice interface unit is used to interface with audio acquisition hardware to obtain raw voice command data. The UI interface unit is used to interface with the frame buffer interface of the smart TV's main control chip to obtain the original image data of the UI interface; the central processing unit is used to generate UI interface thumbnails and parse the final response results to obtain UI control commands; the wireless transmission unit is used to establish a wireless local area network data transmission link with the remote server; and the data storage unit is used to cache image data and command data.

Images