Video data processing method and device, electronic equipment, storage medium and program

Abstract

The embodiment of the invention discloses a video data processing method and device, electronic equipment, a storage medium and a program. The video data processing method applied to intelligent security terminal equipment comprises the following steps: determining target processing video data; generating associated video display element data of the target processing video data; determining target associated transmission video data according to the target processing video data and the associated video display element data; sending the target associated transmission video data to a host device; wherein the host device is used for displaying a video picture according to the target associated transmission video data. According to the technical scheme provided by the embodiment of the invention, the packet loss probability and delay rate of the video data in the intelligent security terminal equipment can be reduced, the data links for transmitting the video data are reduced, and the image quality of the original video is ensured to the greatest extent, so that the image quality of the video application of the intelligent security terminal equipment is improved, and the user experience is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the fields of computer software applications and video data processing technology, and in particular to a video data processing method, apparatus, electronic device, storage medium and program. Background Technology

[0002] To meet the need for complete isolation and protection of privacy and sensitive data on devices, a smart security terminal device that reuses the peripheral resources of the host device has emerged. These smart security terminal devices are portable, such as miniature devices, integrating a System-on-Chip (SoC) processor and storage system, as well as communication modules such as WiFi (Wireless Fidelity) or USB (Universal Serial Bus) interfaces. After connecting to a host device such as a smartphone, tablet, or computer via USB or wirelessly, they independently run a complete operating system, displaying the operating system interface through the host device. During use, the smart security terminal device does not rely on the host device's CPU (Central Processing Unit), GPU (Graphics Processing Unit), memory, or storage resources, thus achieving complete isolation between the operating system and the host device, ensuring data security and system stability. For user portability, this device can securely reuse the host device's peripheral resources, providing effective protection of privacy and sensitive data in a low-cost, highly portable manner, improving data security and user convenience, and simplifying user operations.

[0003] While reusing peripheral resources from the host device helps achieve low cost and high portability, smart security terminal devices have high requirements for data transmission in audio and video applications. In particular, when calling the video application function of the smart security terminal device, video data needs to be obtained from the host device, encoded and compressed, and then transmitted to the smart security terminal device for decoding and decompression. After being encoded and compressed again, it is transmitted to the host device to complete the visualization interface display.

[0004] In the process of realizing this invention, the inventors discovered the following defects in the prior art: the above video data needs to be repeatedly encoded and decoded during the display process, resulting in a high probability of data loss, large delays, stuttering, unclear or out-of-sync images, reduced image quality, and an inability to provide a good user experience. Summary of the Invention

[0005] This invention provides a video data processing method, apparatus, electronic device, storage medium, and program that can reduce the probability of packet loss and latency of video data in smart security terminal devices, reduce the data links for video data transmission, and maximize the image quality of the original video, thereby improving the image quality of video applications on smart security terminal devices and enhancing the user experience.

[0006] According to one aspect of the present invention, a video data processing method is provided, applied to a smart security terminal device, comprising:

[0007] Identify the target video data to process;

[0008] Generate associated video display element data for the target processed video data;

[0009] The target associated transmission video data is determined based on the target processed video data and the associated video display element data;

[0010] The target-associated video data is sent to the host device;

[0011] The host device is used to transmit video data and display video images according to the target association.

[0012] According to another aspect of the present invention, a video data processing method is provided, applied to a host device, comprising:

[0013] Acquire target-related transmitted video data sent by a smart security terminal device; wherein, the smart security terminal device sends the target-related transmitted video data using a video data processing method executed by the smart security terminal device;

[0014] Based on the target associated transmitted video data, determine the target processed video data and the associated video display element data of the target processed video data;

[0015] The video image is displayed based on the target processed video data and the associated video display elements.

[0016] According to another aspect of the present invention, a video data processing apparatus is provided, configured in a smart security terminal device, comprising:

[0017] The target processing video data determination module is used to determine the target processing video data;

[0018] The associated video display element data generation module is used to generate associated video display element data of the target processed video data;

[0019] A target-associated transmission video data determination module is used to determine target-associated transmission video data based on the target processed video data and the associated video display element data.

[0020] The target-associated transmission video data sending module is used to send the target-associated transmission video data to the host device;

[0021] The host device is used to transmit video data and display video images according to the target association.

[0022] According to another aspect of the present invention, a video data processing apparatus is provided, configured in a host device, comprising:

[0023] A target-associated transmission video data acquisition module is used to acquire target-associated transmission video data sent by a smart security terminal device; wherein, the smart security terminal device sends the target-associated transmission video data using a video data processing method executed by the smart security terminal device;

[0024] A target-associated transmission video data processing module is used to determine target-processed video data and associated video display element data of the target-processed video data based on the target-associated transmission video data.

[0025] The video display module is used to process video data and display the associated video display elements according to the target to display the video screen.

[0026] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0027] At least one processor; and

[0028] A memory communicatively connected to the at least one processor; wherein,

[0029] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the video data processing method according to any embodiment of the present invention.

[0030] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the video data processing method according to any embodiment of the present invention.

[0031] According to another aspect of the present invention, a computer program product is also provided, comprising a computer program that, when executed by a processor, implements the video data processing method described in any embodiment of the present invention.

[0032] This invention, through an intelligent security terminal device, determines the target processed video data, generates associated video display element data, and then determines the target associated transmission video data based on the target processed video data and the associated video display element data, thereby sending the target associated transmission video data to the host device. The host device receives the target associated transmission video data sent by the intelligent security terminal device, determines the target processed video data and the associated video display element data based on the target associated transmission video data, and then displays the video image based on the target processed video data and the associated video display element data. This technical solution solves the problems of high probability of video data loss and high latency, video stuttering, unclear images, or desynchronization caused by repeated encoding and decoding during video data transmission between the intelligent security terminal device and the host device. It can reduce the probability of packet loss and latency of video data within the intelligent security terminal device, reduce the data links for video data transmission, and maximize the preservation of the original video quality, thereby improving the image quality of video applications on the intelligent security terminal device and enhancing the user experience.

[0033] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a schematic diagram of the structure of an intelligent security terminal device provided in an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram of the layered architecture of an intelligent security terminal device provided in an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram illustrating the process of video data transmission in existing intelligent security terminal devices.

[0038] Figure 4 This is a flowchart of a video data processing method provided in an embodiment of the present invention;

[0039] Figure 5 This is a flowchart of another video data processing method provided in an embodiment of the present invention;

[0040] Figure 6 This is a flowchart of another video data processing method provided in an embodiment of the present invention;

[0041] Figure 7 This is a flowchart illustrating a video data transmission method between a smart security terminal device and a host device according to an embodiment of the present invention.

[0042] Figure 8 This is a flowchart illustrating a video data transmission method between a smart security terminal device and a host device according to an embodiment of the present invention.

[0043] Figure 9 This is a schematic diagram of a video data processing device provided in an embodiment of the present invention;

[0044] Figure 10 This is a schematic diagram of another video data processing device provided in an embodiment of the present invention;

[0045] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0046] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0047] It should be noted that the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product or device.

[0048] Figure 1 This is a schematic diagram of the structure of an intelligent security terminal device provided in an embodiment of the present invention, exemplarily, as shown below. Figure 1As shown, the intelligent security terminal device is presented in a portable form, such as a miniature device. Internally, it integrates a SoC, memory, flash memory, and a USB or Wi-Fi wireless communication module. After connecting to a host device such as a smartphone, tablet, or computer via USB interface or Wi-Fi wireless communication, it independently runs a complete operating system, displaying the operating system interface through the host device. When the intelligent security terminal device connects to the host device, the host device's eSIM (Embedded Subscriber Identity Module) / SIM (Subscriber Identity Module), microphone, speaker, camera, and various sensors are shared with the intelligent security terminal device via USB or Wi-Fi communication. During operation, if an application uses audio playback, it transmits audio data to the host device via USB interface or Wi-Fi wireless data transmission, and plays the sound using the host device's speaker. If an application uses sound capture, it uses the host device's microphone to capture sound data and transmits it to the intelligent security terminal device's SoC. If an application uses photography or video recording, it uses the host device's camera to capture image or video data and transmits it to the intelligent security terminal device's SoC. Applications utilizing various sensor data acquire sensor data using the host device's sensors and transmit it to the SoC of the smart security terminal device. These sensors include, but are not limited to, common sensors found in smart terminals such as light sensors, GPS (Global Positioning System) sensors, distance sensors, gravity sensors, accelerometers, gyroscopes, barometers, and magnetic field sensors. Users can launch the operating system within the smart security terminal device through the host device's client application. Upon successful launch, the operating system will be displayed on the host device's screen. To improve display quality and efficiency, operating system images or videos are compressed into data packets and transmitted to the host device, where they are decoded and restored to a visual display.

[0049] Figure 2 This is a schematic diagram of the layered architecture of a smart security terminal device provided in an embodiment of the present invention. In a specific example, such as... Figure 2As shown, the layered architecture of intelligent security terminal devices divides hardware and software into several layers, each with a clear role and division of labor, jointly supporting the operation and development of the system. Layers communicate with each other through software interfaces. In this embodiment of the invention, taking the Android system as an example, the layered architecture of the intelligent security terminal device can be divided into five layers, from top to bottom: Application Layer (APP), Application Framework Layer (Framework), System Hardware Abstraction Layer (HAL), Linux Kernel Layer (Linux Kernel), and Hardware Layer (Hardware). It is understood that the layered architecture may differ depending on the operating system used by the intelligent security terminal device.

[0050] Specifically, the application layer of a smart secure terminal device is the user interface through which the user directly interacts. It includes pre-installed applications and third-party applications, such as basic system functions like phone calls, SMS, camera, browser, and settings, as well as third-party applications. The application framework layer of a smart secure terminal device is the core support layer of the Android system, providing application developers with a rich set of APIs (Application Programming Interfaces) and services, allowing developers to build complex applications without directly manipulating the underlying hardware or system kernel. Through modular components and predefined interfaces, the underlying capabilities of the system are abstracted into tools that developers can directly call. The core components include: an Activity Manager for managing the application's lifecycle and task stack; a Content Provider for enabling cross-application data sharing, such as reading contacts and photos; a View System for providing UI (User Interface Design) controls such as buttons, lists, and text boxes, and also for layout management; a Notification Manager for managing status bar notifications, floating notifications, sound and vibration alerts; a Resource Manager for unified management of resources such as strings, images, layout files, and colors; a Location Manager for providing GPS and network positioning (Wi-Fi / base station) services; a Window Manager for managing the hierarchy, size, and display of application windows; and a Package Manager for managing application installation, uninstallation, and permission queries.

[0051] The Hardware Abstraction Layer (HAL) of smart secure terminal devices is a crucial bridging layer in the Android architecture. It shields hardware differences through standardized interfaces, allowing upper-layer frameworks and applications to operate without directly manipulating hardware drivers. Typical HAL modules include cameras, screens, networks, sensors, audio, and GPS. The Linux kernel layer of smart secure terminal devices is the core foundation of the Android system, responsible for directly managing hardware resources, providing basic system services, and ensuring system stability and security. Its core functions include: hardware driver management, managing hardware such as screens, batteries, sensors, and communication modules; process and thread scheduling, controlling CPU resource allocation; memory management, optimizing memory allocation and reclamation; file system and storage, implementing full disk encryption (FDE) or file-based encryption (FBE) based on kernel-based dm-crypt (a Linux kernel-level encryption mechanism); network protocol stack, handling network data transmission and optimizing bandwidth utilization in mobile networks; and security mechanisms for access control. The hardware layer consists of physical hardware devices, including processors, sensors, communication modules, storage, and batteries. The intelligent security terminal device does not have independent hardware peripherals such as display screen, camera, GPS and various sensors. If the above hardware peripherals are required, they are shared with the host device.

[0052] Figure 3 This is a schematic diagram illustrating the video data transmission process of intelligent security terminal devices in existing technologies. In a specific example, such as... Figure 3 As shown, during the operation of the operating system of the intelligent security terminal device, an audio-visual connection channel is established between the intelligent security terminal device and the host device. The operating system of the intelligent security terminal device compresses and encodes images or videos into data packets, which are then transmitted to the host device at a custom frequency through the audio-visual connection channel. The host device obtains the data packets through the audio-visual connection channel, decodes and decompresses the data packets through the client APP, and then synthesizes them into an image or video display, which is then visually displayed on the host device's screen.

[0053] If video applications are required, the host device needs to acquire image or video data. The host device's client app compresses and encodes the image or video data into a compressed video data stream packet, and transmits the compressed video data stream packet to the smart security terminal device at a custom frequency via the host device's client app. The smart security terminal device decodes and decompresses the compressed video data stream packet into video data through its encoding and decoding module. The video data, along with the control layer of the smart security terminal device's video application and the UI layer of the smart security terminal device's operating system, is transformed into an uncompressed video frame through a layer compositing module. It is then encoded and compressed again through the encoding and decoding module into a complete compressed video data stream. The smart security terminal device's management app transmits the complete compressed video data stream to the host device, where it is decoded and decompressed into an uncompressed video frame through the host device's encoding and decoding module. Finally, the layer compositing module synthesizes a visual image and displays it on the host device's screen.

[0054] Among them, the management APP of the smart security terminal device can be a mobile application designed specifically for the smart security terminal device to assist in the management and optimization of various business processes, tasks and resources.

[0055] It is evident that during the video data transmission and display process between the aforementioned intelligent security terminal device and the host device, the video data undergoes multiple compression encoding, decompression decoding, and layer synthesis processes, resulting in a high probability of packet loss, significant latency, and reduced image quality, which greatly impacts the user experience.

[0056] Figure 4 This is a flowchart of a video data processing method provided in an embodiment of the present invention. This embodiment is applicable to situations where a smart security terminal device sends target associated transmission video data (excluding video compression data) to a host device so that the host device can display the video image. This method can be executed by a video data processing device, which can be implemented in software and / or hardware, and is generally integrated into an electronic device. This electronic device can be a smart security terminal device used in conjunction with a host device for displaying the video image. The present invention does not limit the specific type of smart security terminal device. Correspondingly, as... Figure 4 As shown, the method includes the following operations:

[0057] S110, Determine the target and process the video data.

[0058] The target video data to be processed can be video data that the smart security terminal device needs to display via the host device. Optionally, the target video data to be processed can be compressed video data, or it can be raw video data collected or generated by the host device. That is, the target video data to be processed can be a compressed video data stream or a raw video data stream. The host device is a device that establishes a communication connection with the smart security terminal device and can provide display hardware to the smart security terminal device to display video images. For example, the host device can include smartphones, smartwatches, tablets, and personal computers, as long as they have display hardware and can establish a communication connection with the smart security terminal device for data transmission. This embodiment of the invention does not limit the type of host device.

[0059] When processing video data that needs to be displayed on the host device's screen, the smart security terminal device can first determine the type of video data to be displayed. Understandably, some video data requires internal processing by the smart security terminal device's video application, and therefore needs to be sent from the host device to the smart security terminal device; this type of video data can be a compressed video stream. However, some video data does not require internal processing by the smart security terminal device's video application, and therefore does not need to be sent from the host device to the smart security terminal device; this type of video data can be a raw video stream.

[0060] S120. Generate associated video display element data of the target processed video data.

[0061] The associated video display element data can be data required for displaying video images on the host device, related to the target processed video data, and used to assist in the display of video images. For example, the associated video display element data may include metadata and UI layer data related to the target processed video data.

[0062] Accordingly, once the intelligent security terminal device identifies the target video data to be processed, it can generate corresponding metadata and UI layer data, as well as other associated video display element data. It is understandable that different target video data will result in different content of the generated associated video display element data.

[0063] In an optional embodiment of the present invention, generating the associated video display element data of the target processed video data may include: generating current associated video control layer data of the target processed video data based on operation controls pre-stored in the video application of the local device; generating current associated system control layer data based on the operating system of the local device; and generating associated video display element data of the target processed video data based on the current associated video control layer data and the current associated system control layer data.

[0064] Specifically, the currently associated video control layer data can be the operation controls generated by the smart security terminal device based on the operation controls pre-stored by the video application for processing the current target video data. The currently associated system control layer data can be the operating system-related control layer data automatically generated by the smart security terminal device based on its own operating system for the current target video application.

[0065] It is understood that video applications typically include video content and dynamic interactive elements such as bullet comments, filters, stickers, and UI controls. Before processing video data, the smart security terminal device can pre-store the dynamic interactive elements required by the video data, such as bullet comments, filters, stickers, and UI controls, as an operation control layer through the video application. When the smart security terminal device receives the target video data to be processed, it determines the type of interactive elements required by the target video data to be processed based on the operation controls pre-stored in the video application of the device, and dynamically generates the current associated video control layer data of the target video data to be processed based on the relevant information of the target video data. Optionally, the current associated video control layer data may include one or more operation control types, and this embodiment of the invention does not limit this. The transmission channel of the current associated video control layer data is different from the transmission channel of the target associated video data.

[0066] Simultaneously, the intelligent security terminal device can also automatically generate current associated system control layer data related to the target video application through the operating system. For example, the current associated system control layer data may include, but is not limited to, UI data such as the operating system's status bar controls. Accordingly, the intelligent security terminal device can generate associated video display element data for the target processed video data based on both the current associated video control layer data and the current associated system control layer data.

[0067] In an optional embodiment of the present invention, the step of generating associated video display element data of the target processed video data based on the currently associated video control layer data and the currently associated system control layer data may include: performing layer compositing processing on the currently associated video control layer data and the currently associated system control layer data according to layer compositing rules to obtain UI screen layer data; extracting description information data of the UI screen layer from the UI screen layer data to obtain video metadata; and performing encoding and compression processing on the UI screen layer data to obtain a UI data compressed stream.

[0068] Among them, video metadata can be metadata used to describe relevant descriptive information in video data.

[0069] In this embodiment of the invention, the layer compositing module of the smart security terminal device can generate associated video display element data of the target processed video data based on the currently associated video control layer data and the currently associated system control layer data. The layer compositing module can efficiently combine multiple layers into a single complete image frame, thereby providing an optimal user experience. Specifically, the smart security terminal device sends the currently associated video control layer data to the layer compositing module through the video application. Simultaneously, the layer compositing module obtains the currently associated system control layer data generated by the operating system of the local device through the window manager of the smart security terminal device.

[0070] Accordingly, after obtaining the currently associated video control layer data and the currently associated system control layer data, the layer compositing module performs layer compositing processing on the two types of data according to certain layer compositing rules to obtain the UI screen layer data. Optionally, the layer compositing rules can be selected and configured as needed. For example, the layer compositing rules can be to combine layers below the video data layer into one layer, combine layers above the video data layer into one layer, and treat the video data as a separate layer. Alternatively, the layer compositing rules can also treat dynamic interactive elements such as bullet comments, filters, and stickers as separate layers, and combine video application UI controls and operating system UI layers into one layer, etc. After obtaining the UI screen layer data, the layer compositing module can extract descriptive information data such as layout information, position coordinate information, and other auxiliary information of the UI screen layer data to generate video metadata of the target processed video data. At the same time, in order to improve data transmission efficiency, the layer compositing module can encode and compress the UI screen layer data to generate a compressed UI data stream of the target processed video data. The intelligent security terminal device can use the video metadata and UI data compressed stream of the target processed video data as the associated video display element data of the target processed video data.

[0071] S130. Determine the target associated transmission video data based on the target processed video data and the associated video display element data.

[0072] Among them, the target-related video data to be transmitted can be data related to the target's video data that needs to be transmitted between the smart security terminal device and the host device.

[0073] The content of the target-related video data transmitted between the smart security terminal device and the host device will differ depending on the type of video data being processed. Therefore, it is necessary to determine the target-related video data transmitted between the smart security terminal device and the host device based on the specific type of video data being processed and the associated video display element data.

[0074] Understandably, if the target video data needs to be processed internally by the video application of the smart security terminal device, then after the target video data is sent to the smart security terminal device, it needs to be sent back to the host device by the smart security terminal device. In this scenario, the target-associated video data transmitted includes both the target video data and the associated video display element data. If the target video data does not need to be processed internally by the video application of the smart security terminal device, then the target video data does not need to be sent to the smart security terminal device, and the smart security terminal device still needs to send the target video data back to the host device. In this scenario, the target-associated video data transmitted may only include the associated video display element data.

[0075] In an optional embodiment of the present invention, determining the target associated transmission video data based on the target processed video data and the associated video display element data may include: if the video data type of the target processed video data is determined to be locally tightly coupled video data, using the target processed video data and the associated video display element data as the target associated transmission video data; if the video data type of the target processed video data is determined to be locally loosely coupled video data, using the associated video display element data as the target associated transmission video data.

[0076] Locally tightly coupled video data can be the type of video data that requires internal processing by video applications on intelligent secure terminal devices. Locally loosely coupled video data can be the type of video data that does not require internal processing by video applications on intelligent secure terminal devices.

[0077] Specifically, the smart security terminal device can parse the video data type of the target processed video data. If the target processed video data is determined to be locally tightly coupled video data, it indicates that the target processed video data needs to be sent to the smart security terminal device's video application for internal processing before being sent to the host device. In this case, the smart security terminal device can simultaneously transmit the target processed video data and associated video display element data as target associated video data. If the target processed video data is determined to be locally loosely coupled video data, it indicates that the target processed video data does not need to be sent to the smart security terminal device's video application for internal processing. In this case, the smart security terminal device can transmit the associated video display element data as target associated video data. In this situation, the target processed video data is usually acquired and generated by the host device and displayed on the host device in combination with the associated video display element data sent by the smart security terminal device.

[0078] For example, the smart security terminal device obtains compressed video data streams from the host device's video server. Alternatively, video data obtained from the host device's camera is compressed into a compressed video data stream by the host device's encoding / decoding module and then sent to the smart security terminal device as a stream file via the host device's video server. Typically, the compressed video data streams received or sent by the smart security terminal device's video applications are encrypted, requiring the video application to access them. Therefore, this compressed video stream can usually be considered as tightly coupled local video data of the smart security terminal device. Optionally, some video applications only acquire and display video from the host device's local camera. If only the host device's local camera video is acquired and displayed, and decryption by the smart security terminal device's video application is not required, such as some dynamic video recognition applications, then this can be considered as loosely coupled local video data of the smart security terminal device.

[0079] It should be noted that after generating the video metadata and UI data compressed stream of the target processed video data, the intelligent security terminal device can tag the video metadata and UI data compressed stream with corresponding labels to identify the corresponding data types. Additionally, if the video data type of the target processed video data is locally tightly coupled video data, and needs to be sent to the host device via the intelligent security terminal device, the intelligent security terminal device can also tag the target processed video data with corresponding labels.

[0080] S140. The target-associated transmission video data is sent to the host device; wherein the host device is used to display video images according to the target-associated transmission video data.

[0081] Correspondingly, the intelligent security terminal device can send the generated target-associated transmission video data to the host device. Upon receiving the target-associated transmission video data, the host device parses it and displays the corresponding video frame based on the parsing results using display hardware such as a screen. It can be understood that the video frame displayed by the host device includes not only the video content corresponding to the target-processed video data but also various types of control data. In other words, the host device can display the complete video frame content.

[0082] It should be noted that for smart security terminal devices, the target processed video data does not need to undergo further decompression and encoding / compression processing at the smart security terminal device. If the smart security terminal device needs to receive the target processed video data, i.e., the video data type of the target processed video data is locally tightly coupled video data, it can directly send the target processed video data as part of the target associated transmission video data to the host device, and simultaneously send the associated video display element data as part of the target associated transmission video data to the host device. If the smart security terminal device does not need to receive the target processed video data, i.e., the video data type of the target processed video data is locally loosely coupled video data, it can only send the associated video display element data as the target associated transmission video data to the host device.

[0083] This invention, through an intelligent security terminal device, determines the target processed video data, generates associated video display element data, and then determines the target associated transmission video data based on the target processed video data and the associated video display element data. This target associated transmission video data is then sent to the host device. The host device receives the target associated transmission video data sent by the intelligent security terminal device and displays the video frame based on it. This technical solution solves the problems of high probability of video data loss and high latency, as well as video frame stuttering, unclear images, or desynchronization caused by repeated encoding and decoding during video data transmission between the intelligent security terminal device and the host device. It reduces the probability of packet loss and latency of video data within the intelligent security terminal device, reduces the number of data links in video data transmission, and maximizes the preservation of the original video quality, thereby improving the image quality of video applications on the intelligent security terminal device and enhancing the user experience.

[0084] Figure 5This is a flowchart of another video data processing method provided by an embodiment of the present invention. This embodiment is applicable to situations where a host device displays video images based on a target-related transmitted video data sent by a smart security terminal device, excluding video compression data. This method can be executed by a video data processing device, which can be implemented in software and / or hardware, and is generally integrated into an electronic device. This electronic device can be a host device, used in conjunction with a smart security terminal device that sends target-related transmitted video data. This embodiment of the present invention does not limit the specific type of host device. Correspondingly, as... Figure 5 As shown, the method includes the following operations:

[0085] S210. Obtain the target associated transmission video data sent by the intelligent security terminal device.

[0086] The intelligent security terminal device uses the video data processing method executed by the intelligent security terminal device in the above embodiment to send target-related transmission video data.

[0087] S220. Determine the target processed video data and the associated video display element data of the target processed video data based on the target associated transmitted video data.

[0088] In this embodiment of the invention, the host device can receive target associated transmission video data sent by the smart security terminal device, and parse the target associated transmission video data, thereby determining the target processed video data and the associated video display element data of the target processed video data based on the parsing result.

[0089] In an optional embodiment of the present invention, determining the target processed video data and the associated video display element data of the target processed video data based on the target associated transmitted video data may include: if it is determined that the target associated transmitted video data includes the target processed video data, obtaining the target processed video data and the associated video display element data from the target associated transmitted video data; if it is determined that the target associated transmitted video data does not include the target processed video data, obtaining the associated video display element data from the target associated transmitted video data, and obtaining the target processed video data from the target peripheral hardware of the local device.

[0090] The target peripheral hardware can be peripheral hardware on the host device, which can collect or generate corresponding video data.

[0091] Optionally, when the target-associated transmitted video data includes target-processed video data, the host device can directly obtain the target-processed video data and its associated video display element data based on the target-associated transmitted video data. When the target-associated transmitted video data does not include target-processed video data, the host device can obtain the associated video display element data of the target-processed video data based on the target-associated transmitted video data, and obtain the corresponding target-processed video data from the target peripheral hardware of the host device, such as obtaining real-time captured video data from the host device's camera.

[0092] In an optional embodiment of the present invention, obtaining the associated video display element data from the target associated transmitted video data may include: determining the data category label of each transmitted video data in the target associated transmitted video data; if the data category label of the currently transmitted video data in the target associated transmitted video data is determined to be a metadata label, determining the currently transmitted video data to be video metadata; if the data category label of the currently transmitted video data in the target associated transmitted video data is determined to be a UI data compressed stream.

[0093] Specifically, the host device can retrieve the corresponding associated video display element data from the target associated transmitted video data based on the tag types carried by the various types of data included in the target associated transmitted video data. Specifically, the host device can first determine the data category tags of each transmitted video data stream in the target associated transmitted video data and analyze these tags. If the data category tag of the currently transmitted video data in the target associated transmitted video data is determined to be a metadata tag, then the currently transmitted video data can be identified as video metadata. If the data category tag of the currently transmitted video data in the target associated transmitted video data is determined to be a UI data compressed stream, then the currently transmitted video data can be identified as UI data compressed stream. Correspondingly, if the target associated transmitted video data also includes target processed video data, and the data category tag of the currently transmitted video data is determined to be a video data tag, then the currently transmitted video data can be identified as target processed video data.

[0094] S230. The video image is displayed based on the target processed video data and the associated video display element data.

[0095] Once the host device has determined the target processed video data and the associated video display element data, it can further display the video image corresponding to the target processed video data on the device's display hardware, such as a display screen, based on the target processed video data and the associated video display element data.

[0096] In an optional embodiment of the present invention, the step of displaying video frames based on the target processed video data and the associated video display element data may include: when it is determined that the target associated transmitted video data includes the target processed video data, decoding and decompressing the target processed video data and the UI data compressed stream to obtain restored video data and UI frame layer data; performing layer compositing processing based on the restored video data, the UI frame layer data, and the video metadata according to layer compositing rules to obtain display frame layer data, and displaying video frames based on the display frame layer data.

[0097] Specifically, the restored video data can be the video data obtained after decompressing the compressed video data stream corresponding to the target processed video data. The display layer data can be the video layer data used for display on the host device.

[0098] Specifically, if the target-related transmitted video data includes target-processed video data, but the smart security terminal device does not repeatedly compress and decompress the target-processed video data, then the target-processed video data sent by the smart security terminal device to the host device is the original compressed video data stream. Simultaneously, the UI data sent by the smart security terminal device is also a compressed UI data stream. Therefore, the host device can first decode and decompress the target-processed video data to obtain the restored video data, and then decode and decompress the UI data compressed stream to obtain the original UI screen layer data.

[0099] The host device sends the decompressed restored video data and UI screen layer data, along with video metadata, to the local device's layer compositing module. The local device's layer compositing module performs layer compositing processing on the restored video data, UI screen layer data, and video metadata according to layer compositing rules to obtain display screen layer data. Based on the display screen layer data, the local device displays the video screen corresponding to the target processed video data on its display screen.

[0100] In an optional embodiment of the present invention, the step of displaying video images based on the target processed video data and the associated video display element data may include: when it is determined that the target associated transmitted video data does not include the target processed video data, decoding and decompressing the UI data compressed stream to obtain UI image layer data; performing layer compositing processing according to layer compositing rules based on the target processed video data, the UI image layer data, and the video metadata to obtain display image layer data, and displaying video images based on the display image layer data.

[0101] Specifically, if the target-related transmitted video data does not include the target-processed video data, the host device can only decode and decompress the UI data compressed stream to obtain the original UI screen layer data. Furthermore, the host device can send the target-processed video data obtained from other channels, along with the UI screen layer data and video metadata, to the local device's layer compositing module. The local device's layer compositing module then performs layer compositing processing on the target-processed video data, UI screen layer data, and video metadata according to layer compositing rules to obtain the display screen layer data. Based on this display screen layer data, the host device displays the video screen corresponding to the target-processed video data on its display screen.

[0102] It is understandable that, since the target processing video data does not need to be sent to the host device through the smart security terminal device, the target processing video data obtained by the scenario or the host device is not a compressed video data stream, but can be raw, uncompressed video data.

[0103] It should be noted that the layer compositing rules used by the layer compositing modules of the host device and the smart security terminal device must be consistent. For example, if the layer compositing module of the smart security terminal device uses the rule that video data is treated as a separate layer, layers below the video data layer are combined into one layer, and layers above the video data layer are combined into one layer, then the host device, when compositing layers, must combine the uncompressed display layer data in the following stacking order: downward layers, video data layer, and upward layers.

[0104] This invention, through an intelligent security terminal device, determines the target processed video data, generates associated video display element data, and then determines the target associated transmission video data based on the target processed video data and the associated video display element data, thereby sending the target associated transmission video data to the host device. The host device receives the target associated transmission video data sent by the intelligent security terminal device, determines the target processed video data and the associated video display element data based on the target associated transmission video data, and then displays the video image based on the target processed video data and the associated video display element data. This technical solution solves the problems of high probability of video data loss and high latency, video stuttering, unclear images, or desynchronization caused by repeated encoding and decoding during video data transmission between the intelligent security terminal device and the host device. It can reduce the probability of packet loss and latency of video data within the intelligent security terminal device, reduce the data links for video data transmission, and maximize the preservation of the original video quality, thereby improving the image quality of video applications on the intelligent security terminal device and enhancing the user experience.

[0105] Figure 6 This is a flowchart of another video data processing method provided in an embodiment of the present invention, such as... Figure 6 As shown, the method includes the following operations:

[0106] S310, the intelligent security terminal device determines the target processed video data; generates associated video display element data of the target processed video data.

[0107] Optionally, the intelligent security terminal device generates associated video display element data of the target processed video data, including: the intelligent security terminal device generating current associated video control layer data of the target processed video data based on operation controls pre-stored in the video application of the local device; the intelligent security terminal device generating current associated system control layer data based on the operating system of the local device; and the intelligent security terminal device generating associated video display element data of the target processed video data based on the current associated video control layer data and the current associated system control layer data.

[0108] Optionally, the intelligent security terminal device generates associated video display element data of the target processed video data based on the currently associated video control layer data and the currently associated system control layer data, including: the intelligent security terminal device performs layer compositing processing on the currently associated video control layer data and the currently associated system control layer data according to layer compositing rules to obtain UI screen layer data; extracts description information data of the UI screen layer from the UI screen layer data to obtain video metadata; and performs encoding and compression processing on the UI screen layer data to obtain a UI data compressed stream.

[0109] S320, the intelligent security terminal device determines the target associated transmission video data based on the target processed video data and the associated video display element data.

[0110] S330, the intelligent security terminal device sends the target-associated transmission video data to the host device.

[0111] Optionally, the intelligent security terminal device determines the target associated transmission video data based on the target processed video data and the associated video display element data, including: when the intelligent security terminal device determines that the video data type of the target processed video data is local tightly coupled video data, it uses the target processed video data and the associated video display element data as the target associated transmission video data; when it determines that the video data type of the target processed video data is local loosely coupled video data, it uses the associated video display element data as the target associated transmission video data.

[0112] S340. The host device determines the target processed video data and the associated video display element data of the target processed video data based on the target associated transmitted video data.

[0113] Optionally, the host device determines the target processed video data and the associated video display element data based on the target associated transmitted video data, including: if the host device determines that the target associated transmitted video data includes the target processed video data, it obtains the target processed video data and the associated video display element data from the target associated transmitted video data; if the host device determines that the target associated transmitted video data does not include the target processed video data, it obtains the associated video display element data from the target associated transmitted video data and obtains the target processed video data from the target peripheral hardware of the host device.

[0114] Optionally, the host device may obtain the associated video display element data from the target associated transmitted video data, which may include: the host device determining the data category label of each transmitted video data in the target associated transmitted video data; if the host device determines that the data category label of the currently transmitted video data in the target associated transmitted video data is a metadata label, the host device determines that the currently transmitted video data is video metadata; if the host device determines that the data category label of the currently transmitted video data in the target associated transmitted video data is a UI data label, the host device determines that the currently transmitted video data is a UI data compressed stream.

[0115] S350, The host device processes the video data according to the target and displays the video screen using the associated video display elements.

[0116] Optionally, the host device displays video frames based on the target processed video data and the associated video display element data, including: when the host device determines that the target associated transmitted video data includes the target processed video data, it decodes and decompresses the target processed video data and the UI data compressed stream to obtain restored video data and UI frame layer data; the host device performs layer compositing processing based on the restored video data, the UI frame layer data, and the video metadata according to layer compositing rules to obtain display frame layer data, and displays video frames based on the display frame layer data.

[0117] Optionally, the host device displays video frames based on the target processed video data and the associated video display element data, including: when the host device determines that the target associated transmitted video data does not include the target processed video data, it decodes and decompresses the UI data compressed stream to obtain UI frame layer data; the host device performs layer compositing processing based on the target processed video data, the UI frame layer data, and the video metadata according to layer compositing rules to obtain display frame layer data, and displays video frames based on the display frame layer data.

[0118] This invention, through an intelligent security terminal device, determines the target processed video data, generates associated video display element data, and then determines the target associated transmission video data based on the target processed video data and the associated video display element data, thereby sending the target associated transmission video data to the host device. The host device receives the target associated transmission video data sent by the intelligent security terminal device, determines the target processed video data and the associated video display element data based on the target associated transmission video data, and then displays the video image based on the target processed video data and the associated video display element data. This technical solution solves the problems of high probability of video data loss and high latency, video stuttering, unclear images, or desynchronization caused by repeated encoding and decoding during video data transmission between the intelligent security terminal device and the host device. It can reduce the probability of packet loss and latency of video data within the intelligent security terminal device, reduce the data links for video data transmission, and maximize the preservation of the original video quality, thereby improving the image quality of video applications on the intelligent security terminal device and enhancing the user experience.

[0119] To more clearly illustrate the technical solutions provided in the embodiments of the present invention, the embodiments of the present invention take tightly coupled video data of a smart security terminal device as an example to specifically explain the video data transmission method between the smart security terminal device and the host device.

[0120] Multiple encoding and decoding of video data leads to generational loss, with each encoding iteration resulting in the loss of more information, causing image details to gradually blur and exhibiting blockiness or ringing effects. It also consumes significant computing resources, especially for high-resolution and high-frame-rate videos, placing a heavy burden on the CPU and GPU. Furthermore, multiple encoding and decoding operations increase processing time, file size, or bandwidth requirements, reducing efficiency and causing significant latency issues for real-time applications.

[0121] To address the aforementioned issues, this invention provides a video data transmission method. After the host device's client app sends a compressed video data stream to the smart security terminal device, the smart security terminal device does not need to encode or decode the compressed video data stream. Instead, the smart security terminal device's device management app directly sends the original compressed video data stream, metadata stream, and UI data stream to the host device. Once the host device's client app receives the multiple data streams, it processes them accordingly and displays the system video feed on the host device's screen.

[0122] Figure 7 This is a flowchart illustrating a video data transmission method between a smart security terminal device and a host device according to an embodiment of the present invention. In a specific example, such as... Figure 7As shown, when the video data to be transmitted is tightly coupled video data of the smart security terminal device, the video data transmission process between the smart security terminal device and the host device may include the following operations:

[0123] The video application of the intelligent security terminal device obtains locally coupled video data from the host device. For example, it obtains compressed video data streams from the host device's video server or real-time video data from peripheral devices such as cameras on the host device. This data is then compressed into a compressed video data stream by the host device's encoding and decoding module. The operating system of the intelligent security terminal device responds to user operations, such as user commands to display the video screen, and runs the video application. The video application's display interface will then be shown on the host device's screen.

[0124] The video application on the smart security terminal device includes video content and dynamic interactive elements such as bullet comments, filters, stickers, and UI controls. The smart security terminal device launches and runs the video application in response to user input. If the video application on the smart security terminal device obtains a compressed video data stream transmitted as a streaming file from a video server, and / or obtains local video from the host device but needs to be transmitted as a streaming file through a video server, then the video application on the smart security terminal device obtains the compressed video data stream from the host device. Specifically, if the compressed video data stream is from a video server, it is obtained from the host device's video server; if the compressed video data stream is from a camera, the camera video from the host device is compressed into a compressed video data stream by the encoding / decoding module on the host device. Further, the compressed video data stream is transmitted to the video application on the smart security terminal device via USB or wirelessly through a client application on the host device. The video data compression streams involved in the above different scenarios are all encoded raw video data, such as H.264 / H.265 / AV1, which can include YUV (Luminance, Chrominance) / RGB (Red, Green and Blue) pixel information and video metadata, as well as timestamp information for audio-visual synchronization, and may also include encoding parameters, resolution, frame rate and bit rate and other related information.

[0125] like Figure 7As shown, the video application of the smart security terminal device directly transmits the compressed video data stream of the video content to the host device through the management APP of the smart security terminal device. The compressed video data stream has an independent video data transmission channel. The management APP of the smart security terminal device can transmit the compressed video data stream to the host device through this independent channel without performing intermediate operations such as layer compositing and encoding / decoding, thereby reducing the computing resources consumed by video data processing and preserving image details to the greatest extent. Dynamic interactive elements involved in the video data, such as bullet comments, filters, stickers, and UI controls, can be pre-stored as operation control layers through the video application. The video application of the smart security terminal device can generate currently associated video control layer data for the dynamic interactive control elements involved in the compressed video data stream. Optionally, the currently associated video control layer data can include one or more control layers, and each control layer can include at least one interactive control. The currently associated video control layer data enters the layer compositing module of the smart security terminal device, and simultaneously, the layer compositing module of the smart security terminal device obtains the currently associated system control layer data of the smart security terminal device through the window manager. Figure 7 The operating system UI layer data shown is used to synthesize the currently associated video control layer data and the currently associated system control layer data in the layer synthesis module of the smart security terminal device to obtain the UI screen layer data.

[0126] When performing layer compositing, the layer compositing module of the smart security terminal device can compose layers according to the layer order. For example, layers below the video data layer can be combined into one layer, and layers above the video data layer can be combined into one layer, with the video data itself treated as a separate layer. Alternatively, the layer compositing module of the smart security terminal device can also compose layers according to other layer compositing principles, such as treating dynamic interactive elements like bullet comments, filters, and stickers as separate layers, and combining video application UI controls with the operating system UI layer into one layer.

[0127] In addition, the layer compositing module of the smart security terminal device can extract layout information, position coordinate information, and other auxiliary information from associated video display element data as video metadata. The smart security terminal device can directly transmit video metadata to the host device via its management app through an independent video metadata data transmission channel. The video metadata provides auxiliary information to help compose and display video images on the host device.

[0128] At the same time, such as Figure 7 As shown, the UI layer data generated by the layer compositing module of the smart security terminal device is encoded and compressed by the encoding and decoding module of the smart security terminal device, becoming a compressed UI data stream. The smart security terminal device can transmit the compressed UI data stream to the host device through the management APP.

[0129] Optionally, the data transmission method used between the intelligent security terminal device and the host device can be wired transmission via USB interface or wireless transmission, etc., and the embodiments of the present invention do not limit this.

[0130] like Figure 7 As shown, in this scenario, the intelligent security terminal device can assign corresponding ID tags to the video data compression stream, UI data compression stream, and video metadata, which are used to distinguish data categories after transmission to the host device. After receiving the data stream, the host device's client application determines the data stream type based on its ID tags. If the host device's client application determines that the ID tag of the currently transmitted video data is a metadata tag, it directly sends the currently transmitted video data to the host device's layer compositing module; if the host device's client application determines that the ID tag of the currently transmitted video data is not a metadata tag, it sends the currently transmitted video data to the host device's encoding / decoding module. The host device can use the encoding / decoding module to re-decode and decompress the UI data compression stream into UI screen layer data, and decode and decompress the video data compression stream into restored video data. The decoded and decompressed UI screen layer data and restored video data are then sent to the host device's layer compositing module.

[0131] In the host device's layer compositing module, using auxiliary information provided by video metadata, the video data and UI layer data are recombined into an uncompressed display layer according to compositing principles, and finally displayed on the host device's screen. The layer compositing rules used by the smart security terminal device and the host device's layer compositing modules should be consistent. For example, when compositing layers, if the data is transmitted based on the layers above and below the video data, then an uncompressed display layer is synthesized according to this layer compositing principle; if other principles are followed, such as separate layers for bullet comments, filters, and stickers, then the UI layer, bullet comments, filters, and stickers are recombined into an uncompressed display layer according to this layer compositing principle.

[0132] Therefore, in the above solution, when the intelligent security terminal device shares tightly coupled video data with the host device, it does not need to encode and decode the video data multiple times. It can directly send the acquired compressed video data stream to the host device, thereby reducing the impact of multiple encoding and decoding operations on video quality. Consequently, when the host device finally synthesizes the uncompressed display image, it can reduce latency during video data transmission and the poor image quality caused by compression processing, significantly reducing the amount of video data transmitted. This improves video data transmission quality and efficiency, enabling efficient output of the video image, maximizing the preservation of the original video quality, and enhancing the user experience of video application visuals.

[0133] To more clearly illustrate the technical solutions provided in the embodiments of the present invention, this embodiment uses loosely coupled video data from a smart security terminal device as an example to specifically explain the video data transmission method between the smart security terminal device and the host device. The loosely coupled video data from the smart security terminal device does not need to be sent to the video application of the smart security terminal device for internal processing. For example, if the video application of the smart security terminal device only acquires and displays video from the local camera of the host device, such as some dynamic video recognition applications, then the acquired video from the local camera of the host device does not need to be sent to the smart security terminal device.

[0134] Figure 8 This is a flowchart illustrating a video data transmission method between a smart security terminal device and a host device according to an embodiment of the present invention. In a specific example, such as... Figure 8 As shown, when the video data to be transmitted is loosely coupled video data of the smart security terminal device, the video data transmission process between the smart security terminal device and the host device may include the following operations:

[0135] The video application of the smart security terminal device includes video content and dynamic interactive elements such as bullet comments, filters, stickers, and UI controls. The smart security terminal device launches and runs the video application in response to user input. Dynamic interactive elements involved in the video data, such as bullet comments, filters, stickers, and UI controls, can be pre-stored as an operation control layer through the video application. The video application of the smart security terminal device can generate currently associated video control layer data from the dynamic interactive control elements involved in the compressed video data stream. Optionally, the currently associated video control layer data can include one or more control layers, and each control layer can include at least one interactive control. The currently associated video control layer data enters the layer compositing module of the smart security terminal device, and simultaneously, the layer compositing module of the smart security terminal device obtains the currently associated system control layer data of the smart security terminal device through the window manager. Figure 8 The operating system UI layer data shown is used to synthesize the currently associated video control layer data and the currently associated system control layer data in the layer synthesis module of the smart security terminal device to obtain the UI screen layer data.

[0136] When performing layer compositing, the layer compositing module of the smart security terminal device can compose layers according to the layer order. For example, layers below the video data layer can be combined into one layer, and layers above the video data layer can be combined into one layer, with the video data itself treated as a separate layer. Alternatively, the layer compositing module of the smart security terminal device can also compose layers according to other layer compositing principles, such as treating dynamic interactive elements like bullet comments, filters, and stickers as separate layers, and combining video application UI controls with the operating system UI layer into one layer.

[0137] In addition, the layer compositing module of the smart security terminal device can extract layout information, position coordinate information, and other auxiliary information from associated video display element data as video metadata. The smart security terminal device can directly transmit video metadata to the host device via its management app through an independent video metadata data transmission channel. The video metadata provides auxiliary information to help compose and display video images on the host device.

[0138] At the same time, such as Figure 8 As shown, the UI layer data generated by the layer compositing module of the smart security terminal device is encoded and compressed by the encoding and decoding module of the smart security terminal device, becoming a compressed UI data stream. The smart security terminal device can transmit the compressed UI data stream to the host device through the management APP.

[0139] Optionally, the data transmission method used between the intelligent security terminal device and the host device can be wired transmission via USB interface or wireless transmission, etc., and the embodiments of the present invention do not limit this.

[0140] like Figure 8 As shown, in this scenario, the intelligent security terminal device can configure corresponding ID tags for the UI data compression stream and video metadata, which are used to distinguish data categories after transmission to the host device. After receiving the data stream, the host device's client app determines the data stream type based on its ID tags. If the host device's client app determines that the ID tag of the currently transmitted video data is a metadata tag, it directly sends the currently transmitted video data to the host device's layer compositing module; if the host device's client app determines that the ID tag of the currently transmitted video data is not a metadata tag, it sends the currently transmitted video data to the host device's encoding / decoding module. The host device can then re-decode and decompress the UI data compression stream into UI image layer data through the encoding / decoding module. The decoded and decompressed UI image layer data is then sent to the host device's layer compositing module.

[0141] like Figure 8As shown, in the host device's layer compositing module, using auxiliary information provided by video metadata, the video data directly provided by the host device client APP (such as camera video data) and UI image layer data are recombined according to compositing principles to form an uncompressed display image layer, which is then displayed on the host device's screen. The layer compositing rules used by the smart security terminal device and the host device's layer compositing modules should be consistent. For example, if the layer compositing is based on the layer data above and below the video data, then the uncompressed display image layer is synthesized according to this layer compositing principle; if it follows other principles, such as separate layers for bullet comments, filters, and stickers, then the UI layer, bullet comments, filters, and stickers are recombined into an uncompressed display image layer according to this layer compositing principle.

[0142] Therefore, in the above solution, when the intelligent security terminal device shares loosely coupled video data with the host device, it not only eliminates the need for multiple encoding and decoding of video data, but also eliminates the need for multiple transmissions of video data. It can directly obtain the original video data from the host device and synthesize an uncompressed display layer on the host device. This better solves the problems of poor image quality caused by latency and compression during video data transmission, further greatly reducing the amount of video data transmitted. It can improve the quality and efficiency of video data transmission, achieve efficient output of video images, maximize the guarantee of the original video image quality, and enhance the user experience of video application image quality.

[0143] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data comply with the relevant laws, regulations and standards of the relevant regions.

[0144] It should be noted that any arrangement or combination of the technical features in the above embodiments also falls within the protection scope of this invention.

[0145] Figure 9 This is a schematic diagram of a video data processing device provided in an embodiment of the present invention, which is configured in a smart security terminal device. Figure 9 As shown, the device includes: a target processing video data determination module 410, an associated video display element data generation module 420, a target associated transmission video data determination module 430, and a target associated transmission video data sending module 440, wherein:

[0146] The target processing video data determination module 410 is used to determine the target processing video data;

[0147] The associated video display element data generation module 420 is used to generate associated video display element data of the target processed video data;

[0148] The target associated transmission video data determination module 430 is used to determine the target associated transmission video data based on the target processed video data and the associated video display element data.

[0149] The target-associated transmission video data sending module 440 is used to send the target-associated transmission video data to the host device;

[0150] The host device is used to transmit video data and display video images according to the target association.

[0151] This invention, through an intelligent security terminal device, determines the target processed video data, generates associated video display element data, and then determines the target associated transmission video data based on the target processed video data and the associated video display element data. This target associated transmission video data is then sent to the host device. The host device receives the target associated transmission video data sent by the intelligent security terminal device and displays the video frame based on it. This technical solution solves the problems of high probability of video data loss and high latency, as well as video frame stuttering, unclear images, or desynchronization caused by repeated encoding and decoding during video data transmission between the intelligent security terminal device and the host device. It reduces the probability of packet loss and latency of video data within the intelligent security terminal device, reduces the number of data links in video data transmission, and maximizes the preservation of the original video quality, thereby improving the image quality of video applications on the intelligent security terminal device and enhancing the user experience.

[0152] Optionally, the associated video display element data generation module 420 is further configured to: generate current associated video control layer data of the target processed video data based on the operation controls pre-stored in the video application of the local device; generate current associated system control layer data based on the operating system of the local device; and generate associated video display element data of the target processed video data based on the current associated video control layer data and the current associated system control layer data.

[0153] Optionally, the associated video display element data generation module 420 is further configured to: perform layer compositing processing on the currently associated video control layer data and the currently associated system control layer data according to layer compositing rules to obtain UI screen layer data; extract description information data of the UI screen layer from the UI screen layer data to obtain video metadata; and perform encoding and compression processing on the UI screen layer data to obtain a UI data compressed stream.

[0154] Optionally, the target associated transmission video data determination module 430 is further configured to: if the video data type of the target processed video data is determined to be local tightly coupled video data, use the target processed video data and the associated video display element data as the target associated transmission video data; if the video data type of the target processed video data is determined to be local loosely coupled video data, use the associated video display element data as the target associated transmission video data.

[0155] The aforementioned video data processing device can execute the video data processing method performed by the intelligent security terminal device provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in this embodiment can be found in the video data processing method performed by the intelligent security terminal device provided in any embodiment of the present invention.

[0156] Figure 10 This is a schematic diagram of another video data processing device provided in an embodiment of the present invention, which is configured on a host device. Figure 10 As shown, the device includes: a target-associated transmission video data acquisition module 510, a target-associated transmission video data processing module 520, and a video display module 530, wherein:

[0157] The target-associated transmission video data acquisition module 510 is used to acquire target-associated transmission video data sent by the intelligent security terminal device; wherein, the intelligent security terminal device sends the target-associated transmission video data using the video data processing method executed by the intelligent security terminal device described above.

[0158] The target-associated transmission video data processing module 520 is used to determine target-processed video data and associated video display element data of the target-processed video data based on the target-associated transmission video data.

[0159] The video display module 530 is used to process video data and display video images based on the target and the associated video display elements.

[0160] This invention, through an intelligent security terminal device, determines the target processed video data, generates associated video display element data, and then determines the target associated transmission video data based on the target processed video data and the associated video display element data, thereby sending the target associated transmission video data to the host device. The host device receives the target associated transmission video data sent by the intelligent security terminal device, determines the target processed video data and the associated video display element data based on the target associated transmission video data, and then displays the video image based on the target processed video data and the associated video display element data. This technical solution solves the problems of high probability of video data loss and high latency, video stuttering, unclear images, or desynchronization caused by repeated encoding and decoding during video data transmission between the intelligent security terminal device and the host device. It can reduce the probability of packet loss and latency of video data within the intelligent security terminal device, reduce the data links for video data transmission, and maximize the preservation of the original video quality, thereby improving the image quality of video applications on the intelligent security terminal device and enhancing the user experience.

[0161] Optionally, the target associated transmission video data processing module 520 is further configured to: if it is determined that the target associated transmission video data includes the target processed video data, obtain the target processed video data and the associated video display element data from the target associated transmission video data; if it is determined that the target associated transmission video data does not include the target processed video data, obtain the associated video display element data from the target associated transmission video data, and obtain the target processed video data from the target peripheral hardware of the local device.

[0162] Optionally, the target associated transmission video data processing module 520 is further configured to: determine the data category label of each transmission video data in the target associated transmission video data; if the data category label of the current transmission video data in the target associated transmission video data is determined to be a metadata label, determine the current transmission video data as video metadata; if the data category label of the current transmission video data in the target associated transmission video data is determined to be a UI data compressed stream.

[0163] Optionally, the video display module 530 is further configured to: when it is determined that the target associated transmission video data includes the target processed video data, decode and decompress the target processed video data and the UI data compressed stream to obtain restored video data and UI screen layer data; perform layer compositing processing according to the restored video data, the UI screen layer data and the video metadata according to the layer compositing rules to obtain display screen layer data, and display the video screen according to the display screen layer data.

[0164] Optionally, the video display module 530 is further configured to: decode and decompress the UI data compression stream to obtain UI screen layer data when it is determined that the target associated transmission video data does not include the target processing video data; perform layer compositing according to the target processing video data, the UI screen layer data and the video metadata according to the layer compositing rules to obtain display screen layer data, and display the video screen according to the display screen layer data.

[0165] The video data processing apparatus described above can execute the video data processing method executed by the host device provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in this embodiment can be found in the video data processing method executed by the host device provided in any embodiment of the present invention.

[0166] Figure 11 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0167] like Figure 11 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0168] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks. It is understood that when the electronic device is a smart security terminal device, its output unit 17 does not include a display.

[0169] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as video data processing methods.

[0170] Optional. The video data processing method can be applied to intelligent security terminal devices, including: determining target processed video data; generating associated video display element data based on the target processed video data; determining target associated transmission video data based on the target processed video data and the associated video display element data; and sending the target associated transmission video data to a host device; wherein the host device is used to display video frames based on the target associated transmission video data.

[0171] Optional. The video data processing method can be applied to a host device, including: acquiring target associated transmission video data sent by a smart security terminal device; wherein the smart security terminal device sends the target associated transmission video data using a video data processing method executed by the smart security terminal device; determining target processed video data and associated video display element data of the target processed video data based on the target associated transmission video data; and displaying video frames based on the target processed video data and the associated video display element data.

[0172] In some embodiments, the video data processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the video data processing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the video data processing method by any other suitable means (e.g., by means of firmware).

[0173] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0174] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0175] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0176] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0177] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0178] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0179] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0180] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video data processing method, characterized in that, Applications in intelligent security terminal devices, including: Identify the target video data to process; Generate associated video display element data for the target processed video data; The target associated transmission video data is determined based on the target processed video data and the associated video display element data; The target-associated video data is sent to the host device; The host device is used to transmit video data and display video images according to the target association.

2. The method according to claim 1, characterized in that, The associated video display element data for generating the target processed video data includes: Generate the current associated video control layer data of the target processed video data based on the operation controls pre-stored in the video application of the local device; Generate the currently associated system control layer data based on the operating system of the local device; Generate associated video display element data for the target processed video data based on the currently associated video control layer data and the currently associated system control layer data.

3. The method according to claim 2, characterized in that, The step of generating associated video display element data for the target processed video data based on the currently associated video control layer data and the currently associated system control layer data includes: The currently associated video control layer data and the currently associated system control layer data are processed by layer composition rules to obtain UI screen layer data. The description information data of the UI screen layers is extracted from the UI screen layer data to obtain video metadata; The UI screen layer data is encoded and compressed to obtain a compressed UI data stream.

4. The method according to claim 1, characterized in that, The step of determining the target associated transmission video data based on the target processed video data and the associated video display element data includes: If the video data type of the target processed video data is determined to be local tightly coupled video data, the target processed video data and the associated video display element data are used as the target associated transmission video data; If the video data type of the target processed video data is determined to be local loosely coupled video data, the associated video display element data is used as the target associated transmission video data.

5. A video data processing method, characterized in that, Applied to host devices, including: Acquire target-related transmitted video data sent by a smart security terminal device; wherein the smart security terminal device sends the target-related transmitted video data using any one of the video data processing methods described in claims 1-4; Based on the target associated transmitted video data, determine the target processed video data and the associated video display element data of the target processed video data; The video image is displayed based on the target processed video data and the associated video display elements.

6. The method according to claim 5, characterized in that, The step of determining the target processed video data and the associated video display element data of the target processed video data based on the target associated transmitted video data includes: If it is determined that the target associated transmission video data includes the target processed video data, the target processed video data and the associated video display element data are obtained from the target associated transmission video data; If it is determined that the target associated transmission video data does not include the target processing video data, the associated video display element data is obtained from the target associated transmission video data, and the target processing video data is obtained from the target peripheral hardware of the local device.

7. The method according to claim 6, characterized in that, Obtaining the associated video display element data from the target associated transmitted video data includes: Determine the data category label for each channel of transmitted video data in the target associated transmitted video data; If the data category tag of the currently transmitted video data in the target associated transmitted video data is determined to be a metadata tag, then the currently transmitted video data is determined to be video metadata; If the data category label of the currently transmitted video data in the target associated transmitted video data is determined to be a UI data label, then the currently transmitted video data is determined to be a UI data compressed stream.

8. The method according to claim 7, characterized in that, The step of displaying video frames based on the target processed video data and the associated video display element data includes: If it is determined that the target associated transmission video data includes the target processed video data, the target processed video data and the UI data compressed stream are decoded and decompressed to obtain restored video data and UI screen layer data. Based on the restored video data, the UI screen layer data, and the video metadata, layer compositing is performed according to layer compositing rules to obtain display screen layer data, and the video screen is displayed based on the display screen layer data.

9. The method according to claim 7, characterized in that, The step of displaying video frames based on the target processed video data and the associated video display element data includes: If it is determined that the target associated transmission video data does not include the target processing video data, the UI data compression stream is decoded and decompressed to obtain UI screen layer data; Based on the target video data, the UI screen layer data, and the video metadata, layer compositing is performed according to layer compositing rules to obtain display screen layer data, and the video screen is displayed based on the display screen layer data.

10. A video data processing apparatus, characterized in that, Configured in intelligent security terminal devices, including: The target processing video data determination module is used to determine the target processing video data; The associated video display element data generation module is used to generate associated video display element data of the target processed video data; A target-associated transmission video data determination module is used to determine target-associated transmission video data based on the target processed video data and the associated video display element data. The target-associated transmission video data sending module is used to send the target-associated transmission video data to the host device; The host device is used to transmit video data and display video images according to the target association.

11. A video data processing apparatus, characterized in that, Configured on the host device, including: A target-associated transmission video data acquisition module is used to acquire target-associated transmission video data sent by a smart security terminal device; wherein, the smart security terminal device sends the target-associated transmission video data using any one of the video data processing methods described in claims 1-4; A target-associated transmission video data processing module is used to determine target-processed video data and associated video display element data of the target-processed video data based on the target-associated transmission video data. The video display module is used to process video data and display the associated video display elements according to the target to display the video screen.

12. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that is executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the video data processing method according to any one of claims 1-4, or to perform the video data processing method according to any one of claims 5-9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the video data processing method according to any one of claims 1-4, or to perform the video data processing method according to any one of claims 5-9.

14. A computer program product comprising a computer program / instructions, wherein, When the computer program / instructions are executed by the processor, they implement the video data processing method according to any one of claims 1-4, or perform the video data processing method according to any one of claims 5-9.

Images