Video workflow data processing system

Abstract

The invention provides a video workflow data processing system. According to the system, collected video data are sent to a server through corresponding communication links by terminal devices in a terminal device cluster so as to generate a video data set, and video workflow data are generated according to link states of the communication links and the video data set, each frame of workflow image comprises at least one layer of image frame, and the resolution of each layer of image frame is associated with the corresponding link state, so that the system can automatically adjust the video quality according to the communication link state between the server and the terminal equipment; and the integrity and fluency of video transmission between the server and each terminal device are ensured.

Description

Technical Field

[0001] The present application relates to data processing technology, and in particular to a video workflow data processing system. Background Art

[0002] With the rapid development of information technology, video data is increasingly used in various fields, such as online conferencing, security monitoring, online education, telemedicine, intelligent transportation, etc. Real-time transmission and processing of video data has become one of the core technologies supporting these applications. However, in practical applications, the transmission of video data faces many challenges.

[0003] Video data has the characteristics of large data volume and high real-time requirements. High-resolution and high-frame-rate video data requires a large amount of network bandwidth, and the complexity and uncertainty of the network environment (such as network congestion, packet loss, delay, etc.) often lead to unstable video data transmission, which in turn affects the video playback quality and user experience.

[0004] To address the challenges in video data transmission, existing technical solutions are optimized mainly from the following aspects: Compression coding technology: By adopting efficient video compression coding algorithms, the size of video data can be reduced and the demand for network bandwidth can be reduced. However, although compression coding technology can improve the transmission efficiency of video data, it will also reduce the quality of the video to a certain extent, especially in the case of high compression ratios.

[0005] Adaptive bitrate adjustment technology: dynamically adjusts the bitrate of video data according to the real-time changes in network bandwidth to adapt to different network environments. However, this technology usually requires frequent communication and negotiation between the server and the terminal device, which increases the complexity and delay of the system. Summary of the invention

[0006] The present application provides a video workflow data processing system, which enables the system to automatically adjust the video quality according to the communication link status between the server and the terminal device, thereby ensuring the integrity and smoothness of the video transmission between the server and each terminal device.

[0007] In a first aspect, the present application provides a video workflow data processing system, including: a terminal device cluster and a server; Each terminal device in the terminal device cluster sends the collected video data to the server through a corresponding communication link to generate a video data set, wherein the video data set includes an image data sequence at each time node; The server generates video workflow data according to the link status of each communication link and the video data set, wherein each workflow image frame in the video workflow data includes at least one layer of image frame, and the resolution of each layered image frame is associated with the corresponding link status; The server sends the corresponding workflow images in the video workflow data to the corresponding terminal devices through each communication link.

[0008] In the above scheme, each terminal device in the terminal device cluster sends the collected video data to the server through the corresponding communication link to generate a video data set, and generates video workflow data according to the link status of each communication link and the video data set, wherein each frame of the workflow image includes at least one layer of image frame, and the resolution of each layered image frame is associated with the corresponding link status, so that the system can automatically adjust the video quality according to the communication link status between the server and the terminal device, ensuring the integrity and smoothness of the video transmission between the server and each terminal device. It is worth noting that the video workflow data generated by the layered image frame mode can reduce the image synthesis time after being transmitted to the terminal device, compared with the traditional server directly transmitting the final image, and can be more adapted to the image processing and image display requirements of the corresponding terminal device.

[0009] Optionally, the data structure of the video workflow data includes: a time guide part and an image data part; Wherein, the time guide part is used to accommodate various time nodes arranged according to time; The image data part includes workflow images at various time nodes, each layered image frame of the workflow image has a different resolution, and each layered image frame is used to accommodate at least one image data in the image data sequence.

[0010] In the above scheme, through the design of the layered image frame, the system can dynamically adjust the resolution of each layered image frame in the workflow image according to the link status of the communication link. For example, when the link status is good, the system can select a high-resolution image frame for transmission to provide a higher quality video experience; and when the link status is poor, a low-resolution image frame can be selected for transmission to ensure the smoothness of the video. In addition, each layered image frame is used to accommodate at least one image data in the image data sequence, and different resolution processing can be achieved for different image data, so as to ensure the quality of the transmitted workflow image while also being able to adapt to the specific link status to ensure the smoothness of the transmission.

[0011] Optionally, the number of layers of the layered images included in the workflow image is associated with the corresponding link status.

[0012] In the above scheme, by associating the number of hierarchical image layers in the workflow image with the link status, the system can perceive and adapt to different network conditions in real time, wherein the above link status, such as bandwidth, delay and packet loss rate, is a key factor affecting the quality of video transmission.

[0013] Optionally, the number of layers of the layered images included in the workflow image changes dynamically as the link state changes.

[0014] In the above scheme, by dynamically adjusting the number of layers of layered images, the system can more effectively utilize network resources. When the link bandwidth is sufficient, the system can select high-resolution layered images for transmission to provide a higher-definition video experience.

[0015] Optionally, each layer of the image frame in the workflow image includes a sequence of accommodation areas corresponding to the number of terminal devices in the terminal device cluster, and each accommodation area in the sequence of accommodation areas is used to correspond to one of the image data in the image data sequence.

[0016] In the above scheme, by designing each layer of the image frame in the workflow image to contain a sequence of accommodation areas corresponding to the number of terminal devices, the system realizes the mapping between video data and terminal devices. Each accommodation area corresponds to the image data of a terminal device, ensuring the accuracy and consistency of video data during processing, transmission and display, which not only simplifies the data processing process, but also improves the efficiency and accuracy of data processing.

[0017] Optionally, after the server sends the corresponding workflow images in the video workflow data to the corresponding terminal devices through each communication link, the method further includes: If the number of image frames in the workflow image is not unique, the terminal device aligns the image frames in the workflow image to generate and display the corresponding target image.

[0018] In the above scheme, through position alignment processing, the terminal device can accurately superimpose multiple image frames together to form a complete target image. This step is crucial to ensure the integrity and accuracy of the video content. In the video workflow, image frames at different levels may contain image data of different resolutions or different perspectives. Through position alignment, these image data can be accurately combined to generate a high-quality fused image. It is worth noting that each image frame in the above workflow image can be partially filled with corresponding image data, while the unfilled area is set to empty.

[0019] Optionally, the link status is the link status of downlink transmission from the server to a corresponding terminal device.

[0020] In the above scheme, by focusing the link status on the downlink transmission from the server to the terminal device, the system can monitor the key indicators of the downlink bandwidth, delay, packet loss rate, etc. in real time. This real-time monitoring mechanism enables the system to quickly perceive changes in the link status and respond in time. For example, when insufficient downlink bandwidth is detected, the system can dynamically adjust the resolution of the image frame in the video stream to ensure smooth transmission of the video content. Based on the real-time monitoring of the downlink status, the system can optimize the transmission strategy of the video stream. When the link status is good, the system can select an image frame with a higher video bit rate and resolution to provide a higher quality video experience. When the link status is poor, the system can choose to reduce the bit rate or resolution of the image frame to reduce the transmission delay and packet loss rate, and ensure the continuity and stability of the video stream. This dynamic transmission strategy adjustment based on the link status can significantly improve the efficiency and quality of video transmission.

[0021] In a second aspect, the present application provides a video workflow data processing method, which is applied to a video workflow data processing system, the system comprising: a terminal device cluster and a server; the method comprising: Each terminal device in the terminal device cluster sends the collected video data to the server through a corresponding communication link to generate a video data set, wherein the video data set includes an image data sequence at each time node; The server generates video workflow data according to the link status of each communication link and the video data set, wherein each workflow image frame in the video workflow data includes at least one layer of image frame, and the resolution of each layered image frame is associated with the corresponding link status; The server sends the corresponding workflow images in the video workflow data to the corresponding terminal devices through each communication link.

[0022] In a third aspect, the present application provides an electronic device, including: processor; and, A memory, configured to store executable instructions of the processor; The processor is configured to perform any possible method described in the first aspect by executing the executable instructions.

[0023] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement any possible method described in the first aspect.

[0024] The present application provides a video workflow data processing system, which sends collected video data to a server through corresponding communication links through each terminal device in a terminal device cluster to generate a video data set, and generates video workflow data according to the link status of each communication link and the video data set, wherein each frame of the workflow image includes at least one layer of image frame, and the resolution of each layered image frame is associated with the corresponding link status, so that the system can automatically adjust the video quality according to the communication link status between the server and the terminal device, thereby ensuring the integrity and smoothness of the video transmission between the server and each terminal device. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0026] Figure 1 is a structural diagram of a video workflow data processing system according to an exemplary embodiment of the present application; Figure 2 is a flowchart of a method for processing video workflow data according to an exemplary embodiment of the present application; Figure 3 is a schematic diagram of mapping a layered image framework and an image data sequence according to an exemplary embodiment of the present application; Figure 4 is a schematic structural diagram of a layer of image framework according to an exemplary embodiment of the present application; Figure 5 is a schematic structural diagram of a multi-layer image framework according to an exemplary embodiment of the present application; Figure 6 is a schematic structural diagram of a layer of image framework according to an exemplary embodiment of the present application; Figure 7 is a schematic structural diagram of a multi-layer image framework according to an exemplary embodiment of the present application; Figure 8 It is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present application.

[0027] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0028] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0029] In order to solve the above problems, the embodiment provided in this application proposes a video workflow data processing system, which mainly includes two major components: a terminal device cluster and a server. The terminal device cluster is composed of multiple devices with video data acquisition functions, such as cameras, video recording devices, etc., which are connected to the server and transmit data through their respective communication links. As the center of data processing and transmission, the server is responsible for receiving video data from the terminal device cluster and generating video workflow data according to the link status of the communication link. Its specific technical concepts are as follows: Layered image frame design: This system adopts a layered image frame design. Each frame of the workflow image contains at least one layer of image frame, and the resolution of each layered image frame is associated with the corresponding link status. Specifically, the server dynamically adjusts the resolution of each layered image frame in the workflow image according to the real-time monitored communication link status (such as bandwidth, delay, packet loss rate, etc.). When the link status is good, a high-resolution image frame is selected to provide a high-quality video experience; when the link status is poor, a low-resolution image frame is selected to ensure the smoothness of the video.

[0030] Differentiated data transmission: The system dynamically adjusts the structure and resolution of the generated video workflow images according to the link status and display capabilities of the terminal devices. For terminal devices with good link status, it provides integrated high-quality images; for terminal devices with poor link status, it provides layered images to reduce the transmission and display burden. This differentiated data transmission strategy not only improves the user experience, but also effectively avoids resource waste.

[0031] Efficient data processing and transmission: By adopting layered image framework design and data differentiation transmission strategy, the system can more effectively utilize network resources and reduce unnecessary data transmission and processing burden. At the same time, the system also has powerful data processing capabilities, and can analyze, process and optimize the received video data in real time to ensure the integrity and smoothness of video transmission.

[0032] Audio signal integrated processing: The system also integrates audio signal detection and processing functions. When an audio signal is detected in the target video data, the corresponding image data is configured into the image frame with the highest resolution to ensure the clear display of the displayed content. This design is particularly important in scenarios such as multi-person online meetings, and can significantly improve the user's communication efficiency and experience.

[0033] Figure 1 FIG. 1 is a schematic diagram of a video workflow data processing system according to an exemplary embodiment of the present application. Figure 1 As shown, the video workflow data processing system 100 provided in this embodiment includes: a terminal device cluster 110 and a server 120 .

[0034] The terminal device cluster 110 is composed of multiple terminal devices, each of which has a video data acquisition function, such as a camera, a video recording device, etc. These terminal devices are connected to the server 120 and transmit data through their respective communication links (such as wired networks, Wi-Fi, 4G / 5G, etc.).

[0035] The server 120, as a data processing and transmission center, is responsible for receiving video data from the terminal device cluster 110 and generating video workflow data according to the link status of the communication link. The server 120 has a powerful data processing capability and can perform real-time analysis, processing and optimization on the received video data.

[0036] In order to further illustrate the architecture and working principle of the video workflow data processing system in this application, several possible application scenarios are provided below: In one possible application scenario, a multi-person online video conference is held, and each participant uses the camera on the terminal device to send the collected video data to the server through their respective communication links. Each terminal device (camera) in the terminal device cluster continuously collects video data. These video data are transmitted to the server in real time through their respective communication links (such as Wi-Fi, 4G / 5G, etc.).

[0037] The server receives video data from the terminal device cluster, and organizes and stores the data according to time nodes to form a video data set, which includes image data sequences at each time node.

[0038] The server generates video workflow data based on the link status of each communication link (such as bandwidth, delay, packet loss rate, etc.) and the video data set. Each workflow image in the video workflow data includes at least one layer of image frame, and the resolution of each layered image frame is associated with the corresponding link status. For example, when the communication link status of a terminal device is good, the server generates a high-resolution image frame for the terminal device; when the link status is poor, a low-resolution image frame is generated.

[0039] The server sends the corresponding workflow images in the video workflow data to the corresponding terminal devices through various communication links. The terminal devices receive and display these workflow images, thereby achieving smooth online video conferencing.

[0040] In addition, the video workflow data processing system in this application can also be used in large-scale events such as sports events, concerts, exhibitions, etc., which require real-time monitoring and recording of every corner of the event. By deploying multiple terminal devices (such as cameras) and connecting them to the server, comprehensive monitoring and recording of the event site can be achieved.

[0041] Figure 2 FIG. 1 is a flow chart of a method for processing video workflow data according to an exemplary embodiment of the present application. Figure 2 As shown, the video workflow data processing method provided in this embodiment includes: S201. Each terminal device in the terminal device cluster sends collected video data to a server through a corresponding communication link to generate a video data set.

[0042] In this step, each terminal device in the terminal device cluster sends the collected video data to the server through a corresponding communication link to generate a video data set, which includes an image data sequence at each time node.

[0043] Specifically, the terminal device cluster includes multiple terminal devices, which may be deployed in different geographical locations or application scenarios. Each terminal device is equipped with a video acquisition module, such as a camera, for capturing video images in real time, and the collected video data exists in the form of continuous image frames. Each terminal device establishes a connection with the server through its own dedicated communication link. These communication links may be wired networks (such as Ethernet) or wireless networks (such as Wi-Fi, 4G / 5G mobile networks), etc. The terminal device encodes and encapsulates the collected image data into data packets in real time, and then sends them to the server through its communication link. The server receives video data packets from each terminal device and organizes them into video data sets based on information such as timestamps.

[0044] S202: The server generates video workflow data according to the link status of each communication link and the video data set.

[0045] In this step, the server generates video workflow data according to the link status of each communication link and the video data set. Each workflow image in the video workflow data includes at least one layer of image frame, and the resolution of each layered image frame is associated with the corresponding link status. The resolution of the image data contained in each layer of image frame is consistent with the resolution of the layered image frame. It can be understood that the resolution of the layered image frame is used to constrain the resolution of the image data contained therein. It is worth noting that the link status is the link status of the server's downlink transmission to the corresponding terminal device.

[0046] Specifically, the video data set is an ordered data structure that contains image data sequences at various time nodes. The time nodes can be fixed time intervals or dynamic time nodes triggered by events. The image data sequence is the set of video frames collected from all terminal devices at these time nodes.

[0047] The server monitors the communication link status between each terminal device in real time. The link status includes key indicators such as bandwidth, delay, and packet loss rate, which directly affect the transmission quality and efficiency of video data. The server obtains link status information by periodically sending detection packets or using existing network protocols. Based on the monitored link status and video data set, the server dynamically generates video workflow data. Each workflow image in the video workflow data is designed to contain at least one layer of image frame. The resolution of these image frames is associated with the corresponding link status: when the link status is good, the image data is configured into a high-resolution image frame; when the link status is poor, the image data is configured into a low-resolution image frame. The server also dynamically adjusts the number of layers of the image frame according to the stability of the link status to adapt to the configuration of image data with different resolutions under different link states.

[0048] Optionally, the data structure of the video workflow data includes: a time guide part and an image data part, wherein the time guide part is used to accommodate each time node arranged according to time.

[0049] The image data part includes workflow images at various time nodes. Each layered image frame of the workflow image has a different resolution. Each layered image frame is used to accommodate at least one image data in the image data sequence.

[0050] Specifically, the main function of the time guide is to accommodate various time nodes arranged in time. These time nodes represent key time points in the video data, and they are used to organize and index the video data in chronological order. Through the time guide, the system can quickly locate the video data at any time point, thereby achieving accurate playback and control of the video. In a specific implementation, the time guide can be implemented using data structures such as linked lists, arrays, or databases. These data structures can efficiently store and retrieve time node information, ensuring that the system can still maintain good performance when processing large amounts of video data. The image data unit contains workflow images at each time node. These workflow images are generated by the server based on the received video data and the link status of the communication link. Each workflow image consists of at least one layer of image frames, which have different resolutions and are used to accommodate at least one image data in the image data sequence.

[0051] Each workflow image is composed of multiple layers of image frames, and the resolution of these image frames is associated with the corresponding link status. When the communication link status is good, the system can select high-resolution image frames to transmit high-quality video data; when the link status is poor, it can select low-resolution image frames to ensure the smoothness of the video.

[0052] In order to implement the design of layered image frames, the system needs to maintain an image frame hierarchy table. This table records the number of layers of workflow images at each time node, the resolution of each layer, and the corresponding link status information. When the server receives new video data, it will select the appropriate image frame from the image frame hierarchy table to accommodate this data based on the current link status.

[0053] After determining the hierarchical image frames, the system needs to accommodate the received image data into these frames. Since each image frame has a different resolution, the system needs to perform scaling, cropping, and other image processing according to the original resolution of the image data and the resolution of the target frame to ensure that the image data can be correctly filled into the frame. In a specific implementation, the system can use image processing algorithms to achieve scaling and cropping of image data. These algorithms can efficiently process large amounts of image data and generate image frames that meet the requirements. At the same time, the system can also perform other processing on the image data according to actual needs, such as enhancement, filtering, etc., to improve the quality and visual effects of the video.

[0054] in, Figure 3 is a schematic diagram of mapping a hierarchical image framework and an image data sequence according to an exemplary embodiment of the present application, such as Figure 3As shown, each layer of the image framework in the workflow image includes a housing area sequence corresponding to the number of terminal devices in the terminal device cluster 110, and each housing area in the housing area sequence is used to correspond to one of the image data in the image data sequence.

[0055] By designing each layer of image frames in the workflow image to contain a sequence of accommodation areas corresponding to the number of terminal devices, the system realizes the mapping between video data and terminal devices. Each accommodation area corresponds to the image data of a terminal device, which ensures the accuracy and consistency of video data during processing, transmission and display, not only simplifying the data processing process, but also improving the efficiency and accuracy of data processing. In addition, since each accommodation area corresponds to the image data of a terminal device, the system can process the video data of multiple terminal devices in parallel. This parallel processing mechanism greatly improves the processing speed of video data, allowing the system to respond to requests from terminal devices more quickly. In addition, each layer of image frames in the workflow image contains a sequence of accommodation areas corresponding to the number of terminal devices, allowing the system to flexibly adapt to different numbers and types of terminal devices. Regardless of how the number of terminal devices changes, the system can adapt to new requirements by adjusting the number and layout of accommodation areas. In addition, in terms of video data display, the system can select a suitable image data frame for display according to the resolution and display capability of the terminal device. Furthermore, each accommodation area corresponds to the image data of a terminal device, and this design also ensures the integrity and consistency of video data. During the video data processing, the system can monitor the data status of each storage area in real time, and promptly detect and handle data loss or damage. At the same time, during the video data transmission and display process, the system can ensure the integrity and consistency of the data through the verification and verification mechanism to avoid video quality problems caused by data errors.

[0056] Finally, the image frames of each layer in the workflow image are designed to contain a sequence of accommodation areas corresponding to the number of terminal devices, which also improves the scalability and maintainability of the system. As the number of terminal devices increases or decreases, the system can easily adapt to new needs by adjusting the number and layout of accommodation areas without large-scale modification or upgrade of the system. In addition, in terms of system maintenance, since each accommodation area corresponds to a specific terminal device, the system can more conveniently perform troubleshooting and repair work, improving the stability and reliability of the system.

[0057] Optionally, the number of layers of layered images included in the above-mentioned workflow image is associated with the corresponding link state. By associating the number of layered image layers in the workflow image with the link state, the system can perceive and adapt to different network conditions in real time, wherein the above-mentioned link state, such as bandwidth, delay and packet loss rate, is a key factor affecting the quality of video transmission. When the link state is good, the system can configure the image data in the video content to a higher resolution image frame, and when the link state is not good, some of the image data with higher priority can be configured to a high-resolution image frame, and some of the image data with lower priority can be configured to a low-resolution image frame, thereby reducing the bandwidth requirement for video transmission and ensuring smooth display of the video. The design of the number of layered image layers associated with the link state also realizes the dynamic adjustment and optimization of the video resolution. During the video transmission process, the system can adjust the resolution of each layered image in the workflow image in real time according to the change of the link state.

[0058] Furthermore, the number of layers of layered images included in the workflow image changes dynamically as the link status changes. By dynamically adjusting the number of layers of layered images, the system can utilize network resources more effectively. When the link bandwidth is sufficient, the system can select high-resolution layered images for transmission to provide a higher-definition video experience. When the link bandwidth is limited, the system can select low-resolution layered images to reduce the amount of transmitted data and ensure continuous playback of the video. This dynamic resolution adjustment mechanism enables the system to provide adaptive video transmission effects under various network conditions. In addition, the design of associating the number of layered image layers with the link status also makes the system more flexible and scalable. The system can flexibly adjust the number of layers and resolution of layered images according to actual needs to adapt to the needs of different application scenarios and terminal devices.

[0059] S203: The server sends corresponding workflow images in the video workflow data to corresponding terminal devices through various communication links.

[0060] In this step, the server sends the corresponding workflow images in the video workflow data to the corresponding terminal devices through various communication links.

[0061] Specifically, the server encapsulates the generated video workflow data into data packets and sends them to each terminal device through the corresponding communication link. During the sending process, the server dynamically adjusts the sending rate and priority of the data packet according to the link status to ensure the smoothness and integrity of the video transmission.

[0062] After receiving the video workflow data packet from the server, the terminal device performs decoding and reassembly operations to restore the original workflow image. If the workflow image contains multiple image frames (that is, the number of layers is greater than 1), the terminal device will perform an alignment operation. Alignment processing refers to accurately aligning multiple image frames in spatial position so that they can be merged into a complete target image for display. Alignment processing may involve technical means such as image transformation (such as translation, rotation, scaling, etc.) and interpolation algorithms. The terminal device selects the appropriate workflow image for display or storage based on its own display capabilities and user needs.

[0063] If the number of image frames in the workflow image is not unique, the terminal device aligns the image frames in the workflow image to generate and display the corresponding target image. Through position alignment, the terminal device can accurately superimpose multiple image frames to form a complete target image. This step is critical to ensuring the integrity and accuracy of the video content. In the video workflow, image frames at different levels may contain image data of different resolutions or different perspectives. Through position alignment, these image data can be accurately combined to generate a high-quality fused image. It is worth noting that each image frame in the above-mentioned workflow image can be partially filled with corresponding image data, while the unfilled area is set to empty.

[0064] In the above scheme, each terminal device in the terminal device cluster sends the collected video data to the server through the corresponding communication link to generate a video data set, and generates video workflow data according to the link status of each communication link and the video data set, wherein each frame of the workflow image includes at least one layer of image frame, and the resolution of each layered image frame is associated with the corresponding link status, so that the system can automatically adjust the video quality according to the communication link status between the server and the terminal device, thereby ensuring the integrity and smoothness of the video transmission between the server and each terminal device.

[0065] Based on the above embodiment, in a possible implementation, the first terminal device in the terminal device cluster sends the collected first video data to the server via the first communication link, and the second terminal device sends the collected second video data to the server via the second communication link. The video data set includes a target data sequence at a target time node, and the target data sequence includes the first image data of the first video data at the target time node and the second image data of the second video data at the target time node.

[0066] Figure 4 FIG. 1 is a schematic diagram of a structure of a layer of image frame according to an exemplary embodiment of the present application. Figure 4As shown, if the first communication link and the second communication link are in the same state, each frame of the workflow image in the video workflow data includes only one layer of image frame, namely the first layer of image frame 310, and the first image data and the second image data are accommodated in the one layer of image frame (as shown in the filled part).

[0067] The system monitors and evaluates the link status of the first communication link and the second communication link in real time to identify whether the two are in the same state. When it is determined to be the same state, the system automatically triggers a specific video workflow data processing logic. This technical solution effectively avoids confusion or errors in processing logic caused by differences in link status, and ensures the consistency and predictability of system behavior under different link environments. By uniformly processing situations where the link status is the same, the system can perform subsequent video workflow data processing tasks more efficiently. Among them, when the link status of the first communication link and the second communication link is good, the first layer image frame 310 is configured with high resolution, and when the link status of the first communication link and the second communication link is poor, the first layer image frame 310 is configured with low resolution.

[0068] In a single-layer image frame, the system cleverly accommodates the first image data and the second image data. Through advanced data integration technology, the system can seamlessly integrate data from different links into the same image frame while maintaining the integrity and accuracy of the data. It achieves efficient integration and presentation of multi-source data, providing users with a richer and more comprehensive video workflow data view. Users can simultaneously observe data information from different links in the same image frame.

[0069] If the first communication link and the second communication link are in different states, and the link state of the first communication link is better than the link state of the second communication link, the first workflow image in the video workflow data includes a first layer image frame 310 (such as Figure 4 as shown). Figure 5 FIG. 1 is a schematic diagram of a multi-layer image framework according to an exemplary embodiment of the present application. Figure 5 As shown, the second workflow image includes a first layer image frame 310 for accommodating the first image data and a second layer image frame 320 for accommodating the second image data, wherein the resolution of the first layer image frame 310 is higher than the resolution of the second layer image data, the first workflow image is used to be sent down to the first terminal device, and the second workflow image is used to be sent down to the second terminal device.

[0070] In view of the difference in link status, the system adopts a layered image frame design strategy. In the first workflow image, the first layer image frame is designed to accommodate the first image data and the second image data at the same time; while in the second workflow image, the first layer image frame is designed to accommodate the first image data, and the second layer image frame is additionally designed to accommodate the second image data. At the same time, ensure that the resolution of the first layer image frame is higher than that of the second layer image frame.

[0071] The layered image frame design achieves differentiated presentation and efficient transmission of data. The high-resolution first-layer image frame ensures the clarity and integrity of key data, while the low-resolution second-layer image frame reduces the occupancy of bandwidth resources. This design not only meets the needs of high-quality data transmission, but also takes into account the effective use of system resources.

[0072] According to the difference in link status, the system generates a first workflow image and a second workflow image with different data storage structures and resolution characteristics. The first workflow image is oriented to the first terminal device with a better link status, providing an integrated high-quality data view; while the second workflow image is oriented to the second terminal device with a relatively poor link status, providing a hierarchical data view. The data differentiation presentation technology ensures that terminal devices in different link states can receive the data that best suits their transmission capabilities and display requirements. This not only improves the user experience, but also avoids resource waste or user dissatisfaction caused by excessive data transmission or poor display effects.

[0073] In other words, the system dynamically adjusts the structure and resolution of the generated video workflow image according to the link status and display capabilities of the terminal device. For the first terminal device with a better link status, an integrated high-quality image is provided; for the second terminal device with a relatively poor link status, a layered display image is provided to reduce the transmission and display burden. The terminal device adaptability optimization technology significantly improves the system's compatibility and support capabilities for different terminal devices. Regardless of the network environment or display capabilities of the terminal device, the system can provide an adaptive data transmission and display solution to ensure user experience.

[0074] In addition, through the layered image framework design and data differentiation presentation technology, the system can dynamically adjust the data transmission volume and display quality according to the link status and terminal device requirements. Under the premise of ensuring data integrity and clarity, the occupation of system resources is reduced as much as possible. The efficient utilization technology of system resources significantly improves the overall performance and stability of the system. By reducing unnecessary data transmission and display burdens, the system can more efficiently utilize computing resources, bandwidth resources, and storage resources, thereby improving the system's response speed and processing capabilities. At the same time, this also reduces the system's operation and maintenance costs and energy consumption levels.

[0075] Figure 6 FIG. 1 is a schematic diagram of a structure of a layer of image frame according to an exemplary embodiment of the present application. Figure 6 As shown, the third terminal device in the terminal device cluster sends the collected third video data to the server through the third communication link, and the target data sequence includes the third image data of the third video data at the target time node. If the first communication link, the second communication link and the third communication link are in the same state, each workflow image frame in the video workflow data includes only one layer of image frame, that is, the first layer of image frame 410, and the first image data, the second image data and the third image data are accommodated in the one layer of image frame.

[0076] If the first communication link and the second communication link are in the same state, and the link states of the first communication link and the second communication link are better than the link state of the third communication link, the first workflow image in the video workflow data includes a first layer image frame 410 (as shown in the figure) for accommodating the first image data, the second image data and the third image data. Figure 7 FIG. 1 is a schematic diagram of a multi-layer image framework according to an exemplary embodiment of the present application. Figure 7 As shown, the second workflow image includes a first-layer image frame 410 for accommodating the first image data and the second image data and a second-layer image frame 420 for accommodating the third image data, wherein the resolution of the first-layer image frame 410 is higher than that of the second-layer image frame 420, the first workflow image is used to be sent to the first terminal device and the second terminal device, and the second workflow image is used to be sent to the third terminal device. It is worth noting that if there are other terminal devices, a third-layer image frame 430 can be further set to accommodate the corresponding data image.

[0077] The system forms a link status evaluation model by real-time monitoring of the link status (such as bandwidth, delay, packet loss rate and other parameters) of the first communication link, the second communication link and the third communication link. When the link status of the first communication link and the second communication link are consistent and both are better than the third communication link, the system automatically triggers the differentiated workflow image generation logic based on the link status. In view of the link status difference, the system adopts the layered image framework technology to design the first workflow image as a single-layer high-resolution framework (first-layer image framework) for accommodating the first image data, the second image data and the third image data; while the second workflow image is designed as a double-layer framework structure, in which the first-layer image framework (high resolution) accommodates the first image data and the second image data, and the second-layer image framework (low resolution) accommodates the third image data. This design ensures that when the link status is limited, the continuity of the overall data transmission can be guaranteed by reducing the transmission quality (resolution) of some image data.

[0078] The first-layer image frame adopts a high-resolution design to ensure that when it is transmitted to the first terminal device and the second terminal device, it can maintain a high image quality and meet the requirements of high-precision visual processing. For the third communication link with poor link status, the system uses a low-resolution second-layer image frame to accommodate the third image data, and reduces the data transmission volume by reducing the image resolution, thereby achieving stable transmission of the third image data under limited link bandwidth.

[0079] The first workflow image contains all image data (first image data, second image data and third image data), adopts a high-resolution first-layer image framework, and is suitable for the first terminal device and the second terminal device with good link status. This strategy ensures that the terminal device can receive complete and high-resolution video workflow data in a high-quality link environment. The second workflow image is designed for the third terminal device with limited link status, and realizes differentiated transmission of image data through a layered framework structure. The high-resolution first-layer image framework ensures the transmission quality of the first image data and the second image data, while the low-resolution second-layer image framework ensures the stable transmission of the third image data under limited bandwidth. This strategy effectively balances the data transmission requirements under different link states and improves the data transmission efficiency of the overall system.

[0080] If the first communication link and the second communication link are in the same state, and the link state of the third communication link is better than the link states of the first communication link and the second communication link, the first workflow image in the video workflow data includes a first layer image frame 410 for accommodating the third image data and a second layer image frame 420 for accommodating the first image data and the second image data, and the second workflow image includes a first layer image frame 410 for accommodating the first image data, the second image data and the third image data, wherein the resolution of the first layer image frame 410 is higher than the resolution of the second layer image frame 420, the first workflow image is used to be sent to the first terminal device and the second terminal device, and the second workflow image is used to be sent to the third terminal device.

[0081] Through the differentiated workflow image generation and distribution strategy based on link status, the system can dynamically adjust the transmission mode of image data according to the actual status of different links, thereby maximizing the use of limited link bandwidth resources while ensuring data transmission continuity. The differentiated workflow image design fully considers the performance differences of different terminal devices, ensuring that under various link states, terminal devices can receive video workflow data suitable for their processing capabilities. This design avoids resource waste and performance bottlenecks, and improves the overall system operation efficiency.

[0082] It is worth noting that the layered image framework technology and differentiated workflow image generation logic adopt modular design ideas to facilitate the expansion and upgrade of system functions. In the future, more layers of image frameworks can be added or the generation rules of workflow images can be adjusted according to actual needs to adapt to the changing link status and terminal device requirements. Through standardized interface and protocol design, the system can be compatible with various types of terminal devices and communication links, ensuring stable and efficient data transmission in different network environments. This design enhances the cross-platform compatibility and interoperability of the system, laying a solid foundation for the wide application of the system.

[0083] On the basis of the above embodiments, it is worth explaining that if the workflow image includes a multi-layer image frame, then when it is determined that there is an audio signal in the target video data corresponding to the target terminal device in the video data set, each target image data corresponding to the target terminal device is configured in the target image frame in the multi-layer image frame, wherein the target image frame is the image frame with the highest resolution in the multi-layer image frame.

[0084] The system integrates audio signal detection to analyze the audio stream in the target video data in real time and identify whether there is an audio signal. This mechanism can be based on audio feature extraction algorithms (such as Mel-frequency cepstral coefficient MFCC analysis) to ensure high-precision judgment of the existence of audio signals and avoid misjudgment or missed judgment.

[0085] When an audio signal is detected in the target video data, the system automatically triggers the image frame allocation logic, and uniformly configures all target image data corresponding to the target terminal device (including but not limited to video frames, key frames, etc.) to the target image frame with the highest resolution in the multi-layer image frame. This process is implemented through an association mapping table between image data and audio signals to ensure the accuracy and real-time nature of the allocation. It is worth noting that in a multi-person online conference scenario, if an audio signal is detected in the target video data corresponding to the target terminal device, it means that the user corresponding to the target terminal device is performing voice input, and the corresponding content may be being displayed. Therefore, by configuring the corresponding image data to the target image frame with the highest resolution, the clear display of the displayed content can be ensured.

[0086] Figure 8 is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present application. Figure 8 As shown, an electronic device 500 provided in this embodiment includes: a processor 501 and a memory 502; wherein: The memory 502 is used to store computer programs, and the memory may also be a flash memory.

[0087] The processor 501 is used to execute the execution instructions stored in the memory to implement each step in the above method. For details, please refer to the relevant description in the above method embodiment.

[0088] Optionally, the memory 502 may be independent or integrated with the processor 501 .

[0089] When the memory 502 is a device independent of the processor 501, the electronic device 500 may further include: The bus 503 is used to connect the memory 502 and the processor 501 .

[0090] This embodiment further provides a readable storage medium, in which a computer program is stored. When at least one processor of an electronic device executes the computer program, the electronic device executes the methods provided in the above-mentioned various implementation modes.

[0091] This embodiment also provides a program product, which includes a computer program stored in a readable storage medium. At least one processor of the electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device implements the methods provided in the above various embodiments.

[0092] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0093] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A video workflow data processing system, characterized in that: include: Terminal device clusters and servers; Each terminal device in the terminal device cluster sends the collected video data to the server through a corresponding communication link to generate a video data set, wherein the video data set includes an image data sequence at each time node; The server generates video workflow data according to the link status of each communication link and the video data set, wherein each workflow image frame in the video workflow data includes at least one layer of image frame, and the resolution of each layered image frame is associated with the corresponding link status; The server sends the corresponding workflow images in the video workflow data to the corresponding terminal devices through each communication link.

2. The video workflow data processing system according to claim 1, characterized in that: The data structure of the video workflow data includes: a time guide part and an image data part; Wherein, the time guide part is used to accommodate various time nodes arranged according to time; The image data part includes workflow images at various time nodes, each layered image frame of the workflow image has a different resolution, and each layered image frame is used to accommodate at least one image data in the image data sequence.

3. The video workflow data processing system according to claim 2, characterized in that: The number of layers of the layered images included in the workflow image is associated with the corresponding link status.

4. The video workflow data processing system according to claim 3, characterized in that: The number of layers of the hierarchical images included in the workflow image changes dynamically as the link status changes.

5. The video workflow data processing system according to any one of claims 1 to 4, characterized in that: Each layer of the image frame in the workflow image includes a sequence of accommodation areas corresponding to the number of terminal devices in the terminal device cluster, and each accommodation area in the sequence of accommodation areas is used to correspond to one of the image data in the sequence of image data.

6. The video workflow data processing system according to claim 5, characterized in that: After the server sends the corresponding workflow images in the video workflow data to the corresponding terminal devices through each communication link, the method further includes: If the number of image frames in the workflow image is not unique, the terminal device aligns the image frames in the workflow image to generate and display the corresponding target image.

7. The video workflow data processing system according to any one of claims 1 to 4, characterized in that: The link status is the link status of downlink transmission from the server to the corresponding terminal device.

8. A video workflow data processing method, characterized in that: Applied to a video workflow data processing system, the system comprises: a terminal device cluster and a server; the method comprises: Each terminal device in the terminal device cluster sends the collected video data to the server through a corresponding communication link to generate a video data set, wherein the video data set includes an image data sequence at each time node; The server generates video workflow data according to the link status of each communication link and the video data set, wherein each workflow image frame in the video workflow data includes at least one layer of image frame, and the resolution of each layered image frame is associated with the corresponding link status; The server sends the corresponding workflow images in the video workflow data to the corresponding terminal devices through each communication link.

9. An electronic device, characterized in that: include: processor; as well as, A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to perform the method of claim 8 by executing the executable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method of claim 8 when executed by a processor.

Images