A multimedia data transmission method and its system

Abstract

The present application provides a multimedia data transmission method and its system, which relates to the technical field of video data transmission. By performing inter-frame difference decoding verification on the original video stream collected by the multimedia server side, a core data stream is obtained. The core pixels transmitted by the multimedia server side are determined from the core data stream. Based on the core pixels, image reconstruction is performed on the image frames in the original video stream to obtain a core reconstructed image. Morphological filtering is performed on the core reconstructed image according to the pixel information of the core reconstructed image to obtain reconstructed core points and boundary tampering points. Then, spatio-temporal correlation aggregation is performed on the boundary tampering points through temporal features and spatial features to obtain a correlation aggregation image. Finally, the correlation aggregation image is repaired and transmitted through the image repair operator determined by the reconstructed core points to obtain the video stream received by the multimedia client. The present application can achieve anti-tampering filtering and repair of video data during efficient video transmission to improve the quality of the video stream received by the multimedia client.

Description

Technical Field

[0001] This application relates to the technical field of video data transmission. More specifically, this application relates to a multimedia data transmission method and its system. Background Art

[0002] Video data transmission, as an important part of modern information technology, is widely used in scenarios such as video conferencing, real-time monitoring, and live broadcast services. Among them, the transmission speed of video data and the quality of image frames are the key to user experience.

[0003] With the rapid development of video technology, multimedia data has been widely used in various application scenarios, such as high-definition video live broadcast, video surveillance, remote conferencing, and virtual reality. This has led to higher quality requirements for the real-time performance, reliability, and quality of video data transmission. However, video data usually contains a large amount of redundant information, which will occupy a large amount of network bandwidth during the acquisition and transmission processes on the multimedia server side, resulting in problems such as delay, packet loss, and jitter in the network environment for multimedia video data transmission. Existing methods do not fully extract the key information in video data before video data transmission, and are also vulnerable to data tampering or attacks during the transmission process, making it difficult to repair image frames by inter-frame difference decoding, resulting in a decline in the quality of the video stream received by the client and unable to ensure the integrity of the video stream received by the client. Therefore, how to achieve anti-tampering filtering and repair of video data during efficient video transmission to improve the quality of the video stream received by the multimedia client has become a difficult problem faced by the industry. Summary of the Invention

[0004] This application provides a multimedia data transmission method and its system, which can achieve anti-tampering filtering and repair of video data during efficient video transmission to improve the quality of the video stream received by the multimedia client.

[0005] In the first aspect, this application provides a multimedia data transmission method, and the transmission method includes the following steps:

[0006] Obtain the original video stream collected by the multimedia server side;

[0007] Perform inter-frame difference decoding verification on the original video stream to obtain the core data stream, determine the core pixels transmitted by the multimedia server side through the core data stream, and then reconstruct the image frames in the original video stream based on the core pixels to obtain the core reconstructed image;

[0008] Perform morphological filtering on the core reconstructed image according to the pixel information of the core reconstructed image to obtain the reconstruction core points and boundary tampering points during image reconstruction;

[0009] Obtain the spatio-temporal information of the original video stream, and perform spatio-temporal correlation aggregation on the boundary tampering points through the time features and space features in the spatio-temporal information to obtain an associated aggregated image;

[0010] Determine an image restoration operator according to the reconstructed core points, and perform restoration transmission on the associated aggregated image based on the image restoration operator, so as to obtain the video stream received by the multimedia client.

[0011] In this embodiment, the original video stream collected by the multimedia server is obtained by connecting to the secure access machine.

[0012] In this embodiment, performing inter-frame differential decoding verification on the original video stream to obtain the core data stream specifically includes:

[0013] Perform frame-by-frame decoding on the original video stream to obtain a continuous sequence of image frames;

[0014] Determine the pixel threshold through the sequence of image frames;

[0015] Filter the core data of each image frame in the sequence of image frames according to the pixel threshold, so as to obtain the core data stream.

[0016] In this embodiment, determining the core pixels transmitted by the multimedia server through the core data stream specifically includes:

[0017] Perform spatio-temporal collaborative filtering on the core data stream to obtain a joint mask;

[0018] Filter the core pixel data according to the joint mask;

[0019] Encode the core pixel data to obtain the core pixels transmitted by the multimedia server.

[0020] In this embodiment, performing image reconstruction on the image frames in the original video stream based on the core pixels to obtain the core reconstructed image specifically includes:

[0021] Construct a blank image matrix;

[0022] Determine the compensation domain of the image frames in the original video stream;

[0023] Use the blank image matrix to perform compensation reconstruction on the compensation domain based on the core pixels to obtain the core reconstructed image.

[0024] In this embodiment, performing morphological filtering on the core reconstructed image according to the pixel information of the core reconstructed image to obtain the reconstructed core points and boundary tampering points during image reconstruction specifically includes:

[0025] Perform dilation and erosion filtering operations on the core reconstructed image respectively to obtain the filtered coincidence area of the core reconstructed image;

[0026] Screen the reconstruction core points during image reconstruction from the filtered coincidence area;

[0027] Based on the filtered coincidence area, mark the boundary tampering points during image reconstruction from the core reconstructed image through an edge detection algorithm.

[0028] In this embodiment, the spatio-temporal information of the original video stream is obtained through a spatio-temporal segmentation algorithm.

[0029] In this embodiment, the multimedia server is a hybrid communication server.

[0030] In this embodiment, the multimedia client is a software client.

[0031] In a second aspect, the present application provides a multimedia data transmission system for performing a multimedia data transmission method. The transmission system includes:

[0032] An acquisition module for acquiring the original video stream collected by the multimedia server side;

[0033] A pixel reconstruction module for performing inter-frame difference decoding verification on the original video stream to obtain a core data stream, determining the core pixels transmitted by the multimedia server side through the core data stream, and then performing image reconstruction on the image frames in the original video stream based on the core pixels to obtain a core reconstructed image;

[0034] An image filtering module for performing morphological filtering on the core reconstructed image according to the pixel information of the core reconstructed image to obtain the reconstruction core points and boundary tampering points during image reconstruction;

[0035] An association aggregation module for obtaining the spatio-temporal information of the original video stream, and performing spatio-temporal association aggregation on the boundary tampering points through the time features and space features in the spatio-temporal information to obtain an associated aggregated image;

[0036] A repair transmission module for determining an image repair operator according to the reconstruction core points, performing repair transmission on the associated aggregated image based on the image repair operator, and then obtaining the video stream received by the multimedia client.

[0037] The technical solutions provided by the embodiments disclosed in the present application have the following beneficial effects:

[0038] First, obtain the original video stream collected by the multimedia server; secondly, perform inter-frame differential decoding verification on the original video stream to obtain the core data stream, determine the core pixels transmitted by the multimedia server through the core data stream, and then perform image reconstruction on the image frames in the original video stream based on the core pixels to obtain the core reconstructed image; then, perform morphological filtering on the core reconstructed image according to the pixel information of the core reconstructed image to obtain the reconstructed core points and boundary tampering points during image reconstruction; then, obtain the spatio-temporal information of the original video stream, and perform spatio-temporal correlation aggregation on the boundary tampering points through the time feature and space feature in the spatio-temporal information to obtain the associated aggregated image; finally, determine the image repair operator according to the reconstructed core points, perform repair transmission on the associated aggregated image based on the image repair operator, and then obtain the video stream received by the multimedia client.

[0039] It can be seen that in this application, video data anti-tampering filtering and repair can be achieved during efficient video transmission. First, by obtaining the original video stream from the server side, basic data is provided for subsequent operations, and frame-by-frame decoding and differential analysis are performed to extract the core data stream in the video, so as to reduce the transmission of redundant information and reduce the bandwidth pressure; secondly, the core pixels to be transmitted are determined through the core data stream, and then image reconstruction is performed to obtain the core reconstructed image, which can supplement the lost or tampered part, and by extracting and using the core pixels, the lost or incorrect data during transmission can be solved, thus avoiding the decline in the quality of the transmitted image frames; then, through dilation and erosion morphological filtering operations on the core reconstructed image, the tampered areas and core pixels are further extracted from the core reconstructed image, providing accurate points to be repaired and key data for image repair and integrity assurance, which is beneficial to the high-quality output of the image, and considering both time and space factors, the boundary tampering points are spatio-temporally correlated, which can effectively repair the errors caused by packet loss or tampering during network transmission and ensure the integrity of the repaired image; finally, based on the repair operator, repair transmission is performed on the associated aggregated image, which can ensure that the quality of the video stream finally received by the client is not affected.

[0040] In summary, the technical solution adopted in this application can achieve video data anti-tampering filtering and repair during efficient video transmission to improve the quality of the video stream received by the multimedia client. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for description in the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0042] Figure 1 is a flowchart of a multimedia data transmission method provided according to this application;

[0043] Figure 2 is an exemplary flowchart for determining a core data stream provided according to this application;

[0044] Figure 3 is an exemplary flowchart for determining a reconstruction core point and a boundary tampering point during image reconstruction provided according to this application;

[0045] Figure 4 is a module structure diagram of a multimedia data transmission system provided according to this application. Specific Embodiments

[0046] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0047] The embodiments of this application provide a multimedia data transmission method and its system. The core is to obtain the original video stream collected by the multimedia server side; then perform inter-frame difference decoding verification on the original video stream to obtain the core data stream, determine the core pixels transmitted by the multimedia server side through the core data stream, and then perform image reconstruction on the image frames in the original video stream based on the core pixels to obtain the core reconstructed image; then, perform morphological filtering on the core reconstructed image according to the pixel information of the core reconstructed image to obtain the reconstruction core point and the boundary tampering point during image reconstruction; then, obtain the spatio-temporal information of the original video stream, and perform spatio-temporal correlation aggregation on the boundary tampering points through the time feature and the space feature in the spatio-temporal information to obtain the associated aggregated image; finally, determine the image repair operator according to the reconstruction core point, perform repair transmission on the associated aggregated image based on the image repair operator, and then obtain the video stream received by the multimedia client.

[0048] Embodiment 1. To better understand the above technical solutions, the above technical solutions will be described in detail below in conjunction with the accompanying drawings of the specification and specific embodiments. Refer to Figure 1 As shown, this figure is an exemplary flowchart of a multimedia data transmission method shown in this embodiment of this application. The transmission method includes the following steps:

[0049] In step S1, obtain the original video stream collected by the multimedia server side.

[0050] In specific implementation, the original video stream collected by the multimedia server is obtained by connecting to the secure access machine. A physical or logical connection can be established between the secure access machine and the multimedia server to ensure the integrity and confidentiality of data during transmission. In actual implementation, a network tunnel (such as SSL VPN) can be configured through the secure access machine and a secure protocol can be used to complete the mutual authentication between the client and the secure access machine, so as to decrypt the video stream. After verifying the data integrity, the decrypted video stream is used as the original video stream collected by the multimedia server side.

[0051] It should be noted that the acquisition of the original video stream through the secure access machine in this application can effectively enhance the security and stability of data transmission, which is beneficial to ensuring the confidentiality of video data, initially preventing unauthorized devices or users from accessing illegally, and preventing data from being intercepted or tampered with. Therefore, it is beneficial to improve the confidentiality and anti-tampering ability of the original video stream during the transmission from the multimedia server to the multimedia client.

[0052] In step S2, the inter-frame difference decoding check is performed on the original video stream to obtain the core data stream. The core pixels transmitted by the multimedia server side are determined through the core data stream, and then the image frames in the original video stream are reconstructed based on the core pixels to obtain the core reconstructed image.

[0053] Preferably, in this embodiment, refer to Figure 2 As shown, this figure is an exemplary flowchart for determining the core data stream in the embodiment of this application. The inter-frame difference decoding check on the original video stream to obtain the core data stream in this embodiment can be implemented by the following steps:

[0054] First, in step S21, the original video stream is decoded frame by frame to obtain a continuous sequence of image frames;

[0055] Then, in step S22, the pixel threshold is determined through the sequence of image frames;

[0056] Finally, in step S23, the core data of each image frame in the sequence of image frames is screened according to the pixel threshold, and then the core data stream is obtained.

[0057] In specific implementation, first, the original video stream can be decoded into multiple independent image frames through a standard video decoding algorithm, and the sequence obtained by sorting all the image frames according to the generation time is used as the image frame sequence. As a preferred embodiment, the H.265 can be used as the standard video decoding algorithm to facilitate the transmission of higher-quality video data under limited bandwidth. In other embodiments, other standard video decoding algorithms can also be adopted, which are not limited here. Then, for each image frame in the image frame sequence, the value obtained by weighted summation of the average value and standard deviation of all pixels in the image frame is used as the weighted pixel value of the image frame, and thus the weighted pixel values of each image frame in the image frame sequence are obtained. Then, the average value of all the weighted pixel values is used as the pixel threshold. Among them, the summation weights of the average value and standard deviation of all pixels in each image frame can be determined by regression analysis. Finally, for each image frame in the image frame sequence, the pixels greater than the pixel threshold in the image frame are used as the core data of the image frame, and thus the core data of each image frame in the image frame sequence are obtained, and all the core data are compressed and encoded by a Huffman encoder to obtain the core data stream.

[0058] It should be noted that by extracting the core data stream, the present application can retain the core video data, reduce the bandwidth requirement for the transmission of the video data stream, improve the transmission speed, and is conducive to effectively filtering and identifying malicious tampering behaviors during transmission, thereby being conducive to improving the transmission efficiency from the multimedia server to the multimedia client.

[0059] In this embodiment, the core pixels transmitted by the multimedia server side can be determined through the core data stream in the following specific manner, that is:

[0060] Perform spatio-temporal collaborative filtering on the core data stream to obtain a joint mask;

[0061] Screen the core pixel data according to the joint mask;

[0062] Encode the core pixel data to obtain the core pixels transmitted by the multimedia server side.

[0063] In specific implementation, by calculating the pixel difference and fluctuation amplitude of the inter-frame change, calculating the regions with no significant change in the core data stream, and screening the important regions of the core data stream according to neighborhood similarity, and then encoding the filtered core data as 1 and the filtered-out core data as 0 to obtain the joint mask. Among them, both the calculation and screening processes can be completed by a Gaussian filter. The joint mask is the standard code for screening core pixel data and can prevent the video data stream from being tampered with and intercepted during transmission.

[0064] In addition, in specific implementation, the data stream during the transmission from the multimedia server to the client is filtered and screened, and the combined mask is used as the filtering condition. The data identified as 1 by the filtering condition is retained through a Gaussian filter, and all the filtered and screened data is used as the core pixel data. Finally, the core pixel data is encoded through the H.265 compression coding algorithm to obtain the core pixels transmitted from the multimedia server side.

[0065] It should be noted that in this embodiment, the combined mask can ensure the accuracy of core pixel extraction. At the same time, filtering reduces the bandwidth occupation, improves the transmission efficiency, and ensures the anti-tampering and anti-error capabilities of data transmission, which is beneficial to the reliability of video data stream transmission.

[0066] In this embodiment, based on the core pixels, image reconstruction is performed on the image frames in the original video stream to obtain the core reconstructed image. Specifically, the following method can be adopted, that is:

[0067] Construct a blank image matrix;

[0068] Determine the compensation domain of the image frame in the original video stream;

[0069] Using this blank image matrix, based on the core pixels, compensation reconstruction is performed on the compensation domain to obtain the core reconstructed image.

[0070] In specific implementation, first, obtain the resolution of the image frame of the original video stream, and initialize a blank image matrix based on this resolution. Among them, the blank image matrix is composed of blank pixels with the same resolution as the original video frame. Then, the compensation domain is the pixel area that needs to be repaired or filled in the image frame. The compensation domain of the image frame in the original video stream can be located by using the interpolation method, which can effectively avoid image distortion caused by missing areas. Finally, fill the values of the core pixels directly into the blank matrix, and use bilinear interpolation to compensate the compensation domain. Among them, the process of compensating and reconstructing the compensation domain can be expressed by the following expression, that is:

[0071]

[0072] Among them, is the pixel after compensation reconstruction, is the non-empty set of core pixels, which can be determined by screening out the pixels with a value of 0. i represents the i-th pixel, is the weight factor of the i-th pixel, which is used to measure the influence of the core pixel on the compensation position and can be determined by the ratio of the pixel value of the i-th pixel to the pixel mean value in the compensation domain, is the i-th core pixel, is the compensation area; it should be noted that by compensating and reconstructing the compensation area, the obtained core reconstructed image has higher clarity, which is beneficial to the filtering and repair of video data.

[0073] In step S3, morphological filtering is performed on the core reconstructed image according to the pixel information of the core reconstructed image to obtain the reconstruction core points and boundary tampering points during image reconstruction.

[0074] Preferably, in this embodiment, refer to Figure 3 As shown, this figure is an exemplary flowchart for determining the reconstruction core points and boundary tampering points during image reconstruction in an embodiment of the present application. In this embodiment, morphological filtering is performed on the core reconstructed image according to the pixel information of the core reconstructed image to obtain the reconstruction core points and boundary tampering points during image reconstruction, which can be implemented by the following steps:

[0075] First, in step S31, dilation and erosion filtering operations are respectively performed on the core reconstructed image to obtain the filtered overlapping area of the core reconstructed image;

[0076] Then, in step S32, the reconstruction core points during image reconstruction are screened from the filtered overlapping area;

[0077] Finally, in step S33, based on the filtered overlapping area, the boundary tampering points during image reconstruction are marked from the core reconstructed image through an edge detection algorithm.

[0078] Specifically, when implementing, first, expand the highlighted area in the core reconstructed image (the area where the pixel value in the core reconstructed image is greater than the average pixel value). By filling and connecting adjacent pixel areas, contract the highlighted area in the core reconstructed image to remove isolated noise points or thin the edges. Dilation and erosion operations can be performed using a square element. Among them, the dilation operation makes the potential boundary of the core reconstructed image more prominent, which is helpful for boundary detection. The erosion operation removes background noise and can retain the significant information of the core reconstructed image to provide support for the extraction of reconstruction core points and tampering points. Furthermore, the intersection area of the results of the dilation and erosion operations is the filtered overlapping area. Then, traverse the pixels in the filtered overlapping area and select the points with significant pixel values or significant gradient changes as the reconstruction core points, which can be determined through statistical analysis. Among them, the reconstruction core points are the key areas of the core reconstructed image and can provide an effective reference for boundary detection. Finally, use an edge detection algorithm (such as: Canny algorithm, Sobel algorithm or other edge detection algorithms) to extract the edges of the filtered overlapping area, and combine Gaussian blur on the basis of edge detection to screen the boundary tampering points, and mark the screened boundary tampering points through this edge detection algorithm to facilitate tracking and repairing the image, thereby optimizing the video stream received by the multimedia client.

[0079] It should be noted that the core point of reconstruction in this application is the pixel area with high credibility in the core reconstructed image, which can provide a stable reference benchmark for subsequent image restoration. The boundary tampering points are the noise boundaries after morphological filtering or the pixels with inconsistent image features. Marking the boundary tampering points is beneficial to identifying the tampered boundary positions, isolating the tampered content, and thus beneficial to improving the quality of the video stream received by the multimedia client.

[0080] In step S4, obtain the spatio-temporal information of the original video stream, and perform spatio-temporal correlation aggregation on the boundary tampering points through the time features and space features in the spatio-temporal information to obtain a correlation aggregation image.

[0081] It should be noted that in this application, the spatio-temporal information of the original video stream is obtained through a spatio-temporal segmentation algorithm. The original video stream is analyzed using the spatio-temporal segmentation algorithm to extract the time features and space features of the original video stream. Then, the combination of the time features and space features is used as the spatio-temporal information. Among them, the time features are used to describe the trend of inter-frame changes in the video stream, such as: pixel change frequency and inter-frame difference value. The space features are used to describe the features of the local texture, edge distribution, and regional shape of a single-frame image. In addition, in this embodiment, the spatio-temporal information can provide context information of the boundary tampering points, facilitating the association of the features of adjacent pixels, and thus beneficial to reducing the noise of the image frames.

[0082] In this embodiment, performing spatio-temporal correlation aggregation on the boundary tampering points through the time features and space features in the spatio-temporal information to obtain a correlation aggregation image can be specifically implemented in the following manner, that is:

[0083] Perform continuity analysis on the boundary tampering points on the time axis through the time features in the spatio-temporal information to obtain continuous boundary tampering points;

[0084] Perform local area aggregation in space on the boundary tampering points through the space features in the spatio-temporal information to obtain aggregated boundary tampering points;

[0085] Perform correlation aggregation on the continuous boundary tampering points and the aggregated boundary tampering points to obtain a correlation aggregation image.

[0086] In specific implementation, first, the time-weighted sliding window algorithm is used to eliminate noise points within a time window, and continuous boundary tampering points are obtained by sliding the time window; then, the region growing algorithm is adopted to extract the spatial distribution characteristics of the boundary tampering points, and local region aggregation is performed on the boundary tampering points based on the region growing algorithm and the spatial distribution characteristics to eliminate isolated points in the boundary tampering points, thereby obtaining aggregated boundary tampering points; finally, a spatio-temporal correlation aggregation model is initialized based on the dynamic programming algorithm, the continuous boundary tampering points are used as the input in the time direction of the spatio-temporal correlation aggregation model, and the aggregated boundary tampering points are used as the input in the spatial direction of the spatio-temporal correlation aggregation model, and a correlated and aggregated image is output through the spatio-temporal correlation aggregation model.

[0087] It should be noted that the spatio-temporal correlation aggregation model in this embodiment is a mathematical model based on an optimization processing mechanism in both time and space dimensions, used to process and analyze information in spatio-temporal data, so as to effectively identify and aggregate relevant data points in both time and space dimensions. The spatio-temporal correlation aggregation model extracts spatio-temporal correlation information from data such as boundary tampering points, and through joint analysis and optimization of data in time and space, a more accurate and continuous image is obtained, which can improve the integrity and repair effect of the data to enhance the smoothness of the video stream.

[0088] In step S5, an image repair operator is determined according to the reconstructed core points, and the associated aggregated image is repaired and transmitted based on the image repair operator, thereby obtaining the video stream received by the multimedia client.

[0089] In specific implementation, the reconstructed core points are the most critical pixel points in the core reconstructed image extracted by morphological filtering. The reconstructed core points include features such as the texture, edges, and illumination of the core reconstructed image, and these features will affect the application of the repair operator. Therefore, the reconstructed core points can be used as initialization parameters for regression analysis of the convolutional neural network and the edge detection operator, and the initialized convolutional neural network is iteratively trained through the edge repair algorithm. When the distortion rate of the convolutional neural network is the smallest, the edge detection operator in the convolutional neural network at this time is used as the image repair operator.

[0090] It should be noted that the regression analysis of initializing the convolutional neural network and the edge detection operator through the reconstructed core points and combining with the edge repair algorithm can effectively train the edge repair operator, which is beneficial to the precise repair of image frames during the transmission process from the multimedia server to the multimedia client.

[0091] In this embodiment, the associated aggregated image is repaired and transmitted based on the image repair operator, and the video stream received by the multimedia client can be specifically obtained in the following manner, that is:

[0092] Establish an image frame restoration model in the channel from the multimedia server to the multimedia client;

[0093] The image frame restoration model restores the associated aggregated image based on the image restoration operator, and outputs the restored image data through the image frame restoration model, thereby obtaining the video stream received by the multimedia client.

[0094] In specific implementation, a server can be deployed between the channels from the multimedia server to the multimedia client, and a generative adversarial network is used to learn the features of image restoration, and the image frame restoration model is deployed on this server; then, when the associated aggregated image is transmitted to the image frame restoration model, the image frame restoration model restores the missing or tampered image data based on the image restoration operator, and outputs the restored image data through the image frame restoration model, and then encodes the continuously output image data through the H.265 compression encoding algorithm to obtain the video stream received by the multimedia client.

[0095] It should be noted that in this application, by establishing an image frame restoration model in the channel from the multimedia server to the multimedia client and using this model to restore the associated aggregated image, the quality of the video stream can be effectively improved. Through the restoration operator, it is convenient to accurately restore the tampered or lost data during the transmission of the video stream, ensuring the integrity and continuity of the video quality.

[0096] It can be seen that in this application, video data anti-tampering filtering and restoration can be realized during efficient video transmission. First, by obtaining the original video stream on the server side, it provides basic data for subsequent operations, and performs frame-by-frame decoding and differential analysis to extract the core data stream in the video, so as to reduce the transmission of redundant information and reduce the bandwidth pressure; secondly, determine the core pixels to be transmitted through the core data stream, and then perform image reconstruction to obtain the core reconstructed image, which can supplement the missing or tampered parts, and by extracting and utilizing the core pixels, solve the lost or incorrect data during the transmission, thus avoiding the decline in the quality of the transmitted image frames; then, through dilation and erosion morphological filtering operations on the core reconstructed image, further extract the tampered areas and core pixels from the core reconstructed image, providing accurate points to be restored and key data for image restoration and integrity guarantee, which is beneficial to the high-quality output of the image, and comprehensively considering time and space factors, associating the boundary tampering points in space-time can effectively repair the errors caused by packet loss or tampering during network transmission, ensuring the integrity of the restored image; finally, based on the restoration operator, the associated aggregated image is restored and transmitted, which can ensure that the quality of the video stream finally received by the client is not affected.

[0097] In summary, the technical solution adopted in this application can implement anti-tampering filtering and repair of video data during high-efficiency video transmission to improve the quality of the video stream received by the multimedia client.

[0098] Embodiment 2. This application provides a multimedia data transmission system. Referring to Figure 4 as shown, this figure is a schematic diagram of the transmission system according to this embodiment of this application. The transmission system includes:

[0099] An acquisition module 100, configured to acquire the original video stream collected by the multimedia server side;

[0100] A pixel reconstruction module 200, configured to perform inter-frame difference decoding verification on the original video stream to obtain a core data stream, determine the core pixels transmitted by the multimedia server side through the core data stream, and then perform image reconstruction on the image frames in the original video stream based on the core pixels to obtain a core reconstructed image;

[0101] An image filtering module 300, configured to perform morphological filtering on the core reconstructed image according to the pixel information of the core reconstructed image to obtain the reconstructed core points and boundary tampering points during image reconstruction;

[0102] An association aggregation module 400, configured to obtain the spatio-temporal information of the original video stream, perform spatio-temporal association aggregation on the boundary tampering points through the time feature and space feature in the spatio-temporal information to obtain an associated aggregated image;

[0103] A repair transmission module 500, configured to determine an image repair operator according to the reconstructed core points, perform repair transmission on the associated aggregated image based on the image repair operator, and then obtain the video stream received by the multimedia client.

[0104] This application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of this application. It should be understood that each process and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the processes and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0105] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. The storage medium includes read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc memories, magnetic disk memories, tape memories, or any other medium that can be used to carry or store data and is computer-readable.

[0106] It should also be noted that the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent in such a process, method, commodity or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, commodity or device including the element.

Claims

1. A multimedia data transmission method, characterized in that, The transmission method includes: Obtain the original video stream collected by the multimedia server side; Perform inter-frame difference decoding verification on the original video stream to obtain the core data stream, determine the core pixels transmitted by the multimedia server side through the core data stream, and then reconstruct the image frames in the original video stream based on the core pixels to obtain the core reconstructed image; Perform morphological filtering on the core reconstructed image according to the pixel information of the core reconstructed image to obtain the reconstruction core points and boundary tampering points during image reconstruction; Obtain the spatio-temporal information of the original video stream, and perform spatio-temporal correlation aggregation on the boundary tampering points through the time feature and space feature in the spatio-temporal information to obtain the associated aggregated image; Determine the image repair operator according to the reconstruction core points, perform repair transmission on the associated aggregated image based on the image repair operator, and then obtain the video stream received by the multimedia client; Among them, performing inter-frame difference decoding verification on the original video stream to obtain the core data stream specifically includes: Perform frame-by-frame decoding on the original video stream to obtain a continuous sequence of image frames; Determine the pixel threshold through the sequence of image frames; Screen the core data of each image frame in the sequence of image frames according to the pixel threshold to obtain the core data stream; Among them, performing morphological filtering on the core reconstructed image according to the pixel information of the core reconstructed image to obtain the reconstruction core points and boundary tampering points during image reconstruction specifically includes: Perform dilation and erosion filtering operations on the core reconstructed image respectively to obtain the filtered overlapping area of the core reconstructed image; Screen the reconstruction core points during image reconstruction from the filtered overlapping area; Mark the boundary tampering points during image reconstruction from the core reconstructed image based on the filtered overlapping area through an edge detection algorithm; Among them, use the reconstruction core points as initialization parameters to perform regression analysis on the convolutional neural network and the edge detection operator, and perform iterative training on the initialized convolutional neural network through an edge repair algorithm. When the distortion rate of the convolutional neural network is the smallest, use the edge detection operator in the convolutional neural network at this time as the image repair operator; Among them, performing repair transmission on the associated aggregated image based on the image repair operator to obtain the video stream received by the multimedia client specifically includes: Establish an image frame repair model in the channel from the multimedia server to the multimedia client; The image frame repair model repairs the associated aggregated image based on the image repair operator, outputs the repaired image data through the image frame repair model, and then obtains the video stream received by the multimedia client.

2. The multimedia data transmission method according to claim 1, characterized in that, Obtain the original video stream collected by the multimedia server side by connecting to the secure access machine.

3. The multimedia data transmission method according to claim 1, characterized in that, Determining the core pixels transmitted by the multimedia server side through the core data stream specifically includes: Perform spatio-temporal collaborative filtering on the core data stream to obtain a joint mask; Screen the core pixel data according to the joint mask; Encode the core pixel data to obtain the core pixels transmitted by the multimedia server side.

4. The multimedia data transmission method according to claim 1, characterized in that Performing image reconstruction on the image frames in the original video stream based on the core pixels to obtain the core reconstructed image specifically includes: Constructing a blank image matrix; Determining the compensation domain of the image frames in the original video stream; Using the blank image matrix to perform compensation reconstruction on the compensation domain based on the core pixels to obtain the core reconstructed image.

5. The multimedia data transmission method according to claim 1, characterized in that, Obtaining the spatio-temporal information of the original video stream through a spatio-temporal segmentation algorithm.

6. The multimedia data transmission method according to claim 1, wherein The multimedia server is a hybrid communication server.

7. A multimedia data transmission method according to claim 1, characterized in that, The multimedia client is a software client.

8. A multimedia data transmission system for performing a multimedia data transmission method according to any one of claims 1 to 7, characterized in that, The transmission system includes: An acquisition module for acquiring the original video stream collected by the multimedia server side; A pixel reconstruction module for performing inter-frame differential decoding verification on the original video stream to obtain the core data stream, determining the core pixels transmitted by the multimedia server side through the core data stream, and then performing image reconstruction on the image frames in the original video stream based on the core pixels to obtain the core reconstructed image; An image filtering module for performing morphological filtering on the core reconstructed image according to the pixel information of the core reconstructed image to obtain the reconstructed core points and boundary tampering points during image reconstruction; An association aggregation module for obtaining the spatio-temporal information of the original video stream and performing spatio-temporal association aggregation on the boundary tampering points through the time feature and space feature in the spatio-temporal information to obtain an associated aggregated image; A repair transmission module for determining an image repair operator according to the reconstructed core points, performing repair transmission on the associated aggregated image based on the image repair operator, and then obtaining the video stream received by the multimedia client.

Images