Multimedia data transmission method and system

Abstract

The invention provides a multimedia data transmission method and system, and relates to the technical field of video data transmission, and the method comprises the steps: carrying out the inter-frame difference decoding verification of an original video stream collected by a multimedia server side, obtaining a core data stream, determining a core pixel transmitted by the multimedia server side through the core data stream, and transmitting the core pixel to the multimedia server side; performing image reconstruction on an image frame in the original video stream based on the core pixel to obtain a core reconstructed image; performing morphological filtering on the core reconstructed image according to pixel information of the core reconstructed image to obtain reconstructed core points and boundary tampering points; performing space-time association aggregation on the boundary tampering points through the time features and the space features to obtain an association aggregation image; and finally, repairing and transmitting the associated aggregated image through an image repairing operator determined by reconstructing the core point to obtain a video stream received by the multimedia client, so that tamper-proof filtering and repairing of the video data can be realized in an efficient video transmission process, and the quality of the video stream received by the multimedia client is improved.

Description

Technical Field

[0001] The present application relates to the technical field of video data transmission, and more specifically, to a multimedia data transmission method and system thereof. Background Art

[0002] As an important part of modern information technology, video data transmission is widely used in scenarios such as video conferencing, real-time monitoring, and live broadcast services. Among them, the transmission speed of video data and the quality of image frames are the key to user experience.

[0003] With the rapid development of video technology, multimedia data has been widely used in various application scenarios, such as high-definition video live broadcast, video surveillance, remote conferencing and virtual reality, which has put higher quality requirements on the real-time, reliability and quality of video data transmission. However, video data usually contains a lot of redundant information, which will occupy a lot of network bandwidth during the acquisition and transmission process of the multimedia server, resulting in delays, packet loss, jitter and other problems in the transmission network environment of multimedia video data. The existing methods do not fully extract the key information in the video data before video data transmission, and will be subject to data tampering or attacks during the transmission process, resulting in difficulty in repairing the image frame through inter-frame difference decoding, which reduces the quality of the video stream received by the client and cannot guarantee the integrity of the video stream received by the client. Therefore, how to realize anti-tampering filtering and repair of video data in the process of efficient video transmission to improve the quality of the video stream received by the multimedia client has become a difficult problem faced by the industry. Summary of the invention

[0004] The present application provides a multimedia data transmission method and system thereof, which can realize anti-tampering filtering and repair of video data during efficient video transmission to improve the quality of video streams received by multimedia clients.

[0005] In a first aspect, the present application provides a multimedia data transmission method, the transmission method comprising the following steps: Get the original video stream collected by the multimedia server; Performing inter-frame difference decoding verification on the original video stream to obtain a core data stream, determining core pixels transmitted by the multimedia server end through the core data stream, and then reconstructing image frames in the original video stream based on the core pixels to obtain a core reconstructed image; Performing morphological filtering on the core reconstructed image according to pixel information of the core reconstructed image to obtain reconstructed core points and boundary tampering points during image reconstruction; Acquire the spatiotemporal information of the original video stream, and perform spatiotemporal correlation aggregation on the boundary tampering points through the time features and spatial features in the spatiotemporal information to obtain a correlation aggregated image; An image restoration operator is determined according to the reconstructed core point, and the associated aggregated image is restored and transmitted based on the image restoration operator, thereby obtaining a video stream received by a multimedia client.

[0006] In this embodiment, the original video stream collected by the multimedia server is obtained by connecting to the security access machine.

[0007] In this embodiment, inter-frame difference decoding verification is performed on the original video stream to obtain a core data stream, which specifically includes: Decoding the original video stream frame by frame to obtain a continuous image frame sequence; Determining a pixel threshold through the image frame sequence; The core data of each image frame in the image frame sequence is filtered according to the pixel threshold to obtain a core data stream.

[0008] In this embodiment, determining the core pixel transmitted by the multimedia server through the core data stream specifically includes: Performing spatiotemporal collaborative filtering on the core data stream to obtain a joint mask; filtering core pixel data according to the joint mask; The core pixel data is encoded to obtain core pixels transmitted by the multimedia server.

[0009] In this embodiment, reconstructing the image frames in the original video stream based on the core pixels to obtain the core reconstructed image specifically includes: Construct a blank image matrix; Determining a compensation domain of an image frame in the original video stream; The blank image matrix is ​​used to perform compensation reconstruction on the compensation domain based on the core pixels to obtain a core reconstructed image.

[0010] In this embodiment, morphological filtering is performed on the core reconstructed image according to the pixel information of the core reconstructed image to obtain the reconstructed core points and boundary tampering points during image reconstruction, specifically including: Performing dilation and erosion filtering operations on the core reconstructed image respectively to obtain a filtering overlap area of ​​the core reconstructed image; Selecting the reconstruction core points during image reconstruction from the filtering overlap area; The boundary tampering points during image reconstruction are marked from the core reconstructed image based on the filtered overlap area by an edge detection algorithm.

[0011] In this embodiment, the spatiotemporal information of the original video stream is obtained by using a spatiotemporal segmentation algorithm.

[0012] In this embodiment, the multimedia server is a hybrid communication server.

[0013] In this embodiment, the multimedia client is a software client.

[0014] In a second aspect, the present application provides a multimedia data transmission system for executing a multimedia data transmission method, the transmission system comprising: The acquisition module is used to acquire the original video stream collected by the multimedia server; A pixel reconstruction module is used to perform inter-frame difference decoding verification on the original video stream to obtain a core data stream, determine the core pixels transmitted by the multimedia server through the core data stream, and then reconstruct the image frames in the original video stream based on the core pixels to obtain a core reconstructed image; An image filtering module, used for performing morphological filtering on the core reconstructed image according to pixel information of the core reconstructed image, to obtain reconstructed core points and boundary tampering points during image reconstruction; An association aggregation module, used to obtain the spatiotemporal information of the original video stream, and perform spatiotemporal association aggregation on the boundary tampering points through the time features and spatial features in the spatiotemporal information to obtain an association aggregation image; The repair and transmission module is used to determine an image repair operator according to the reconstructed core point, and repair and transmit the associated aggregated image based on the image repair operator to obtain a video stream received by a multimedia client.

[0015] The technical solution provided by the embodiments disclosed in this application has the following beneficial effects: First, the original video stream collected by the multimedia server is obtained; secondly, the original video stream is decoded and checked for inter-frame differences to obtain a core data stream, and the core pixels transmitted by the multimedia server are determined through the core data stream, and then the image frames in the original video stream are reconstructed based on the core pixels to obtain a core reconstructed image; then, the core reconstructed image is morphologically filtered according to the pixel information of the core reconstructed image to obtain the reconstructed core points and boundary tampering points during image reconstruction; then, the spatiotemporal information of the original video stream is obtained, and the boundary tampering points are spatiotemporally correlated and aggregated through the time features and spatial features in the spatiotemporal information to obtain a correlated aggregated image; finally, an image repair operator is determined according to the reconstructed core points, and the correlated aggregated image is repaired and transmitted based on the image repair operator to obtain the video stream received by the multimedia client.

[0016] It can be seen that in the present application, anti-tampering filtering and repair of video data can be achieved during efficient video transmission. First, by obtaining the original video stream on the server side, basic data is provided for subsequent operations, and frame-by-frame decoding and difference analysis are performed to extract the core data stream in the video to reduce the transmission of redundant information and reduce bandwidth pressure; secondly, the core pixels of the transmission are determined through the core data stream, and then the image is reconstructed to obtain a core reconstructed image, which can supplement the lost or tampered parts, and by extracting and utilizing the core pixels, the lost or erroneous data during the transmission process is solved, thereby avoiding the degradation of the quality of the transmitted image frame; then, by performing expansion and corrosion morphological filtering operations on the core reconstructed image, the tampered area and core pixels are further extracted from the core reconstructed image, providing accurate points to be repaired and key data for image repair and integrity assurance, which is conducive to high-quality output of the image, and comprehensively considering time and space factors, the boundary tampering points are temporally and spatially associated, which can effectively repair the errors caused by packet loss or tampering during network transmission, and ensure the integrity of the repaired image; finally, the associated aggregated image is repaired and transmitted based on the repair operator, which can ensure that the quality of the video stream finally received by the client is not affected.

[0017] In summary, the technical solution adopted in this application can realize tamper-proof filtering and repair of video data during efficient video transmission, so as to improve the quality of video streams received by multimedia clients. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 is a flowchart of a multimedia data transmission method provided by the present application; Figure 2 is an exemplary flow chart for determining a core data flow according to the present application; Figure 3 is an exemplary flow chart for determining reconstruction core points and boundary tampering points when reconstructing an image according to the present application; Figure 4 It is a module structure diagram of a multimedia data transmission system provided according to the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] The embodiment of the present application provides a multimedia data transmission method and system, the core of which is to obtain the original video stream collected by the multimedia server; then perform inter-frame difference decoding and verification on the original video stream to obtain a core data stream, determine the core pixels transmitted by the multimedia server through the core data stream, and then reconstruct the image frames in the original video stream based on the core pixels to obtain a core reconstructed image; then, perform morphological filtering on the core reconstructed image according to the pixel information of the core reconstructed image to obtain reconstructed core points and boundary tampering points during image reconstruction; then, obtain the spatiotemporal information of the original video stream, perform spatiotemporal correlation aggregation on the boundary tampering points through the time features and spatial features in the spatiotemporal information to obtain an associated aggregated image; finally, determine an image repair operator based on the reconstructed core points, and repair and transmit the associated aggregated image based on the image repair operator to obtain a video stream received by the multimedia client.

[0022] Embodiment 1: In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1 As shown in FIG. 1 , this figure is an exemplary flow chart of a multimedia data transmission method according to this embodiment of the present application, and the transmission method includes the following steps: In step S1, the original video stream collected by the multimedia server is obtained.

[0023] In specific implementation, the original video stream collected by the multimedia server is obtained by connecting to the security access machine. A physical or logical connection can be established between the security access machine and the multimedia server to ensure the integrity and confidentiality of the data during transmission. In actual implementation, a network tunnel (such as SSL VPN) can be configured through the security access machine and a security protocol can be used to complete two-way authentication between the client and the security access machine, thereby decrypting the video stream. After verifying the data integrity, the decrypted video stream can be used as the original video stream collected by the multimedia server.

[0024] It should be noted that the present application obtains the original video stream through a secure access machine, which can effectively enhance the security and stability of data transmission, and is conducive to ensuring the confidentiality of video data, so as to preliminarily prevent illegal access by unauthorized devices or users, and prevent data from being intercepted or tampered with, thereby helping to improve the confidentiality and tamper-resistance of the original video stream during transmission from the multimedia server to the multimedia client.

[0025] In step S2, the original video stream is subjected to inter-frame difference decoding verification to obtain a core data stream, the core pixels transmitted by the multimedia server are determined through the core data stream, and then the image frames in the original video stream are reconstructed based on the core pixels to obtain a core reconstructed image.

[0026] Preferably, in this embodiment, reference Figure 2 As shown, this figure is an exemplary flow chart of determining the core data stream in an embodiment of the present application. In this embodiment, the original video stream is decoded and checked for inter-frame differences, and the core data stream can be obtained by the following steps: First, in step S21, the original video stream is decoded frame by frame to obtain a continuous image frame sequence; Then, in step S22, a pixel threshold is determined through the image frame sequence; Finally, in step S23, the core data of each image frame in the image frame sequence is filtered according to the pixel threshold, thereby obtaining a core data stream.

[0027] In specific implementation, first, the original video stream can be decoded into multiple independent image frames by a standard video decoding algorithm, and a sequence obtained by sorting all the image frames according to the generation time is used as an image frame sequence. As a preferred embodiment, the standard video decoding algorithm can use H.265 to facilitate the transmission of higher quality video data under limited bandwidth. In other embodiments, other standard video decoding algorithms can also be used, which are not limited here; then, for each image frame in the image frame sequence, the value obtained by weighted summing the average value and standard deviation of all pixels in the image frame is used as the weighted pixel value of the image frame, thereby obtaining the weighted pixel value of each image frame in the image frame sequence, and then the mean of all weighted pixel values ​​is used as the pixel threshold, wherein the sum weight of the average value and standard deviation of all pixels in each image frame can be determined by regression analysis; finally, for each image frame in the image frame sequence, the pixels in the image frame that are greater than the pixel threshold are used as the core data of the image frame, thereby obtaining the core data of each image frame in the image frame sequence, and all the core data are compressed and encoded by a Huffman encoder to obtain a core data stream.

[0028] It should be noted that, by extracting the core data stream, the present application can retain the core video data, reduce the bandwidth requirement for video data stream transmission, improve the transmission speed, and facilitate effective filtering and identification of malicious tampering during transmission, thereby facilitating the improvement of the transmission efficiency from the multimedia server to the multimedia client.

[0029] In this embodiment, the core pixel transmitted by the multimedia server can be determined by the core data stream in the following manner, namely: Performing spatiotemporal collaborative filtering on the core data stream to obtain a joint mask; filtering core pixel data according to the joint mask; The core pixel data is encoded to obtain core pixels transmitted by the multimedia server.

[0030] In a specific implementation, the pixel differences and fluctuation amplitudes of inter-frame changes are calculated, the areas without significant changes in the core data stream are calculated, and the important areas of the core data stream are screened according to the neighborhood similarity, and then the filtered core data is encoded as 1, and the filtered core data is encoded as 0, to obtain a joint mask, wherein both the calculation and screening processes can be completed by a Gaussian filter, and the joint mask is a standard code for screening core pixel data, which can prevent the video data stream from being tampered with and intercepted during transmission.

[0031] In addition, in the specific implementation, the data stream in the transmission process from the multimedia server to the client is filtered and screened, and the joint mask is used as the filtering condition. The data identified as 1 by the filtering condition is retained through a Gaussian filter, and the data obtained by filtering and screening are all used as core pixel data; finally, the core pixel data is encoded through the H.265 compression coding algorithm to obtain the core pixels transmitted by the multimedia server.

[0032] It should be noted that, in this embodiment, the accuracy of core pixel extraction can be ensured by joint masking. At the same time, filtering can reduce bandwidth occupancy, improve transmission efficiency, and ensure the anti-tampering and anti-error capabilities of data transmission, which is beneficial to the reliability of video data stream transmission.

[0033] In this embodiment, the image frames in the original video stream are reconstructed based on the core pixels to obtain the core reconstructed image in the following manner, namely: Construct a blank image matrix; Determining a compensation domain of an image frame in the original video stream; The blank image matrix is ​​used to perform compensation reconstruction on the compensation domain based on the core pixels to obtain a core reconstructed image.

[0034] In the specific implementation, first, the resolution of the image frame of the original video stream is obtained, and a blank image matrix is ​​initialized based on the resolution, wherein the blank image matrix is ​​composed of blank pixels with the same resolution as the original video frame; then, the compensation domain is the pixel area in the image frame that needs to be repaired or filled, and the compensation domain of the image frame in the original video stream can be located by interpolation method, which can effectively avoid image distortion caused by missing areas; finally, the value of the core pixel is directly filled into the blank matrix, and the compensation domain is compensated by bilinear interpolation, wherein the process of compensating and reconstructing the compensation domain can be performed by the following expression, namely: in, To compensate the reconstructed pixels, is a non-empty set of core pixels, which can be determined by filtering out pixels that are 0, i represents the i-th pixel, is the weight factor of the ith pixel, which is used to measure the influence of the core pixel on the compensation position and can be determined by the ratio of the pixel value of the ith pixel to the pixel mean in the compensation domain. is the ith core pixel, is the compensation domain; it should be noted that, by performing compensation reconstruction on the compensation domain, the obtained core reconstructed image has higher clarity, which is conducive to filtering and repairing video data.

[0035] In step S3, morphological filtering is performed on the core reconstructed image according to the pixel information of the core reconstructed image to obtain the reconstructed core points and boundary tampering points during image reconstruction.

[0036] Preferably, in this embodiment, reference Figure 3 As shown in the figure, this figure is an exemplary flow chart of determining the reconstruction core points and boundary tampering points during image reconstruction in an embodiment of the present application. In this embodiment, the core reconstructed image is morphologically filtered according to the pixel information of the core reconstructed image, and the reconstruction core points and boundary tampering points during image reconstruction can be obtained by the following steps: First, in step S31, dilation and erosion filtering operations are respectively performed on the core reconstructed image to obtain a filtering overlap area of ​​the core reconstructed image; Then, in step S32, the reconstruction core points during image reconstruction are screened from the filtering overlap area; Finally, in step S33, the boundary tampering points during image reconstruction are marked from the core reconstructed image based on the filtered overlap area by using an edge detection algorithm.

[0037] In the specific implementation, first, the highlight area in the core reconstructed image (the area in the core reconstructed image where the pixel value is greater than the average pixel value) is expanded, and the highlight area in the core reconstructed image is shrunk by filling and connecting the adjacent pixel areas to remove isolated noise points or thin edges. The dilation and erosion operations can be performed by using square elements, wherein the dilation operation makes the potential boundary of the core reconstructed image more prominent, which is helpful for boundary detection, and the erosion operation removes background noise, which can retain the significant information of the core reconstructed image to provide support for the extraction of the reconstructed core points and tampering points, and then the intersection area of ​​the dilation and erosion operation results is the filtered overlap area; then Then, traverse the pixels in the filtered overlap area, and select points with significant pixel values ​​or significant gradient changes as reconstruction core points, which can be determined through statistical analysis, wherein the reconstruction core points are the key areas of the core reconstructed image, which can provide an effective reference for boundary detection; finally, use an edge detection algorithm (for example: Canny algorithm, Sobel algorithm or other edge detection algorithms) to extract the edges of the filtered overlap area, and combine Gaussian blur with edge detection to filter boundary tampering points, and use the edge detection algorithm to mark the filtered boundary tampering points, which is convenient for tracking and repairing images, thereby optimizing the video stream received by the multimedia client.

[0038] It should be noted that the reconstructed core point in the present application is a pixel area with high credibility in the core reconstructed image, which can provide a stable reference benchmark for subsequent image restoration. The boundary tampering point is the noise boundary after morphological filtering or the pixel with inconsistent image features. Marking the boundary tampering point is conducive to identifying the tampered boundary position and isolating the tampered content, which in turn helps to improve the quality of the video stream received by the multimedia client.

[0039] In step S4, the spatiotemporal information of the original video stream is obtained, and the boundary tampering points are spatiotemporally correlated and aggregated using the time features and spatial features in the spatiotemporal information to obtain a correlated aggregated image.

[0040] It should be noted that, in the present application, the spatiotemporal information of the original video stream is obtained by a spatiotemporal segmentation algorithm, the original video stream is analyzed using the spatiotemporal segmentation algorithm, the temporal features and spatial features of the original video stream are extracted, and then the combination of the temporal features and the spatial features is used as the spatiotemporal information, wherein the temporal features are used to describe the trend of inter-frame changes in the video stream, such as: pixel change frequency and inter-frame difference value, and the spatial features are used to describe the local texture, edge distribution, and regional shape characteristics of a single-frame image. In addition, in the present embodiment, the spatiotemporal information can provide contextual information of the boundary tampering point, which is convenient for associating the features of adjacent pixels, thereby helping to reduce the noise of the image frame.

[0041] In this embodiment, the boundary tampering points are temporally and spatially correlated and aggregated by using the time features and spatial features in the temporal and spatial information to obtain the correlated and aggregated image, which can be specifically obtained in the following manner, namely: Performing continuity analysis on the time axis of the boundary tampering points through the time features in the spatiotemporal information to obtain continuous boundary tampering points; Aggregating the boundary tampering points in local areas in space by using the spatial features in the spatiotemporal information to obtain aggregated boundary tampering points; The continuous boundary tampering points and the aggregated boundary tampering points are associated and aggregated to obtain an associated aggregated image.

[0042] In the specific implementation, first, a time-weighted sliding window algorithm is used to eliminate noise points within a time window, and continuous boundary tampering points are obtained by sliding the time window; then, a region growing algorithm is used to extract the spatial distribution characteristics of the boundary tampering points, and based on the region growing algorithm and the spatial distribution characteristics, the boundary tampering points are locally aggregated to eliminate isolated points in the boundary tampering points, thereby obtaining aggregated boundary tampering points; finally, a spatiotemporal association aggregation model is initialized based on a dynamic programming algorithm, and the continuous boundary tampering points are used as the input of the spatiotemporal association aggregation model in the time direction, and the aggregated boundary tampering points are used as the input of the spatiotemporal association aggregation model in the spatial direction, and the associated aggregation image is output through the spatiotemporal association aggregation model.

[0043] It should be noted that the spatiotemporal correlation aggregation model in this embodiment is a mathematical model based on an optimization processing mechanism in the dual dimensions of time and space. It is used to process and analyze information in spatiotemporal data so as to effectively identify and aggregate relevant data points in both time and space dimensions. The spatiotemporal correlation aggregation model extracts spatiotemporal correlation information from data such as boundary tampering points, and obtains more accurate and continuous images by jointly analyzing and optimizing data in time and space, thereby improving data integrity and repair effects, thereby improving the smoothness of the video stream.

[0044] In step S5, an image restoration operator is determined according to the reconstructed core point, and the associated aggregated image is restored and transmitted based on the image restoration operator, thereby obtaining a video stream received by the multimedia client.

[0045] In specific implementation, the reconstructed core point is the most critical pixel point in the core reconstructed image extracted by morphological filtering. The reconstructed core point includes the texture, edge, lighting and other features of the core reconstructed image. These features will affect the application of the repair operator. Therefore, the reconstructed core point can be used as the initialization parameter to perform regression analysis on the convolutional neural network and the edge detection operator, and the initialized convolutional neural network is iteratively trained through the edge repair algorithm. When the distortion rate of the convolutional neural network is the smallest, the edge detection operator in the convolutional neural network at this time is used as the image repair operator.

[0046] It should be noted that by reconstructing the core points to initialize the convolutional neural network and the regression analysis of the edge detection operator and combining it with the edge repair algorithm, the edge repair operator can be effectively trained, which is beneficial to the accurate repair of image frames during the transmission process from the multimedia server to the multimedia client.

[0047] In this embodiment, the associated aggregated image is repaired and transmitted based on the image repair operator, and the video stream received by the multimedia client is obtained in the following manner, namely: Establishing an image frame restoration model in the channel from the multimedia server to the multimedia client; The image frame restoration model restores the associated aggregated image based on the image restoration operator, outputs restored image data through the image frame restoration model, and thereby obtains a video stream received by a multimedia client.

[0048] In specific implementation, a server can be deployed between the channels from the multimedia server to the multimedia client, and the features of image restoration can be learned using a generative adversarial network, and an image frame restoration model can be deployed on the server; then, when the associated aggregation graph is transmitted to the image frame restoration model, the image frame restoration model restores the missing or tampered image data based on the image restoration operator, and outputs the restored image data through the image frame restoration model, and then encodes the continuously output image data through the H.265 compression encoding algorithm to obtain a video stream received by the multimedia client.

[0049] It should be noted that in this application, by establishing an image frame repair model in the channel from the multimedia server to the multimedia client, and using the model to repair the associated aggregated image, the quality of the video stream can be effectively improved. The repair operator facilitates the accurate repair of tampered or lost data during the transmission of the video stream, thereby ensuring the integrity and continuity of the video quality.

[0050] It can be seen that in the present application, anti-tampering filtering and repair of video data can be achieved during efficient video transmission. First, by obtaining the original video stream on the server side, basic data is provided for subsequent operations, and frame-by-frame decoding and difference analysis are performed to extract the core data stream in the video to reduce the transmission of redundant information and reduce bandwidth pressure; secondly, the core pixels of the transmission are determined through the core data stream, and then the image is reconstructed to obtain a core reconstructed image, which can supplement the lost or tampered parts, and by extracting and utilizing the core pixels, the lost or erroneous data during the transmission process is solved, thereby avoiding the degradation of the quality of the transmitted image frame; then, by performing expansion and corrosion morphological filtering operations on the core reconstructed image, the tampered area and core pixels are further extracted from the core reconstructed image, providing accurate points to be repaired and key data for image repair and integrity assurance, which is conducive to high-quality output of the image, and comprehensively considering time and space factors, the boundary tampering points are temporally and spatially associated, which can effectively repair the errors caused by packet loss or tampering during network transmission, and ensure the integrity of the repaired image; finally, the associated aggregated image is repaired and transmitted based on the repair operator, which can ensure that the quality of the video stream finally received by the client is not affected.

[0051] In summary, the technical solution adopted in this application can realize tamper-proof filtering and repair of video data during efficient video transmission, so as to improve the quality of video streams received by multimedia clients.

[0052] Embodiment 2: This application provides a multimedia data transmission system, referring to Figure 4 As shown, this figure is a schematic diagram of a transmission system according to this embodiment of the present application, and the transmission system includes: The acquisition module 100 is used to acquire the original video stream collected by the multimedia server; The pixel reconstruction module 200 is used to perform inter-frame difference decoding verification on the original video stream to obtain a core data stream, determine the core pixels transmitted by the multimedia server through the core data stream, and then reconstruct the image frames in the original video stream based on the core pixels to obtain a core reconstructed image; An image filtering module 300 is used to perform morphological filtering on the core reconstructed image according to the pixel information of the core reconstructed image to obtain the reconstructed core points and the boundary tampering points during image reconstruction; The association aggregation module 400 is used to obtain the spatiotemporal information of the original video stream, and to perform spatiotemporal association aggregation on the boundary tampering points through the time features and spatial features in the spatiotemporal information to obtain an association aggregation image; The repair and transmission module 500 is used to determine an image repair operator according to the reconstructed core point, and repair and transmit the associated aggregated image based on the image repair operator, so as to obtain a video stream received by a multimedia client.

[0053] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0054] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, the storage medium including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically-erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0055] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

Claims

1. A multimedia data transmission method, characterized in that: The transmission method comprises: Get the original video stream collected by the multimedia server; Performing inter-frame difference decoding verification on the original video stream to obtain a core data stream, determining core pixels transmitted by the multimedia server end through the core data stream, and then reconstructing image frames in the original video stream based on the core pixels to obtain a core reconstructed image; Performing morphological filtering on the core reconstructed image according to pixel information of the core reconstructed image to obtain reconstructed core points and boundary tampering points during image reconstruction; Acquire the spatiotemporal information of the original video stream, and perform spatiotemporal correlation aggregation on the boundary tampering points through the time features and spatial features in the spatiotemporal information to obtain a correlation aggregated image; An image restoration operator is determined according to the reconstructed core point, and the associated aggregated image is restored and transmitted based on the image restoration operator, thereby obtaining a video stream received by a multimedia client.

2. A multimedia data transmission method as claimed in claim 1, characterized in that: The original video stream collected by the multimedia server is obtained by connecting to the secure access machine.

3. A multimedia data transmission method as claimed in claim 1, characterized in that: The original video stream is subjected to inter-frame difference decoding verification to obtain a core data stream including: Decoding the original video stream frame by frame to obtain a continuous image frame sequence; Determining a pixel threshold through the image frame sequence; The core data of each image frame in the image frame sequence is filtered according to the pixel threshold to obtain a core data stream.

4. A multimedia data transmission method as claimed in claim 1, characterized in that: Determining the core pixels transmitted by the multimedia server through the core data stream specifically includes: Performing spatiotemporal collaborative filtering on the core data stream to obtain a joint mask; filtering core pixel data according to the joint mask; The core pixel data is encoded to obtain core pixels transmitted by the multimedia server.

5. A multimedia data transmission method as claimed in claim 1, characterized in that: Reconstructing the image frames in the original video stream based on the core pixels to obtain the core reconstructed image specifically includes: Construct a blank image matrix; Determining a compensation domain of an image frame in the original video stream; The blank image matrix is ​​used to perform compensation reconstruction on the compensation domain based on the core pixels to obtain a core reconstructed image.

6. A multimedia data transmission method as claimed in claim 1, characterized in that: The core reconstructed image is morphologically filtered according to the pixel information of the core reconstructed image to obtain the reconstructed core points and the boundary tampering points during image reconstruction, specifically including: Performing dilation and erosion filtering operations on the core reconstructed image respectively to obtain a filtering overlap area of ​​the core reconstructed image; Selecting the reconstruction core points during image reconstruction from the filtering overlap area; The boundary tampering points during image reconstruction are marked from the core reconstructed image based on the filtered overlap area by an edge detection algorithm.

7. A multimedia data transmission method as claimed in claim 1, characterized in that: The spatiotemporal information of the original video stream is obtained through a spatiotemporal segmentation algorithm.

8. A multimedia data transmission method as claimed in claim 1, characterized in that: The multimedia server is a hybrid communication server.

9. A multimedia data transmission method as claimed in claim 1, characterized in that: The multimedia client is a software client.

10. A multimedia data transmission system, used to execute a multimedia data transmission method according to any one of claims 1 to 9, characterized in that: The transmission system comprises: The acquisition module is used to acquire the original video stream collected by the multimedia server; A pixel reconstruction module is used to perform inter-frame difference decoding verification on the original video stream to obtain a core data stream, determine the core pixels transmitted by the multimedia server through the core data stream, and then reconstruct the image frames in the original video stream based on the core pixels to obtain a core reconstructed image; An image filtering module, used for performing morphological filtering on the core reconstructed image according to pixel information of the core reconstructed image, to obtain reconstructed core points and boundary tampering points during image reconstruction; An association aggregation module, used to obtain the spatiotemporal information of the original video stream, and perform spatiotemporal association aggregation on the boundary tampering points through the time features and spatial features in the spatiotemporal information to obtain an association aggregation image; The repair and transmission module is used to determine an image repair operator according to the reconstructed core point, and repair and transmit the associated aggregated image based on the image repair operator to obtain a video stream received by a multimedia client.

Images