Multi-core-based self-adaptive streaming media video processing system

Abstract

The invention, which relates to the technical field of video processing, discloses a multi-core-based and adaptive streaming media video processing system comprising a video acquisition unit, an image frame processing unit, a target area merging unit, a spliced image combination unit, a video coding unit and an image frame rendering unit. According to the invention, all image frames contained in video data are analyzed through the image frame processing unit, projection transformation is carried out on each pixel point under image coordinate systems corresponding to different viewpoints, and if the projected pixel point overlaps with the pixel point at the original position, the image frame processing unit carries out projection transformation on the pixel point. Effective information contained in a current pixel point is expressed by utilizing a difference value, pixel points with less effective information are preferentially deleted in a subsequent processing process, pixel point redundancy and storage space waste are avoided, and meanwhile, in order to reduce the requirement of equipment performance, pixel points are concentrated in a plurality of target blocks by utilizing a target area merging unit, so that the efficiency of the equipment performance is improved. It is ensured that the data packet can store more pixel points in the transmission process, and resource waste is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of video processing, and in particular to a multi-core-based and adaptive streaming media video processing system. Background Art

[0002] Streaming video processing systems usually integrate advanced multi-core processor technology and adaptive streaming technology, aiming to provide efficient, flexible and high-quality video processing services. It has powerful parallel processing capabilities, dynamic video playback quality adjustment capabilities and rich video processing capabilities. The invention patent with application number 202011207039.2 discloses "a video processing system and a video processing method. The video processing system includes: multiple edge devices, which are set at different locations, and each edge device is configured to obtain scene data at the location; an Internet of Things device, coupled to multiple edge devices, and configured to receive and store scene data; a back-end device, coupled to the Internet of Things device, and configured to call corresponding scene data from the Internet of Things device according to business needs, and perform corresponding back-end data processing. The video processing system can provide students with a better high-definition course video viewing experience while reducing equipment costs."

[0003] The above-mentioned existing technology solves the problem that traditional devices are gradually unable to meet users' needs for multimedia video viewing. However, when the system is running, redundant pixels in the video data are not deleted before video transmission, resulting in a waste of storage space, and no priority is set for image blocks, resulting in the loss of some important pixel information during compression and transmission, affecting the integrity of video playback. The system is also unable to complete video encoding in a short time, resulting in excessively long video transmission time, affecting the user experience. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-core based and adaptive streaming media video processing system to solve the problems raised in the above background technology.

[0005] To achieve the above-mentioned object, the present invention provides the following technical solutions: a multi-core-based and adaptive streaming media video processing system, comprising a video encoding unit and an image frame rendering unit;

[0006] A video acquisition unit, which acquires video data and viewpoint parameters using a video acquisition device and transmits them to a storage device;

[0007] An image frame processing unit, after receiving the video data and viewpoint parameters, analyzes the image frames in the video in sequence, determines the final difference value of each pixel under different viewpoints, sets a cumulative interval, lists all image frames within each cumulative interval, accumulates the final difference values ​​of different pixels at the same pixel position, and uses the obtained value as the total difference value of the current pixel position within the cumulative interval. Pruning operations are performed on all image frames using image processing software to obtain pruning mask images corresponding to different viewpoints, sets a difference threshold, and deletes the pixel corresponding to the current total difference value if the total difference value is lower than the difference threshold;

[0008] a target area merging unit, wherein the target area merging unit connects the retained pixel points with the adjacent pixel points to obtain multiple target areas, determines the difference mean and difference variance value of each target area, generates a rectangular frame of corresponding size according to the position of each pixel point in each target area, and when there are other complete target areas in the rectangular frame, calculates the difference variance value according to the difference mean of the target area corresponding to the rectangular frame and the complete target area; if the difference variance value is less than a preset value, merges the two to obtain a new target area, calculates the area ratio between each target area and its corresponding rectangular frame, and segments the target areas whose area ratio exceeds an area threshold and whose difference variance value is greater than the variance threshold, thereby obtaining multiple target blocks;

[0009] A mosaic graph combination unit, after determining the difference mean and partition value of the target block, scans the target blocks one by one. If the difference mean of the current target block is greater than the partition value, it is stored in the high zone set; otherwise, it is stored in the low zone set. After all target blocks in the high zone set are placed in the mosaic graph, all target blocks in the low zone set are placed in the mosaic graph.

[0010] Preferably, the image frame processing unit includes a difference value analysis module and a cumulative interval setting module. After receiving the video data and viewpoint parameters, the difference value analysis module analyzes the image frames in the video in sequence and arbitrarily selects an unmarked viewpoint δ k , all pixels in the image are at the viewpoint δ k Perform projection transformation in the corresponding image coordinate system. If there is a pixel point Projected position and pixel points When the original position of the projected pixel coincides with the original position of the projected pixel, the projected pixel is counted. and Texture difference between Grayscale difference according to and Calculate pixel points At viewpoint δ k The difference value under in If it does not exist If the projected position coincides with the original position of any pixel point, the current viewpoint is marked and other viewpoints are reselected. The cumulative interval setting module counts the difference values ​​of each pixel point under different viewpoints, takes the minimum difference value as the final difference value of the current pixel point, sets the cumulative interval, and lists all image frames within each cumulative interval to determine the final difference values ​​of different pixels at the same pixel position.

[0011] Preferably, the image frame processing unit further includes a null value replacement module, a total difference value determination module, and a pixel point screening module. The null value replacement module determines whether the final difference value of different pixels at the same pixel position is a null value. If it is a null value, the original null value is replaced by a preset value as the final difference value of the different pixels at the pixel position; otherwise, no operation is performed. The total difference value determination module counts the final difference values ​​of different pixels at the same pixel position, accumulates them, and uses the obtained value as the total difference value of the current pixel position within the accumulation interval. The viewpoint corresponding to each total difference value is determined, and the total difference values ​​under the same viewpoint are arranged in descending order. The pixel point screening module uses image processing software to perform a pruning operation on all image frames within the accumulation interval. After obtaining multiple pruning hierarchical graphs corresponding to different viewpoints, these pruning hierarchical graphs are aggregated into a pruning mask graph, and a difference threshold is set according to a specified pixel rate. If the total difference value is lower than the difference threshold, the pixel point corresponding to the current total difference value is deleted; otherwise, the pixel point corresponding to the current total difference value is retained.

[0012] Preferably, the target area merging unit includes a difference mean calculation module, a rectangular frame construction module, a region analysis module and a target block generation module. The difference mean calculation module connects the retained pixel points with the adjacent pixel points to obtain multiple target areas, and after accumulating the total difference values ​​of all pixel points in the target area, determines the number of pixels, and calculates the difference mean of each target area according to the accumulated value and the number of pixels. The rectangular frame construction module uses a variance analysis algorithm to calculate the difference mean of each target area and the total difference value of each pixel point in the corresponding area to obtain the difference variance value of the target area, generates a rectangular frame of corresponding size according to the position of each pixel point in each target area, and determines whether there are other complete target areas in each rectangular frame. If there are no other complete target areas, then the target area is not included. Perform a merging operation. If there are other complete target areas, the regional analysis module determines the difference mean and the number of pixels of the complete target area. The difference variance value is calculated based on the difference mean and the number of pixels between the target area corresponding to the rectangular frame and the complete target area. If the difference variance value is less than a preset value, the two are merged to obtain a new target area. Otherwise, the merging operation is not performed. The target block generation module counts the area ratio between each target area and its corresponding rectangular frame, and segments the target area whose area ratio exceeds the area threshold. The operation is stopped until the area ratio after segmentation is less than or equal to the ratio before segmentation. The difference variance value corresponding to the current target area is determined, and the target area whose difference variance value is greater than the variance threshold is segmented again, thereby obtaining multiple target blocks. The variance analysis algorithm is specifically as follows:

[0013]

[0014] Where, Indicates that in δ k The difference variance value of the lth target area under the viewpoint, p i represents the i-th pixel, R l represents the lth target area, n represents the total number of pixels in the target area, Indicates that in δ k The total difference value of the i-th pixel under the viewpoint, δ k represents the k-th viewpoint, and i, l, k represent parameters.

[0015] Preferably, the mosaic image combination unit includes a target block analysis module, a partition demarcation module and a target block layout module. The target block analysis module calculates the difference mean of the target block using the total difference value of each pixel point contained in the target block, and arranges these target blocks in order from large to small according to the difference mean. The partition demarcation module pre-sets the partition value and scans the target blocks one by one. If the difference mean of the current target block is greater than the partition value, it is stored in the high zone set. If the difference mean of the current target block is less than or equal to the partition value, it is stored in the low zone set. The target block layout module arranges all target blocks in the high zone set in order from large to small in area, and puts them into the mosaic image in sequence. Then, all target blocks in the low zone set are arranged in order from large to small in difference mean. If there is remaining space in the mosaic image, the target blocks are placed in sequence in the mosaic image. If there is no remaining space in the mosaic image, all target blocks in the current low zone set are deleted.

[0016] Preferably, the video encoding unit includes an information conversion module and a key frame encoding module. After the information conversion module determines the viewpoint information corresponding to each spliced ​​image and the position information of each pixel in the target block, it uses an encoder to convert this information into corresponding code stream data. The key frame encoding module receives image frames according to the video sequence through the encoder, where the image frame is specifically a spliced ​​image, divides it into multiple separate pixel blocks, selects key frames from all image frames, performs intra-frame estimation and intra-frame prediction on the pixel blocks contained in these key frames, subtracts the original key frame from the prediction result, and the obtained residual data is linearly transformed to generate data coefficients. The predicted residual data is reversely calculated based on the data coefficients, and is combined with the prediction result to obtain the predicted pixel block. The key frame before division is analyzed based on the predicted pixel block, and the corresponding intra-frame coding signal is determined. The data coefficients and the coding signal are used for entropy coding to obtain the corresponding code stream data. The encoder is specifically an H.265 video encoder.

[0017] Preferably, the video encoding unit further comprises a pixel block matching module and an inter-frame coding analysis module, wherein the pixel block matching module extracts the encoded key frame and the image frame adjacent to the key frame, and uses the pixel blocks corresponding to these image frames as the current pixel block. After selecting five candidate motion vectors, the matching start block position is calculated according to each motion vector and the position of the current pixel block, and a search template is set to search in the encoded image data with the matching start block as the center. The offset value between the current pixel block and each matching block is calculated using a pixel analysis algorithm, and the matching block with the smallest offset value is selected as the optimal matching block. If the optimal matching block If the distance between the optimal matching block and the matching starting block is greater than the search threshold, the current optimal matching block is used as the matching starting block for re-searching. Otherwise, the corresponding motion vector is calculated according to the relative position of the optimal matching block and the current pixel block. The inter-frame coding analysis module reconstructs the current pixel block according to the motion vector and uses it as the prediction result. The corresponding residual data is calculated according to the prediction result and the current pixel block. After determining the data coefficient, the prediction residual data is reversely calculated through the data coefficient, and the residual data is combined with the prediction result to obtain a predicted image frame, and the inter-frame coding information is determined. The corresponding code stream data is analyzed according to the inter-frame coding information and the data coefficient.

[0018] Preferably, the image frame rendering unit includes a mosaic reconstruction module, a position coordinate statistics module and a video output module. The mosaic reconstruction module transmits all the code stream data into the decoder, performs entropy decoding and reordering, performs an inverse transformation operation on the obtained video data, extracts the residual data and generates a predicted pixel block, analyzes the corresponding key frame according to the predicted pixel block, and uses the key frame to infer the adjacent image frame data to complete the reconstruction of all mosaics. After the position coordinate statistics module determines the viewpoint information corresponding to each mosaic and the position information of each pixel in the target block, the target block in the mosaic is restored to its original position, and the statistics are calculated. The pixel position coordinates corresponding to each viewpoint, the video output module transmits the pixel position coordinates corresponding to each viewpoint to the real-time renderer, converts the position coordinates of each pixel point into the corresponding spatial coordinates according to the viewpoint parameters, selects the target viewpoint, converts the spatial coordinates of all pixel points into the position coordinates under the target viewpoint coordinate system, deletes redundant pixels, adjusts the color information of each pixel point, uses the image optimization algorithm to analyze the depth information of each pixel point in the current image frame, fills the vacant positions, and transmits the rendered image frame to the frame buffer for output in sequence. The decoder is specifically an H.265 video decoder.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] 1. The present invention analyzes all image frames contained in the video data through an image frame processing unit, and performs projection transformation on each pixel in the image coordinate system corresponding to different viewpoints. If the projected pixel overlaps with the pixel at the original position, the difference value is used to represent the valid information contained in the current pixel. In the subsequent processing process, the pixel with less valid information is deleted first to avoid pixel redundancy and waste of storage space. At the same time, in order to reduce the performance requirements of the device, the target area merging unit is used to concentrate the retained pixels in the target area and divide the target area to obtain a number of target blocks, ensuring that each target block can contain more pixels during the transmission process to avoid resource waste. The priority order of the target blocks is also set in the mosaic combination unit. In this way, even if the pixel rate is limited, it can be ensured that important target blocks will not be discarded, which is convenient for subsequent video rendering.

[0021] 2. The present invention uses a video encoding unit to encode pixel position information, key frames and image frames adjacent to key frames respectively, to ensure that video data is further compressed before transmission, thereby improving the transmission rate. In order to ensure that the image frame can shorten the processing time during the encoding process, the search range is continuously expanded when matching with the key frame pixel blocks, so that different pixel blocks can filter out the optimal matching blocks and complete the encoding operation in a short time. In order to ensure the real-time effect of the video, the system uses a GPU-based real-time renderer in the image frame rendering unit to perform multi-faceted rendering operations on the image frame at the same time, to ensure that the image picture can be clearer, and the user can specify the viewpoint to obtain the corresponding picture information, thereby enhancing the video viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Provides a schematic diagram of the overall system flow for an embodiment of the present invention;

[0023] Figure 2 A block diagram of the internal modules of the image frame processing unit provided by an embodiment of the present invention;

[0024] Figure 3 A block diagram of the internal modules of the target area merging unit provided in an embodiment of the present invention;

[0025] Figure 4 A block diagram of the internal modules of the mosaic image combination unit provided in an embodiment of the present invention;

[0026] Figure 5 A block diagram of the internal modules of a video encoding unit provided by an embodiment of the present invention;

[0027] Figure 6 This is a block diagram of the internal modules of the image frame rendering unit provided by an embodiment of the present invention.

[0028] In the figure: 1, video acquisition unit; 2, image frame processing unit; 201, difference value analysis module; 202, cumulative interval setting module; 203, null value replacement module; 204, difference total value determination module; 205, pixel point screening module; 3, target region merging unit; 301, difference average calculation module; 302, rectangular frame construction module; 303, region analysis module; 304, target block generation module; 4, spliced picture combination unit; 401, target block analysis module; 402, partitioning module; 403, target block layout module; 5, video encoding unit; 501, information conversion module; 502, key frame encoding module; 503, pixel block matching module; 504, inter-frame encoding analysis module; 6, image frame rendering unit; 601, spliced picture reconstruction module; 602, position coordinate statistics module; 603, video output module. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0030] Please refer to Figures 1-6 The present application provides a technical solution: a multi-core and self-adaptive streaming media video processing system, comprising a video encoding unit 5 and an image frame rendering unit 6.

[0031] The video acquisition unit 1 acquires video data and viewpoint parameters using a video acquisition device, and then transmits them to a storage device.

[0032] The image frame processing unit 2 receives video data and viewpoint parameters, and then analyzes the image frames in the video one by one. After determining the final difference values of each pixel point under different viewpoints, it sets a cumulative interval and lists all the image frames in each cumulative interval. The final difference values of different pixel points at the same pixel position are added up, and the obtained value is taken as the difference total value of the current pixel position in the cumulative interval. The image processing software is used to perform pruning operation on all the image frames, to obtain pruning mask pictures corresponding to different viewpoints. A difference threshold is set. If the difference total value is lower than the difference threshold, the pixel point corresponding to the current difference total value is deleted.

[0033] The target area merging unit 3 connects the retained pixel points with the adjacent pixel points to obtain multiple target areas, determines the difference mean and difference variance value of each target area, generates a rectangular frame of corresponding size according to the position of each pixel point in each target area, and when there are other complete target areas in the rectangular frame, calculates the difference variance value according to the difference mean of the target area corresponding to the rectangular frame and the complete target area. If the difference variance value is less than a preset value, the two are merged to obtain a new target area, and counts the area ratio between each target area and its corresponding rectangular frame. The target areas whose area ratio exceeds the area threshold and whose difference variance value is greater than the variance threshold are segmented to obtain multiple target blocks;

[0034] The mosaic image combination unit 4 determines the difference mean and partition value of the target block, and then scans the target blocks one by one. If the difference mean of the current target block is greater than the partition value, it is stored in the high zone set; otherwise, it is stored in the low zone set. After all the target blocks in the high zone set are placed in the mosaic image, all the target blocks in the low zone set are placed in the mosaic image.

[0035] The image frame processing unit 2 includes a difference value analysis module 201 and a cumulative interval setting module 202. After receiving the video data and viewpoint parameters, the difference value analysis module 201 analyzes the image frames in the video in sequence and randomly selects an unmarked viewpoint δ k , all pixels in the image are at the viewpoint δ k Perform projection transformation in the corresponding image coordinate system. If there is a pixel point The projected position and pixel point p x When the original positions of b2 and y2 coincide, the projected pixel points are counted. and Texture difference between Grayscale difference according to and Calculate pixel points At viewpoint δ k The difference value under in If it does not exist If the projected position coincides with the original position of any pixel, the current viewpoint is marked and another viewpoint is selected. The accumulation interval setting module 202 counts the difference values ​​of each pixel under different viewpoints, takes the minimum difference value as the final difference value of the current pixel, sets the accumulation interval, and lists all image frames within each accumulation interval to determine the final difference values ​​of different pixels at the same pixel position.

[0036] The image frame processing unit 2 also includes a null value replacement module 203, a total difference value determination module 204, and a pixel point screening module 205. The null value replacement module 203 determines whether the final difference value of different pixel points at the same pixel position is a null value. If it is a null value, the original null value is replaced by a preset value as the final difference value of the different pixel points at the pixel position. Otherwise, no operation is performed. The total difference value determination module 204 counts the final difference values ​​of different pixel points at the same pixel position and accumulates them. The obtained value is used as the total difference value of the current pixel position within the accumulation interval, and the viewpoint corresponding to each total difference value is determined. The total difference values ​​under the same viewpoint are arranged in order from large to small. The pixel point screening module 205 uses image processing software to perform a pruning operation on all image frames within the accumulation interval. After obtaining multiple pruning hierarchical graphs corresponding to different viewpoints, these pruning hierarchical graphs are aggregated into a pruning mask graph, and a difference threshold is set according to a specified pixel rate. If the total difference value is lower than the difference threshold, the pixel point corresponding to the current total difference value is deleted. Otherwise, the pixel point corresponding to the current total difference value is retained.

[0037] The target area merging unit 3 includes a difference mean calculation module 301, a rectangular frame construction module 302, a region analysis module 303 and a target block generation module 304. The difference mean calculation module 301 connects the retained pixel points with the adjacent pixel points to obtain multiple target areas, and after accumulating the total difference values ​​of all pixel points in the target area, determines the number of pixel points, and calculates the difference mean of each target area according to the accumulated value and the number of pixel points. The rectangular frame construction module 302 uses a variance analysis algorithm to calculate the difference mean of each target area and the total difference value of each pixel point in the corresponding area to obtain the difference variance value of the target area, generates a rectangular frame of corresponding size according to the position of each pixel point in each target area, and determines whether there are other complete target areas in each rectangular frame. If there are no other complete target areas, If there is no complete target area, the merging operation is not performed. If there are other complete target areas, the regional analysis module 303 determines the difference mean and the number of pixels of the complete target area, and calculates the difference variance value based on the difference mean and the number of pixels between the target area corresponding to the rectangular frame and the complete target area. If the difference variance value is less than a preset value, the two are merged to obtain a new target area. Otherwise, the merging operation is not performed. The target block generation module 304 counts the area ratio between each target area and its corresponding rectangular frame, and segments the target area whose area ratio exceeds the area threshold. The operation is stopped until the area ratio after segmentation is less than or equal to the ratio before segmentation. The difference variance value corresponding to the current target area is determined, and the target area whose difference variance value is greater than the variance threshold is segmented again, thereby obtaining multiple target blocks. The variance analysis algorithm is specifically as follows:

[0038]

[0039] Where, Indicates that in δ k The difference variance value of the lth target area under the viewpoint, p i represents the i-th pixel, R l represents the lth target area, n represents the total number of pixels in the target area, Indicates that in δ k The total difference value of the i-th pixel under the viewpoint, δ k represents the k-th viewpoint, i, l, k represent parameters;

[0040] The mosaic image combination unit 4 includes a target block analysis module 401, a partitioning module 402 and a target block layout module 403. The target block analysis module 401 calculates the difference mean of the target block using the total difference value of each pixel point contained in the target block, and arranges these target blocks in order from large to small according to the difference mean. The partitioning module 402 pre-sets the partition value and scans the target blocks one by one. If the difference mean of the current target block is greater than the partition value, it is stored in the high zone set. If the difference mean of the current target block is less than or equal to the partition value, it is stored in the low zone set. The target block layout module 403 arranges all target blocks in the high zone set in order from large to small in area, puts them into the mosaic image in sequence, and then arranges all target blocks in the low zone set in order from large to small in difference mean. If there is remaining space in the mosaic image, the target blocks are placed in the mosaic image in sequence. If there is no remaining space in the mosaic image, all target blocks in the current low zone set are deleted.

[0041] The video encoding unit 5 includes an information conversion module 501 and a key frame encoding module 502. After the information conversion module 501 determines the viewpoint information corresponding to each mosaic image and the position information of each pixel point in the target block, it uses an encoder to convert this information into corresponding code stream data. The key frame encoding module 502 receives image frames according to the video sequence through the encoder, where the image frame is specifically a mosaic image, divides it into multiple separate pixel blocks, selects key frames from all image frames, performs intra-frame estimation and intra-frame prediction on the pixel blocks contained in these key frames, subtracts the original key frames from the prediction results, and obtains residual data through linear transformation to generate data coefficients. The predicted residual data is reversely calculated based on the data coefficients, and is combined with the prediction results to obtain predicted pixel blocks. After that, the key frames before division are analyzed based on the predicted pixel blocks, and the corresponding intra-frame coding signals are determined. The data coefficients and the coding signals are entropy coded to obtain corresponding code stream data. The encoder is specifically an H.265 video encoder.

[0042] The video encoding unit 5 also includes a pixel block matching module 503 and an inter-frame coding analysis module 504. The pixel block matching module 503 extracts the encoded key frame and the image frame adjacent to the key frame, and uses the pixel blocks corresponding to these image frames as the current pixel block. After selecting five candidate motion vectors, the matching start block position is calculated according to each motion vector and the position of the current pixel block. The set search template is used to search in the encoded image data with the matching start block as the center, and the offset value between the current pixel block and each matching block is calculated using the pixel analysis algorithm. The matching block with the smallest offset value is selected as the optimal matching block. If the optimal matching block is the same as the matching start block, the optimal matching block is selected. If the distance between the blocks is greater than the search threshold, the current best matching block is used as the matching starting block for re-searching. Otherwise, the corresponding motion vector is calculated based on the relative position of the best matching block and the current pixel block. The inter-frame coding analysis module 504 reconstructs the current pixel block based on the motion vector and uses it as the prediction result. The corresponding residual data is calculated based on the prediction result and the current pixel block. After determining the data coefficient, the prediction residual data is reversely calculated based on the data coefficient. The residual data is combined with the prediction result to obtain a predicted image frame, and inter-frame coding information is determined. The corresponding code stream data is analyzed based on the inter-frame coding information and the data coefficient. The pixel analysis algorithm is specifically as follows:

[0043]

[0044] Where M represents the length of the block area, N represents the width of the block area, and C i (m,n) represents the pixel value of the current pixel block at the (m,n) coordinate, C i (m+X,n+Y) represents the pixel value of the encoded pixel block at the coordinate (m+X,n+Y), (m,n) represents the pixel coordinate, (m+X,n+Y) represents the matching pixel coordinate, and A(X,Y) represents the offset value between the current pixel block and the matching pixel block;

[0045] The image frame rendering unit 6 includes a mosaic reconstruction module 601, a position coordinate statistics module 602 and a video output module 603. The mosaic reconstruction module 601 transmits all the code stream data into the decoder, performs entropy decoding and reordering, performs an inverse transformation operation on the obtained video data, extracts the residual data and generates a predicted pixel block, analyzes the corresponding key frame according to the predicted pixel block, and uses the key frame to calculate the adjacent image frame data to complete the reconstruction of all mosaics. After the position coordinate statistics module 602 determines the viewpoint information corresponding to each mosaic and the position information of each pixel in the target block, it restores the target block in the mosaic to its original position and counts the pixel position corresponding to each viewpoint. The video output module 603 transmits the pixel position coordinates corresponding to each viewpoint to the real-time renderer, converts the position coordinates of each pixel point into the corresponding spatial coordinates according to the viewpoint parameters, selects the target viewpoint, converts the spatial coordinates of all pixel points into the position coordinates under the target viewpoint coordinate system, deletes redundant pixels, adjusts the color information of each pixel point, and uses the image optimization algorithm to analyze the depth information of each pixel point in the current image frame. After filling the vacant position, the rendered image frame is transmitted to the frame buffer and output in sequence. The decoder is specifically an H.265 video decoder, the real-time renderer is specifically a real-time renderer based on GPU, and the image optimization algorithm is specifically as follows:

[0046]

[0047] Where Z max Indicates the maximum depth value, Z o Represents the original depth value of the pixel, α represents the coefficient that controls the depth adjustment amplitude, s represents the change of the current pixel, s h Indicates the high threshold, s q represents the low threshold, and f(s) represents the depth change calculation function.

[0048] Working principle: After the video acquisition unit 1 obtains the video data, the present invention uses the difference value analysis module 201 in the image frame processing unit 2 to determine the difference value of each pixel point at different viewpoints, the cumulative interval setting module 202 analyzes the final difference value of different pixel points at the same pixel position, the null value replacement module 203 uses a preset value to replace the original null value as the final difference value of the pixel point, and the total difference value determination module 204 analyzes the total difference value of different pixel positions within the cumulative interval. The pixel point screening module 205 deletes the pixel points whose total difference value is lower than the difference threshold, and the difference mean calculation module 301 in the target area merging unit 3 calculates the difference mean of each target area. The rectangular frame construction module 302 sets a corresponding rectangular frame for each target area. The area analysis module 303 merges part of the target area. The target block generation module 304 divides the target area into multiple target blocks, and the target block division module 305 in the mosaic image combination unit 4 is used to divide the target area into multiple target blocks. The analysis module 401 calculates the mean difference of the target block, stores the target block in a high zone set and a low zone set through the partitioning module 402, and places the target block into the mosaic map in sequence through the target block layout module 403. The viewpoint information and pixel position information are converted into corresponding bitstream data by the information conversion module 501 in the video encoding unit 5. The key frame encoding module 502 analyzes the key frame to obtain the corresponding bitstream data. The pixel block matching module 503 matches the uncoded pixel block with the coded pixel block to select the optimal matching block. The inter-frame coding analysis module 504 analyzes the bitstream data corresponding to the image frame adjacent to the key frame. The mosaic map reconstruction module 601 in the image frame rendering unit 6 completes the reconstruction of all mosaic maps. The position coordinate statistics module 602 calculates the pixel position coordinates corresponding to each viewpoint. The rendered image frames are transmitted to the frame buffer through the video output module 603 and output in sequence.

[0049] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0050] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A multi-core and adaptive streaming media video processing system, comprising a video encoding unit (5) and an image frame rendering unit (6), characterized in that: A video acquisition unit (1), wherein the video acquisition unit (1) acquires video data and viewpoint parameters using a video acquisition device and transmits the acquired data to a storage device; An image frame processing unit (2) is provided. After receiving the video data and viewpoint parameters, the image frame processing unit (2) sequentially analyzes the image frames in the video, determines the final difference value of each pixel point at different viewpoints, sets a cumulative interval, lists all image frames within each cumulative interval, accumulates the final difference values ​​of different pixel points at the same pixel position, and uses the obtained value as the total difference value of the current pixel position within the cumulative interval. The image processing software is used to perform a pruning operation on all image frames to obtain pruning mask images corresponding to different viewpoints, sets a difference threshold, and deletes the pixel point corresponding to the current total difference value if the total difference value is lower than the difference threshold; A target region merging unit (3) is configured to connect the retained pixel points with the adjacent pixel points to obtain a plurality of target regions, determine a difference mean and a difference variance value for each target region, generate a rectangular frame of corresponding size according to the position of each pixel point in each target region, and when there are other complete target regions in the rectangular frame, calculate a difference variance value according to the difference mean between the target region corresponding to the rectangular frame and the complete target region, and if the difference variance value is less than a preset value, merge the two to obtain a new target region, calculate an area ratio between each target region and its corresponding rectangular frame, and segment the target regions whose area ratio exceeds an area threshold and whose difference variance value is greater than a variance threshold, thereby obtaining a plurality of target blocks; The mosaic image combination unit (4) determines the difference mean and partition value of the target block, scans the target blocks one by one, and if the difference mean of the current target block is greater than the partition value, stores it in the high zone set; otherwise, stores it in the low zone set; after all the target blocks in the high zone set are placed in the mosaic image, all the target blocks in the low zone set are placed in the mosaic image.

2. The multi-core and adaptive streaming media video processing system according to claim 1, characterized in that: The image frame processing unit (2) comprises a difference value analysis module (201) and a cumulative interval setting module (202). After receiving the video data and viewpoint parameters, the difference value analysis module (201) sequentially analyzes the image frames in the video and randomly selects an unmarked viewpoint δ k , all pixels in the image are at the viewpoint δ k Perform projection transformation in the corresponding image coordinate system. If there is a pixel point Projected position and pixel points When the original position of the projected pixel coincides with the original position of the projected pixel, the projected pixel is counted. and Texture difference between Grayscale difference according to and Calculate pixel points At viewpoint δ k The difference value under in If it does not exist If the projected position coincides with the original position of any pixel point, the current viewpoint is marked and another viewpoint is reselected. The cumulative interval setting module (202) counts the difference values ​​of each pixel point at different viewpoints, takes the minimum difference value as the final difference value of the current pixel point, sets the cumulative interval, and lists all image frames within each cumulative interval to determine the final difference values ​​of different pixel points at the same pixel position.

3. The multi-core and adaptive streaming media video processing system according to claim 2, characterized in that: The image frame processing unit (2) further comprises a null value replacement module (203), a difference total value determination module (204) and a pixel point screening module (205). The null value replacement module (203) determines whether the final difference value of different pixel points at the same pixel position is a null value. If it is a null value, the original null value is replaced by a preset value as the final difference value of the different pixel points at the pixel position. Otherwise, no operation is performed. The difference total value determination module (204) counts the final difference values ​​of different pixel points at the same pixel position and accumulates them to obtain the value as the final difference value. The total difference value of the previous pixel position within the accumulation interval is determined, the viewpoint corresponding to each total difference value is determined, and the total difference values ​​under the same viewpoint are arranged in order from large to small. The pixel point screening module (205) uses image processing software to perform pruning operations on all image frames within the accumulation interval, and after obtaining multiple pruning hierarchical graphs corresponding to different viewpoints, these pruning hierarchical graphs are aggregated into a pruning mask graph, and a difference threshold is set according to a specified pixel rate. If the total difference value is lower than the difference threshold, the pixel point corresponding to the current total difference value is deleted, otherwise the pixel point corresponding to the current total difference value is retained.

4. The multi-core and adaptive streaming media video processing system according to claim 1, characterized in that: The target area merging unit (3) comprises a difference mean value calculation module (301), a rectangular frame construction module (302), a region analysis module (303) and a target block generation module (304). The difference mean value calculation module (301) connects the retained pixel points with the adjacent pixel points to obtain a plurality of target areas, accumulates the difference total values ​​of all pixel points in the target area, determines the number of pixel points, and calculates the difference mean value of each target area according to the accumulated value and the number of pixel points. The rectangular frame construction module (302) calculates the difference mean value of each target area and the difference total value of each pixel point in the corresponding area using a variance analysis algorithm to obtain the difference variance value of the target area, generates a rectangular frame of corresponding size according to the position of each pixel point in each target area, and determines whether there is other complete target area in each rectangular frame. If there are no other complete target areas, the merging operation is not performed. If there are other complete target areas, the area analysis module (303) determines the difference mean and the number of pixels of the complete target area, and calculates the difference variance value based on the difference mean and the number of pixels of the target area corresponding to the rectangular frame and the complete target area. If the difference variance value is less than a preset value, the two are merged to obtain a new target area. Otherwise, the merging operation is not performed. The target block generation module (304) counts the area ratio between each target area and its corresponding rectangular frame, and segments the target area whose area ratio exceeds the area threshold, and stops the operation until the area ratio after segmentation is less than or equal to the ratio before segmentation, determines the difference variance value corresponding to the current target area, and segments the target area whose difference variance value is greater than the variance threshold again, thereby obtaining multiple target blocks.

5. The multi-core and adaptive streaming media video processing system according to claim 1, characterized in that: The mosaic image combination unit (4) comprises a target block analysis module (401), a partitioning module (402) and a target block layout module (403). The target block analysis module (401) calculates the difference mean of the target block using the total difference value of each pixel point contained in the target block, and arranges the target blocks in descending order of the difference mean. The partitioning module (402) pre-sets a partition value and scans the target blocks one by one. If the difference mean of the current target block is greater than the partition value, it is stored in a high zone set. If the difference mean of the current target block is less than or equal to the partition value, it is stored in a low zone set. The target block layout module (403) arranges all target blocks in the high zone set in descending order of area, and puts them into the mosaic image in sequence. Then, all target blocks in the low zone set are arranged in descending order of difference mean. If there is residual space in the mosaic image, the target blocks are placed in the mosaic image in sequence. If there is no residual space in the mosaic image, all target blocks in the current low zone set are deleted.

6. The multi-core and adaptive streaming media video processing system according to claim 1, characterized in that: The video encoding unit (5) includes an information conversion module (501) and a key frame encoding module (502). After determining the viewpoint information corresponding to each spliced ​​image and the position information of each pixel point in the target block, the information is converted into corresponding code stream data using an encoder. The key frame encoding module (502) receives image frames according to a video sequence through an encoder, wherein the image frames are specifically spliced ​​images, divides the image frames into multiple separate pixel blocks, selects key frames from all image frames, performs intra-frame estimation and intra-frame prediction on the pixel blocks contained in the key frames, subtracts the original key frames from the prediction results, and obtains residual data through linear transformation to generate data coefficients. The predicted residual data is reversely calculated based on the data coefficients, and is combined with the prediction results to obtain predicted pixel blocks. The key frames before division are analyzed based on the predicted pixel blocks, and the corresponding intra-frame coding signals are determined. The data coefficients and the coding signals are used for entropy coding to obtain corresponding code stream data.

7. The multi-core and adaptive streaming media video processing system according to claim 6, characterized in that: The video encoding unit (5) further comprises a pixel block matching module (503) and an inter-frame coding analysis module (504). The pixel block matching module (503) extracts the encoded key frame and the image frame adjacent to the key frame, and uses the pixel blocks corresponding to these image frames as the current pixel block. After selecting five candidate motion vectors, the matching start block position is calculated according to each motion vector and the position of the current pixel block. The encoded image data is searched using a set search template with the matching start block as the center. The offset value between the current pixel block and each matching block is calculated using a pixel analysis algorithm, and the matching block with the smallest offset value is selected as the optimal matching block. If the optimal matching block is the best matching block, the optimal matching block is selected. If the distance between the optimal matching block and the matching starting block is greater than the search threshold, the current optimal matching block is used as the matching starting block for re-searching. Otherwise, the corresponding motion vector is calculated based on the relative position of the optimal matching block and the current pixel block. The inter-frame coding analysis module (504) reconstructs the current pixel block based on the motion vector and uses it as the prediction result. The corresponding residual data is calculated based on the prediction result and the current pixel block. After determining the data coefficient, the prediction residual data is reversely calculated through the data coefficient, and the residual data is combined with the prediction result to obtain a predicted image frame, and inter-frame coding information is determined. The corresponding code stream data is analyzed based on the inter-frame coding information and the data coefficient.

8. The multi-core and adaptive streaming media video processing system according to claim 1, characterized in that: The image frame rendering unit (6) includes a mosaic reconstruction module (601), a position coordinate statistics module (602) and a video output module (603). The mosaic reconstruction module (601) transmits all code stream data into a decoder, performs entropy decoding and reordering, performs an inverse transformation operation on the obtained video data, extracts residual data and generates a predicted pixel block, analyzes the corresponding key frame based on the predicted pixel block, and uses the key frame to calculate the adjacent image frame data to complete the reconstruction of all mosaics. After the position coordinate statistics module (602) determines the viewpoint information corresponding to each mosaic and the position information of each pixel in the target block, the mosaic is reconstructed. After the target block in the figure is restored to its original position, the pixel position coordinates corresponding to each viewpoint are counted, and the video output module (603) transmits the pixel position coordinates corresponding to each viewpoint to the real-time renderer. After converting the position coordinates of each pixel point into the corresponding spatial coordinates according to the viewpoint parameters, the target viewpoint is selected, and the spatial coordinates of all pixel points are converted into position coordinates under the target viewpoint coordinate system, and redundant pixels are deleted. After adjusting the color information of each pixel point, the depth information of each pixel point in the current image frame is analyzed using an image optimization algorithm. After filling the vacant positions, the rendered image frame is transmitted to the frame buffer and output in sequence.

Images