Intelligent picture data replacement system

By processing live videos in real time on the live video client and using the deep neural network model intelligent prediction filtering algorithm, the problem of poor filtering effects of different live videos in the existing technology is solved, and a clearer live video effect is achieved.

CN118945423AActive Publication Date: 2024-11-12HUNAN GUOYAN NETWORK TECHNOLOGY CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411233089.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-11-17
Filing Date
2024-09-04
Publication Date
2024-11-12
Estimated Expiration
2044-09-04

AI Technical Summary

Technical Problem

The prior art is difficult to select the best filtering algorithm suitable for different live broadcast images in the same live broadcast room, resulting in poor filtering effect.

Method used

The live broadcast screen is processed in real time on the live broadcast client, extract multiple information parameters through geometric correction images, and use the deep neural network model intelligent prediction filtering algorithm to select the best filtering algorithm to process the live broadcast screen to generate a clearer image.

Benefits of technology

It realizes targeted optimization processing for different live broadcast pictures, provides the best filtering algorithm selection reference, and improves the clarity and stability of live broadcast pictures.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention relates to a picture data intelligent replacement system, and the system comprises a data receiving device which is used for obtaining and forwarding each frame of real-time live broadcast picture transmitted by a live broadcast client; the content screening device is used for taking the filtering algorithm corresponding to the picture definition with the highest numerical value in the picture definitions corresponding to the filtering algorithms as a target filtering algorithm; and the picture replacement device is used for replacing the frame of real-time live broadcast picture with the image filtered by the target filtering algorithm and sending the real-time live broadcast picture to a far-end live broadcast server. The intelligent picture data replacement system is reliable in logic and stable in operation. Before a live broadcast client transmits a frame of real-time live broadcast picture each time, the real-time live broadcast picture is subjected to targeted optimization processing to obtain the geometric correction image, and the filtering effect after a certain filtering algorithm is adopted is intelligently predicted based on various pieces of visual information of the geometric correction image; therefore, reference information is provided for selection of optimal filtering algorithms of different real-time live broadcast images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of live broadcast picture data analysis, and in particular to a picture data intelligent replacement system. Background Art

[0002] The layout ratio of the live broadcast room screen is an important factor that needs to be paid attention to during the live broadcast process. A reasonable layout ratio can affect the user's perception and experience. If the anchor is too close to the camera, it will give the user a feeling of oppression, and if it is too far away, it will seem distant from the user and not friendly enough. Therefore, when designing the live broadcast room screen, the anchor should occupy about 2 / 3 of the entire screen, so as to ensure that the overall size of the background, foreground and lens is moderate, giving users a better viewing experience.

[0003] However, in actual operation, different live broadcast pictures in the same live broadcast room may need to choose the best filtering algorithm suitable for themselves to ensure the best filtering effect of different picture contents and improve the picture quality of different picture contents. Using the same sharpening mode to perform the same filtering processing on different live broadcast pictures in the same live broadcast room may obtain unsatisfactory results. How to choose the best filtering algorithm suitable for different live broadcast pictures in the same live broadcast room is a technical problem that has not yet been solved by the existing technology.

[0004] The invention of publication number CN102067600A in the prior art provides a method and device for using local adaptive filtering for motion compensated interpolation and reference picture filtering. The device includes an encoder for encoding picture data. The encoder includes at least one local adaptive filter for performing local adaptive filtering for at least one of reference picture filtering and interpolation filtering on the picture data. The method includes encoding the picture data using the local adaptive filtering for at least one of reference picture filtering and interpolation filtering. The device includes a decoder for decoding the picture data. The decoder includes at least one local adaptive filter for performing local adaptive filtering for at least one of reference picture filtering and interpolation filtering on the picture data. The method includes decoding the picture data using the local adaptive filtering for at least one of reference picture filtering and interpolation filtering. Summary of the invention

[0005] In order to overcome the technical problems in the prior art, the present invention proposes an intelligent picture data replacement system, which can perform targeted optimization processing on each frame of real-time live picture before the live broadcast client transmits it to obtain a geometrically corrected image, and extract multiple information parameters from the geometrically corrected image, wherein the multiple information parameters include the number of pixel rows, pixel column values, maximum noise amplitudes corresponding to various noise types, and grayscale values ​​corresponding to each pixel point of the geometrically corrected image, and intelligently predict the picture clarity obtained after filtering the geometrically corrected image using the certain filtering algorithm based on a deep neural network model according to the algorithm ID corresponding to a certain set filtering algorithm, the contrast of the geometrically corrected image, and the multiple information parameters, thereby providing valuable reference information for selecting the optimal filtering algorithm for different real-time live pictures.

[0006] The present invention provides a picture data intelligent replacement system comprising: A data receiving device is arranged on the live broadcast client, and is used to send a frame of real-time live broadcast picture to the neighborhood interpolation device before the live broadcast client transmits each frame of real-time live broadcast picture; A neighborhood interpolation device, connected to the data receiving device, for performing Lanczos interpolation processing of a 4-pixel×4-pixel neighborhood on the received real-time live picture to obtain and output a corresponding neighborhood interpolation image; An artifact removal device, connected to the neighborhood interpolation device, for performing an artifact removal process on the received neighborhood interpolation image to obtain and output a corresponding artifact removed image; a geometric correction device, connected to the artifact removal device, for performing geometric correction processing on the received artifact removal image to obtain and output a corresponding geometric correction image; A signal prediction device, connected to the geometric correction device and comprising a grayscale analysis device, a learning processing device and a clarity prediction device, wherein the clarity prediction device is respectively connected to the grayscale analysis device and the learning processing device, and the signal prediction device is used to extract multiple information parameters from the geometric correction image, wherein the multiple information parameters include the number of pixel rows, pixel column values, respective maximum noise amplitudes corresponding to various noise types, and respective grayscale values ​​corresponding to each pixel point of the geometric correction image, perform multiple learning on the deep neural network to obtain a deep neural network model, and intelligently predict the picture clarity obtained after filtering the geometric correction image using the certain filtering algorithm based on the algorithm ID corresponding to a certain set filtering algorithm, the contrast of the geometric correction image, and the multiple information parameters of the deep neural network model, wherein the number of deep neural learning times is proportional to the resolution of the geometric correction image; A content screening device connected to the signal pre-judgment device, used to use the filtering algorithm corresponding to the highest picture definition among the picture definitions corresponding to the various filtering algorithms as the target filtering algorithm; The picture replacement device is connected to the content screening device and is used to perform filtering processing on the geometrically corrected image using a target filtering algorithm, and replace the frame of real-time live broadcast picture with the filtered image and send it to a remote live broadcast server.

[0007] The picture data intelligent replacement system of the present invention has reliable logic and stable operation. Before a live broadcast client transmits each frame of a real-time live broadcast picture, the frame can be optimized and processed to obtain a geometric correction image, and the filtering effect after using a certain filtering algorithm can be intelligently predicted based on various visual information of the geometric correction image, thereby providing reference information for the selection of the best filtering algorithm for different real-time live broadcast pictures. DETAILED DESCRIPTION

[0008] The implementation scheme of the picture data intelligent replacement system of the present invention will be described in detail below.

[0009] Implementation 1 According to embodiment 1 of the present invention, a system for intelligently replacing picture data is shown, and the system comprises: A data receiving device is arranged on the live broadcast client, and is used to send a frame of real-time live broadcast picture to the neighborhood interpolation device before the live broadcast client transmits each frame of real-time live broadcast picture; For example, the data receiving device is arranged at the live broadcast client, and is used to send a frame of real-time live broadcast picture to the neighborhood interpolation device before the live broadcast client transmits each frame of real-time live broadcast picture, and includes: the live broadcast client is connected to the remote live broadcast server through a wireless communication network; A neighborhood interpolation device, connected to the data receiving device, for performing Lanczos interpolation processing of a 4-pixel×4-pixel neighborhood on the received real-time live picture to obtain and output a corresponding neighborhood interpolation image; An artifact removal device, connected to the neighborhood interpolation device, for performing an artifact removal process on the received neighborhood interpolation image to obtain and output a corresponding artifact removed image; a geometric correction device, connected to the artifact removal device, for performing geometric correction processing on the received artifact removal image to obtain and output a corresponding geometric correction image; A signal prediction device, connected to the geometric correction device and comprising a grayscale analysis device, a learning processing device and a clarity prediction device, wherein the clarity prediction device is respectively connected to the grayscale analysis device and the learning processing device, and the signal prediction device is used to extract multiple information parameters from the geometric correction image, wherein the multiple information parameters include the number of pixel rows, pixel column values, respective maximum noise amplitudes corresponding to various noise types, and respective grayscale values ​​corresponding to each pixel point of the geometric correction image, perform multiple learning on the deep neural network to obtain a deep neural network model, and intelligently predict the picture clarity obtained after filtering the geometric correction image using the certain filtering algorithm based on the algorithm ID corresponding to a certain set filtering algorithm, the contrast of the geometric correction image, and the multiple information parameters of the deep neural network model, wherein the number of deep neural learning times is proportional to the resolution of the geometric correction image; A content screening device connected to the signal pre-judgment device, used to use the filtering algorithm corresponding to the highest picture definition among the picture definitions corresponding to the various filtering algorithms as the target filtering algorithm; A picture replacement device, connected to the content screening device, is used to perform filtering processing on the geometric correction image using a target filtering algorithm, and replace the frame of real-time live picture with the filtered image and send it to a remote live server; Among them, the target filtering algorithm is used to perform filtering processing on the geometrically corrected image, and the filtered image replaces the real-time live broadcast picture of the frame and sends it to the remote live broadcast server, including: the remote live broadcast server is a cloud computing server, a blockchain server or a big data server.

[0010] Implementation 2 According to the second embodiment of the present invention, a system for intelligently replacing picture data is shown, and compared with the first embodiment, the following components may also be included: A current sensor device is connected to the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device, respectively, and is used to measure the current real-time current values ​​of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device respectively; Wherein, the current sensing device is respectively connected to the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device, and is used to respectively measure the current real-time current values ​​of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device, including: the current sensing device includes a plurality of current measurement units, which are respectively connected to the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device, so as to complete the respective measurement of the current real-time current values ​​of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device; Wherein, the current sensing device comprises a plurality of current measuring units, which are respectively connected to the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device to complete the respective measurement of the current real-time current values ​​of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device, including: the plurality of current measuring units are a plurality of current sensing circuits, which are respectively connected to the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device to complete the respective measurement of the current real-time current values ​​of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device; The multiple current measurement units are multiple current sensing circuits, which are used to be connected to the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device respectively, so as to complete the measurement of the current real-time current values ​​of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device respectively, including: the structures of the multiple current sensing circuits are the same; Among them, the multiple current measurement units are multiple current sensing circuits, which are used to be connected to the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device respectively, so as to complete the respective measurements of the current real-time current values ​​of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device, and also include: the multiple current sensing circuits have the same current measurement upper limit value and current measurement lower limit value.

[0011] Implementation 3 According to the picture data intelligent replacement system shown in Embodiment 3 of the present invention, compared with Embodiment 1, it can also include the following components: A real-time display component, connected to the multiple current sensing circuits of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device, respectively, for displaying the current real-time current values ​​of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device in real time; Wherein, the real-time display component is respectively connected to the multiple current sensing circuits of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device, and is used to display the current real-time current values ​​of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device in real time, including: the real-time display component is an LED display array composed of multiple LED display units; And wherein, the real-time display component is respectively connected to multiple current sensing circuits of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device, and is used to display the current real-time current values ​​of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device in real time, including: the real-time display component is an LCD display array composed of multiple LCD display units.

[0012] In addition, in the intelligent replacement system for image data, based on the deep neural network model, the algorithm ID corresponding to a certain set filtering algorithm, the contrast of the geometrically corrected image and the multiple information parameters are intelligently predicted to obtain the image clarity after filtering the geometrically corrected image using the certain filtering algorithm. The number of deep neural learning is proportional to the resolution of the geometrically corrected image, including: using the MATLAB toolbox to simulate the data processing process of intelligently predicting the image clarity after filtering the geometrically corrected image using the certain filtering algorithm based on the deep neural network model according to the algorithm ID corresponding to a certain set filtering algorithm, the contrast of the geometrically corrected image and the multiple information parameters.

[0013] The essential features and current progress of the present invention can be specifically embodied in: 1: A signal prediction device including a grayscale analysis device, a learning processing device and a clarity prediction device is used, wherein the clarity prediction device is connected to the grayscale analysis device and the learning processing device respectively, so as to provide prediction data for the filtering effect of each filtering algorithm on the geometric correction image; 2: Extract multiple information parameters from the geometrically corrected image, the multiple information parameters including the number of pixel rows, pixel column values, maximum noise amplitudes corresponding to various noise types, and grayscale values ​​corresponding to each pixel point of the geometrically corrected image, and intelligently predict the image clarity obtained after filtering the geometrically corrected image using the filtering algorithm based on a deep neural network model according to an algorithm ID corresponding to a certain set filtering algorithm, the contrast of the geometrically corrected image, and the multiple information parameters; 3: Performing multiple learning on the deep neural network to obtain a deep neural network model, wherein the number of deep neural network learning is proportional to the resolution of the geometrically corrected image; 4: The filtering algorithm corresponding to the highest picture clarity among the pictures corresponding to each filtering algorithm is used as the target filtering algorithm. The target filtering algorithm is used to perform filtering processing on the geometrically corrected image, and the filtered image replaces the corresponding real-time live picture and is sent to the remote live server, thereby enhancing the clarity of each frame of the live picture.

[0014] As described herein, one of ordinary skill in the art will recognize that various changes and modifications can be made to the present invention without departing from the broader aspects of the invention.

Claims

1. A picture data intelligent replacement system, characterized in that: The system comprises: A data receiving device is arranged on the live broadcast client, and is used to send a frame of real-time live broadcast picture to the neighborhood interpolation device before the live broadcast client transmits each frame of real-time live broadcast picture; A neighborhood interpolation device, connected to the data receiving device, for performing Lanczos interpolation processing of a 4-pixel×4-pixel neighborhood on the received real-time live picture to obtain and output a corresponding neighborhood interpolation image; An artifact removal device, connected to the neighborhood interpolation device, for performing an artifact removal process on the received neighborhood interpolation image to obtain and output a corresponding artifact removed image; a geometric correction device, connected to the artifact removal device, for performing geometric correction processing on the received artifact removal image to obtain and output a corresponding geometric correction image; A signal prediction device, connected to the geometric correction device and comprising a grayscale analysis device, a learning processing device and a clarity prediction device, wherein the clarity prediction device is respectively connected to the grayscale analysis device and the learning processing device, and the signal prediction device is used to extract multiple information parameters from the geometric correction image, wherein the multiple information parameters include the number of pixel rows, pixel column values, respective maximum noise amplitudes corresponding to various noise types, and respective grayscale values ​​corresponding to each pixel point of the geometric correction image, perform multiple learning on the deep neural network to obtain a deep neural network model, and intelligently predict the picture clarity obtained after filtering the geometric correction image using the certain filtering algorithm based on the algorithm ID corresponding to a certain set filtering algorithm, the contrast of the geometric correction image, and the multiple information parameters of the deep neural network model, wherein the number of deep neural learning times is proportional to the resolution of the geometric correction image; A content screening device connected to the signal pre-judgment device, used to use the filtering algorithm corresponding to the highest picture definition among the picture definitions corresponding to the various filtering algorithms as the target filtering algorithm; The picture replacement device is connected to the content screening device and is used to perform filtering processing on the geometrically corrected image using a target filtering algorithm, and replace the frame of real-time live broadcast picture with the filtered image and send it to a remote live broadcast server.

2. The picture data intelligent replacement system as claimed in claim 1, characterized in that: The target filtering algorithm is used to perform filtering processing on the geometrically corrected image, and the filtered image replaces the frame of real-time live broadcast picture and sends it to a remote live broadcast server, including: the remote live broadcast server is a cloud computing server, a blockchain server or a big data server.

3. The picture data intelligent replacement system as claimed in claim 2, characterized in that: The system further comprises: A current sensor device is connected to the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device, respectively, and is used to measure the current real-time current values ​​of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device respectively; Among them, the current sensing device is respectively connected to the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device, and is used to respectively measure the current real-time current values ​​of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device. The current sensing device includes multiple current measurement units, which are used to be respectively connected to the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device to complete the respective measurements of the current real-time current values ​​of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device.

4. The picture data intelligent replacement system as claimed in claim 3, characterized in that: The current sensing device includes a plurality of current measuring units, which are used to be respectively connected to the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device to complete the respective measurement of the current real-time current values ​​of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device, including: the plurality of current measuring units are a plurality of current sensing circuits, which are used to be respectively connected to the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device to complete the respective measurement of the current real-time current values ​​of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device.

5. The picture data intelligent replacement system as claimed in claim 4, characterized in that: The multiple current measurement units are multiple current sensing circuits, which are used to be connected to the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device respectively, so as to complete the respective measurements of the current real-time current values ​​of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device, including: the structures of the multiple current sensing circuits are the same.

6. The picture data intelligent replacement system as claimed in claim 5, characterized in that: The multiple current measurement units are multiple current sensing circuits, which are used to be connected to the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device respectively to complete the separate measurement of the current real-time current values ​​of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device. It also includes: the multiple current sensing circuits have the same current measurement upper limit value and current measurement lower limit value.

7. The picture data intelligent replacement system according to any one of claims 3 to 6, characterized in that: The system further comprises: A real-time display component is respectively connected to multiple current sensing circuits of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device, and is used to display the current real-time current values ​​of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device in real time.

8. The picture data intelligent replacement system as claimed in claim 7, characterized in that: A real-time display component is respectively connected to multiple current sensing circuits of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device, and is used to display the current real-time current values ​​of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device in real time. The real-time display component includes an LED display array composed of multiple LED display units.

9. The picture data intelligent replacement system according to claim 7, characterized in that: A real-time display component is respectively connected to multiple current sensing circuits of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device, and is used to display the current real-time current values ​​of the neighborhood interpolation device, the artifact removal device, the geometric correction device and the signal prediction device in real time. The real-time display component includes an LCD display array composed of multiple LCD display units.

Citation Information

Patent Citations

  • Content matching-based on-hour analysis platform

    CN115565230A

  • Joint image processing optimization strategy selection system

    CN116385308A

  • Image super-resolution reconstruction method based on SAUGAN network

    CN116739899A

  • Data enrichment processing technology and method for decoding x-ray medical image

    KR1020180123810A