Lightweight video fusion system for multiple devices

By using a lightweight video fusion system, video streams are processed using H.246 and FPGA algorithms, and a lightweight fusion model is constructed. This solves the problem of traditional video fusion relying on high manpower and computing resources, and achieves efficient and low-resource video fusion results.

CN119052433AInactive Publication Date: 2024-11-29TIANJIN TIANKAI GUANGDONG ARTIFICIAL INTELLIGENCE CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411558722.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-04
Publication Date
2024-11-29
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Traditional video fusion processes rely on human labor, have high requirements for computing resources and storage, and are not suitable for mobile devices or performance-constrained environments.

Method used

Design a lightweight video fusion system, including a management center, a video acquisition module, a video processing module, a video fusion module, and a video feedback module. The system uses H.246 encoding algorithm, FPGA algorithm, and deep learning technology to acquire, process, and fuse video streams, build a lightweight fusion model, and perform image quality scoring and performance evaluation.

Benefits of technology

It improves the clarity and contrast of single-frame images in the video stream, optimizes the efficiency of the fusion process, generates high-quality fused video, and reduces computational and storage requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119052433A_ABST
    Figure CN119052433A_ABST
Patent Text Reader

Abstract

The invention discloses a lightweight video fusion system for multiple devices, which relates to the field of image fusion and comprises a management center in communication connection with a video acquisition module, a video processing module, a video fusion module and a video feedback module. The video acquisition module is used for acquiring a video file in a target video terminal to obtain a corresponding video stream; the video processing module is used for performing image processing on the collected video stream to obtain a corresponding enhanced image, and constructing a video image set based on the obtained enhanced image; the video fusion module is used for fusing the enhanced images in the video image set to obtain a corresponding fused image, and obtaining a corresponding fused video based on the fused image; the video feedback module is used for carrying out performance evaluation on the obtained fused video and taking corresponding feedback measures according to an evaluation result; the method has the advantages of light weight, high efficiency, compatibility, high-quality fusion and the like.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image fusion, and in particular to a lightweight video fusion system for multiple devices. Background Art

[0002] With the advancement of science and technology and the rapid development of multimedia technology, the application of video fusion technology among multiple devices has become more and more extensive; video fusion technology mainly refers to the fusion of content from multiple video sources into a single video stream to achieve a richer visual experience and information display; this technology has important application value in video conferencing, security monitoring, film and television production, virtual reality and other fields; Compared with existing technologies, traditional video fusion processes are mostly performed manually, which is a waste of manpower and has delays. It also requires high computing resources and storage space and is not friendly to mobile devices or performance-constrained environments. These are the problems we need to solve, and for this reason we provide a lightweight video fusion system for multiple devices. Summary of the invention

[0003] The object of the present invention is to provide a lightweight video fusion system for multiple devices; The object of the present invention can be achieved by the following technical solutions: A lightweight video fusion system for multiple devices, comprising a management center, wherein the management center is communicatively connected with a video acquisition module, a video processing module, a video fusion module and a video feedback module; The video acquisition module is used to acquire video files in the target video terminal to obtain corresponding video streams; The video processing module is used to perform image processing on the collected video stream to obtain a corresponding enhanced image, and to construct a video image set based on the obtained enhanced image; The video fusion module is used to fuse the enhanced images in the video image set to obtain a corresponding fused image, and obtain a corresponding fused video based on the fused image; The video feedback module is used to evaluate the performance of the obtained fusion video and take corresponding feedback measures according to the evaluation results.

[0004] Furthermore, the video acquisition module acquires the video files in the target video terminal, and the process of obtaining the corresponding video stream includes: The video acquisition module is provided with a plurality of capture terminals, through which the video files in the corresponding video terminals are captured to obtain corresponding video signals; After the acquisition is completed, the acquired video signal is encoded based on the H.246 encoding algorithm to obtain the corresponding video stream, which is then transmitted to the management center for storage.

[0005] Furthermore, the video processing module performs image processing on the collected video stream to obtain a corresponding enhanced image, which includes: Read the stored video stream, decode the stored video stream based on the H.246 encoding algorithm, and obtain a number of single-frame images corresponding to the corresponding video stream and a timestamp corresponding to the corresponding single-frame image; The obtained single-frame image is subjected to image preprocessing. After the preprocessing is completed, a sliding window is constructed based on the FPGA algorithm. The pixels in the corresponding single-frame image are slid according to the constructed sliding window, and mean filtering is performed. The original grayscale values ​​of the pixels in the corresponding single-frame image are replaced based on the mean filtering result to obtain the corresponding enhanced image.

[0006] Furthermore, the process of constructing a video image set based on the obtained enhanced image includes: Obtain all enhanced images and corresponding timestamps corresponding to the corresponding video stream, construct several video image subsets based on the obtained enhanced images, obtain the timestamps of the enhanced images corresponding to other video streams, and store the enhanced images corresponding to other video streams in the corresponding video image subsets based on the obtained timestamps; summarize the obtained video image subsets to obtain the corresponding video image set.

[0007] Furthermore, the video fusion module fuses the enhanced images in the video image set to obtain a corresponding fused image, including: Acquire a standard image and read an enhanced image in a corresponding video image subset; Obtaining an image score corresponding to a corresponding enhanced image based on an image quality score formula; selecting an enhanced image with the highest image quality score as a base image, and other enhanced images in the corresponding video image subset as images to be fused; Performing feature extraction on the obtained basic image to obtain corresponding basic image features and basic coordinates corresponding to the corresponding basic image features; Based on the obtained basic coordinates, obtain the world coordinates corresponding to the corresponding basic features in the world coordinate system; The same method is used to obtain the fused image features corresponding to the corresponding images to be fused and the fused coordinates and world coordinates corresponding to the corresponding fused image features; The world coordinates corresponding to the corresponding features in the corresponding base image and the image to be fused are read, and the coordinates are matched, and the feature points in the same world coordinates are fused to obtain the corresponding fused image.

[0008] Furthermore, the process of obtaining the corresponding fused video based on the fused image includes: Obtain fused images corresponding to all video image subsets; prioritize the obtained fused images according to corresponding timestamps to obtain a corresponding fused image set; Construct a lightweight fusion model; input the obtained fused image set into the constructed lightweight image to obtain a corresponding multi-source fused video.

[0009] Furthermore, the process of the video feedback module for evaluating the performance of the obtained fusion video includes: Read the captured video stream and mark it as the original video; Extracting spatial information from the original video and the fused video based on a sobel gradient operator, wherein the spatial information includes spatial intensity and spatial direction; Acquire the relative spatial information between the corresponding original information and the relative spatial information according to the obtained spatial information, wherein the relative spatial information includes relative spatial intensity and relative spatial direction; Then, based on the obtained relative spatial intensity and spatial direction, a spatial similarity coefficient of the corresponding fused video relative to the original video is obtained; The same method is used to obtain the time similarity coefficient of the corresponding fused video relative to the original video; the obtained spatial similarity coefficient and time similarity coefficient are used to obtain the fusion performance evaluation of the corresponding fused video for the single frame image corresponding to the original video; Furthermore, based on the fusion performance evaluation of the obtained single-frame image, a video fusion score corresponding to the corresponding fused video is obtained.

[0010] Furthermore, the process of taking corresponding feedback measures based on the evaluation results includes: Set a scoring threshold and compare the obtained video fusion score with the corresponding scoring threshold; If the video fusion score is lower than the score threshold, the single-frame image corresponding to the corresponding fused video is used as the basic image, and the single-frame image in the acquired video stream is used as the image to be fused to obtain a new fused video and evaluate whether it meets the requirements; If the video fusion score is not lower than the score threshold, it means that the corresponding fused video meets the requirements, and it will be transferred to the designated device or storage directory for storage according to user needs.

[0011] Compared with the prior art, the beneficial effects of the present invention are: by collecting data from multiple video terminals and performing frame-by-frame enhancement and frame-by-frame fusion of the collected video streams; it is beneficial to enhance the clarity and contrast of single-frame images in the video stream. At the same time, by constructing a lightweight fusion model, the fused images can be fused to generate high-quality fused videos, which not only improves the accuracy of the fusion, but also optimizes the efficiency of the fusion process. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a schematic diagram of the present invention. DETAILED DESCRIPTION

[0013] like Figure 1 As shown, a lightweight video fusion system for multiple devices includes a management center, and the management center is communicatively connected to a video acquisition module, a video processing module, a video fusion module, and a video feedback module; The video acquisition module is used to acquire video files in the target video terminal to obtain corresponding video streams; It should be further explained that, in a specific implementation process, the video acquisition module acquires the video file in the target video terminal, and the process of obtaining the corresponding video stream includes: The video acquisition module is provided with a plurality of capture terminals, which are connected to the video terminal through a network interface, and collect the video files in the corresponding video terminal through the network interface to obtain the corresponding video signal; the video terminal includes camera terminals and mobile terminals of different brands and models; After the acquisition is completed, the acquired video signal is encoded based on the H.246 encoding algorithm to obtain the corresponding video stream, which is then transmitted to the management center for storage; It should be further explained that the H.246 encoding algorithm includes five parts: frame segmentation, motion estimation and compensation, prediction in the spatial domain and the temporal domain, transformation and quantization, and entropy coding; it is used to compress each video frame in the collected video signal to reduce the amount of data, while also ensuring the transmission of the corresponding video stream on different networks and reducing the packet loss rate.

[0014] The video processing module is used to perform image processing on the collected video stream to obtain a corresponding video image set; It should be further explained that, in a specific implementation process, the video processing module performs image processing on the collected video stream to obtain a corresponding video image set, which includes: Read the stored video stream, decode the stored video stream based on the H.246 encoding algorithm, and obtain a number of single-frame images corresponding to the corresponding video stream and a timestamp corresponding to the corresponding single-frame image; it should be further explained that the decoding based on the H.246 encoding algorithm is mainly used to parse the collected video stream and convert it into a single-frame image of a unified format according to a preset standard, and the preset standard includes a unified image resolution, video frame rate and other related video formats; The collected video streams are numbered as i, where i=1, 2, ..., n1, n1>0 and n1 is an integer, and n1 represents the total number of collected video streams; The single-frame images in the corresponding video stream i are numbered as j, where j=1, 2, ..., n2, n2>0 and n2 is an integer, and the corresponding single-frame image is marked as Dij; Performing image preprocessing on the obtained single-frame image Dij, wherein the image preprocessing includes image denoising and image super-resolution; the image denoising refers to reducing or removing the noise in the corresponding video frame image based on a Gaussian filtering algorithm; the image super-resolution refers to increasing the number of pixels in the corresponding video frame image based on an interpolation method to improve the image resolution; After the preprocessing is completed, a two-dimensional rectangular coordinate system is constructed with the pixel point at the lower left corner of the corresponding single-frame image Dij as the coordinate origin, and the corresponding single-frame image Dij is mapped to the constructed two-dimensional rectangular coordinate system; the coordinates of the pixel points in the corresponding single-frame image Dij can be marked as (u, v); wherein u and v are both integers greater than 0; A sliding window is constructed based on an FPGA algorithm, wherein the sliding window is a 3×3 matrix; a pixel point (u, v) is used as the central element of the sliding window, and adjacent pixel points are other elements in the sliding window; Based on the constructed sliding window, the pixels in the corresponding single-frame image Dij are slid and mean filtered; the mathematical calculation formula of the corresponding mean filter is as follows: ;

[0015] Where f() represents the gray value corresponding to the pixel coordinates () in the corresponding single-frame image Dij; H(u, v) represents the gray value corresponding to the pixel (u, v) after mean filtering; Based on the sliding window, all the pixels in the corresponding single-frame image Dij are traversed, and the original grayscale value of the corresponding pixel is replaced according to the obtained H (u, v). After the replacement is completed, the corresponding enhanced image Zij is obtained; Taking a certain video stream as an example, all enhanced images and corresponding timestamps corresponding to the corresponding video stream are obtained, and J video image subsets are constructed based on them. The timestamps of the enhanced images corresponding to other corresponding video streams are obtained, and the enhanced images corresponding to other video streams are stored in the corresponding video image subsets according to the obtained timestamps; all the obtained subsets are summarized to obtain the corresponding video image set; wherein J=j.

[0016] The video fusion module is used to perform video fusion according to the obtained video image set to obtain a corresponding fused video; It should be further explained that, in a specific implementation process, the video fusion module performs video fusion according to the obtained video image set, and the process of obtaining the corresponding fused video includes: Taking any video image subset as an example, read the enhanced image in the corresponding video image subset; obtain a reference image based on big data technology and mark it as a standard image; The image score corresponding to the enhanced image is obtained based on the image quality score formula, and the obtained image quality score is marked as Q (M, N); the corresponding image score quality formula is as follows: ; Where, M represents the enhanced image, M∈ video image subset; N represents the standard image; μ M , μ N Represent the grayscale averages of the enhanced image and the standard image respectively; σ M , σ N Represent the grayscale standard deviation of the enhanced image and the standard image, σ MN represents the grayscale covariance between the enhanced image and the standard image; The enhanced image with the highest image quality score is selected as the basic image, and the other enhanced images in the corresponding video image subset are selected as the images to be fused; Performing feature extraction on the obtained basic image to obtain corresponding basic image features and basic coordinates (x, y) corresponding to the corresponding basic image features, wherein the basic coordinates are coordinates of pixel points corresponding to the corresponding basic image features, and the basic image features include scene features and object features; Based on the obtained basic coordinates (x, y), the world coordinates corresponding to the corresponding basic features in the world coordinate system are obtained, and the obtained world coordinates are marked as (X, Y, Z); the conversion formula between the corresponding basic coordinates and the corresponding world coordinates is: ; In the formula, The basic image corresponds to the angle between the photosensitive plate and the side of the video terminal, LX and LY represent the physical length of the corresponding pixel point in the X and Y directions in the world coordinate system, f represents the distance, R and T represent the rotation matrix and translation matrix, respectively. It should be further explained that the , LX, LY are determined by the parameters in the corresponding video terminal according to the practice requirements, and R, T are determined by the parameters outside the corresponding video terminal according to the actual requirements; The same method is used to obtain the fused image features corresponding to the corresponding images to be fused and the fused coordinates and world coordinates corresponding to the corresponding fused image features; Read the world coordinates corresponding to the corresponding features in the corresponding base image and the image to be fused, match the coordinates, fuse the feature points in the same world coordinates, and obtain the corresponding fused image. It should be further explained that, in the specific implementation process, the feature point refers to a feature point where the basic feature in the corresponding basic image and the fused image feature in the corresponding image to be fused are located in the same world coordinates; Using the same method, all the video image subsets are fused to obtain corresponding fused images; the obtained fused images are prioritized according to corresponding timestamps to obtain corresponding fused image sets; Constructing a corresponding lightweight fusion model based on deep learning technology, wherein the lightweight fusion model is used to fuse the input images according to the time before and after to obtain the corresponding fused video; The obtained fused image set is input into the constructed lightweight image to obtain the corresponding multi-source fused video.

[0017] The video feedback module is used to evaluate the obtained multi-source fusion video and take corresponding information feedback measures according to the evaluation results; It should be further explained that, in the specific implementation process, the video feedback module evaluates the obtained multi-source fusion video and takes corresponding information feedback measures according to the evaluation results, including: Read the captured video stream i and mark it as the original video V A ; The obtained fused video is marked as V B ; Based on the sobel gradient operator, the original video V A Extract spatial information, the spatial information includes spatial intensity and spatial direction, and mark the obtained spatial intensity and spatial direction as Q respectively. A and F A ; in, ; ; Where (e1, e2, z) represents the pixel with coordinates (e1, e2) in the z-th frame image of the original video A; Represents the original video V A The convolution result of the z-th frame image in and the horizontal Sobel gradient operator; Represents the original video V A The convolution result of the z-th frame image in the vertical direction with the Sobel gradient operator; where z∈n2, (e1, e2)∈(u, v); The same method is used to obtain the corresponding fusion video V B The corresponding spatial information is marked as Q B and F B ; Obtain the corresponding original information V based on the obtained spatial informationA and V B The relativity between them is marked as the relative spatial intensity Q AB and the relative spatial direction F AB ; in, ; ; Then, the corresponding fused video V is obtained based on the relative spatial intensity and spatial direction obtained. B Relative to the original video V A The spatial similarity coefficient KJ AB , the corresponding mathematical formula is as follows: ; In the formula, a1, a2, a3 and b1, b2, b3 are all constant coefficients, which are determined by actual needs; The same method is used to obtain the corresponding fusion video V B Relative to the original video V A The corresponding time similarity coefficient is marked as TJ AB ; The obtained spatial similarity coefficient and temporal similarity coefficient are used to obtain the corresponding fused video V B For the original video V A The fusion performance evaluation of the corresponding single frame image is marked as PJ AB ; in, ; Where w1 and w2 represent the fused video V B and the original video V A The weight coefficient of is determined by the spatial similarity coefficient and the temporal similarity coefficient; Then, based on the obtained single-frame image fusion performance evaluation, a video fusion score corresponding to the corresponding fused video is obtained; and the score is recorded as PF; in, ; Set a scoring threshold and compare the obtained video fusion score with the corresponding scoring threshold; If the video fusion score is lower than the score threshold, the corresponding fusion video V B The corresponding single-frame image is used as the basic image, and the single-frame image in the acquired video stream is used as the image to be fused to obtain a new fused video and evaluate whether it meets the requirements, and so on; If the video fusion score is not lower than the score threshold, it means that the corresponding fusion video V BIf it meets the requirements, it will be transferred to a designated device or storage directory for storage according to user needs, and feedback will be given to the corresponding user; the user needs include video format, video size, video resolution, etc.

[0018] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A lightweight video fusion system for multiple devices, characterized in that: It includes a management center, which is communicatively connected to a video acquisition module, a video processing module, a video fusion module and a video feedback module; The video acquisition module is used to acquire video files in the target video terminal to obtain corresponding video streams; The video processing module is used to perform image processing on the collected video stream to obtain a corresponding enhanced image, and to construct a video image set based on the obtained enhanced image; The video fusion module is used to fuse the enhanced images in the video image set to obtain a corresponding fused image, and obtain a corresponding fused video based on the fused image; The video feedback module is used to evaluate the performance of the obtained fusion video and take corresponding feedback measures according to the evaluation results.

2. A lightweight video fusion system for multiple devices according to claim 1, characterized in that: The video acquisition module acquires the video files in the target video terminal, and the process of obtaining the corresponding video stream includes: The video acquisition module is provided with a plurality of capture terminals, through which the video files in the corresponding video terminals are captured to obtain corresponding video signals; After the acquisition is completed, the acquired video signal is encoded based on the H.246 encoding algorithm to obtain the corresponding video stream, which is then transmitted to the management center for storage.

3. A lightweight video fusion system for multiple devices according to claim 2, characterized in that: The video processing module performs image processing on the collected video stream to obtain a corresponding enhanced image, which includes: Read the stored video stream, decode the stored video stream based on the H.246 encoding algorithm, and obtain a number of single-frame images corresponding to the corresponding video stream and a timestamp corresponding to the corresponding single-frame image; The obtained single-frame image is subjected to image preprocessing. After the preprocessing is completed, a sliding window is constructed based on the FPGA algorithm. The pixels in the corresponding single-frame image are slid according to the constructed sliding window, and mean filtering is performed. The original grayscale values ​​of the pixels in the corresponding single-frame image are replaced based on the mean filtering result to obtain the corresponding enhanced image.

4. A lightweight video fusion system for multiple devices according to claim 3, characterized in that: The process of constructing a video image set based on the obtained enhanced images includes: Obtain all enhanced images and corresponding timestamps corresponding to the corresponding video stream, construct several video image subsets based on the obtained enhanced images, obtain the timestamps of the enhanced images corresponding to other video streams, and store the enhanced images corresponding to other video streams in the corresponding video image subsets based on the obtained timestamps; summarize the obtained video image subsets to obtain the corresponding video image set.

5. A lightweight video fusion system for multiple devices according to claim 4, characterized in that: The video fusion module fuses the enhanced images in the video image set to obtain a corresponding fused image, which includes: Acquire a standard image and read an enhanced image in a corresponding video image subset; Obtaining an image score corresponding to a corresponding enhanced image based on an image quality score formula; selecting an enhanced image with the highest image quality score as a base image, and other enhanced images in the corresponding video image subset as images to be fused; Performing feature extraction on the obtained basic image to obtain corresponding basic image features and basic coordinates corresponding to the corresponding basic image features; Based on the obtained basic coordinates, obtain the world coordinates corresponding to the corresponding basic features in the world coordinate system; The same method is used to obtain the fused image features corresponding to the corresponding images to be fused and the fused coordinates and world coordinates corresponding to the corresponding fused image features; The world coordinates corresponding to the corresponding features in the corresponding base image and the image to be fused are read, and the coordinates are matched, and the feature points in the same world coordinates are fused to obtain the corresponding fused image.

6. A lightweight video fusion system for multiple devices according to claim 5, characterized in that: The process of obtaining the corresponding fused video based on the fused image includes: Obtain fused images corresponding to all video image subsets; prioritize the obtained fused images according to corresponding timestamps to obtain a corresponding fused image set; Construct a lightweight fusion model; input the obtained fused image set into the constructed lightweight image to obtain a corresponding multi-source fused video.

7. A lightweight video fusion system for multiple devices according to claim 6, characterized in that: The process of the video feedback module for evaluating the performance of the obtained fusion video includes: Read the captured video stream and mark it as the original video; Extracting spatial information from the original video and the fused video based on a sobel gradient operator, wherein the spatial information includes spatial intensity and spatial direction; Acquire the relative spatial information between the corresponding original information and the relative spatial information according to the obtained spatial information, wherein the relative spatial information includes relative spatial intensity and relative spatial direction; Then, based on the obtained relative spatial intensity and spatial direction, a spatial similarity coefficient of the corresponding fused video relative to the original video is obtained; The same method is used to obtain the time similarity coefficient of the corresponding fused video relative to the original video; the obtained spatial similarity coefficient and time similarity coefficient are used to obtain the fusion performance evaluation of the corresponding fused video for the single frame image corresponding to the original video; Furthermore, based on the fusion performance evaluation of the obtained single-frame image, a video fusion score corresponding to the corresponding fused video is obtained.

8. A lightweight video fusion system for multiple devices according to claim 7, characterized in that: The process of taking appropriate feedback measures based on the evaluation results includes: Set a scoring threshold and compare the obtained video fusion score with the corresponding scoring threshold; If the video fusion score is lower than the score threshold, the single-frame image corresponding to the corresponding fused video is used as the basic image, and the single-frame image in the acquired video stream is used as the image to be fused to obtain a new fused video and evaluate whether it meets the requirements; If the video fusion score is not lower than the score threshold, it means that the corresponding fused video meets the requirements, and it will be transferred to the designated device or storage directory for storage according to user needs.

Citation Information

Patent Citations

  • Video image fusion performance evaluation method based on structure similarity and human vision

    CN102231844A

  • Monocular vision odometer method fusing edge features and deep learning

    CN111311666A

  • Six-degree-of-freedom multi-video splicing defogging recognition system for port ship

    CN117114995A

  • Multi-video and three-dimensional scene fusion method and device, equipment and storage medium

    CN117278731A

  • Industrial robot image processing method based on image fusion

    CN118521859A