Video fusion method and device, storage medium and electronic device

Abstract

The invention discloses a video fusion method and device, a storage medium and an electronic device, and relates to the technical field of smart home, the video fusion method comprises the steps that N videos collected by N image collection devices are acquired, the N image collection areas are located at different positions of a target area, the image collection angles are different, and N is a positive integer; n is an integer greater than or equal to 2; and carrying out fusion processing on the N videos to obtain a target video. By adopting the technical scheme, the problem that the same event cannot be visually observed through a plurality of cameras is solved.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and more specifically, to a video fusion method and device, a storage medium, and an electronic device. Background Art

[0002] When multiple cameras in an area detect an event in the area, they will each record a video, generating continuous or discontinuous audio and video recording files. The image captured by the camera is the orthogonal projection of the scene in the three-dimensional world onto the two-dimensional screen. Due to the principle that things are larger when they are closer and smaller when they are farther away, the image will be distorted. The image will be different due to different internal parameters of the camera.

[0003] When users want to review the events in the video afterwards, the traditional method is to view the videos of each camera separately. In this way, after watching the video of the first camera, they can watch the video of the second camera. For example, after Passerby A leaves the field of view of Camera A, he appears in the field of view of Camera B. This is an event. In this event, Passerby A first appears in Camera A and then in Camera B. In terms of space, the field of view of Camera B is to the right of Camera A. In this case, according to the existing solution, when viewing the video, the user will first watch the video of Camera 1 to see the event of Passerby A, and then watch the video of Camera 2. This not only leads to a poor user experience, but also is out of order in time and fragmented in space, making it not intuitive and convenient to observe the development process of the event.

[0004] With regard to the problem in related technologies that the same event cannot be visually observed through multiple cameras, no effective solution has been proposed so far.

[0005] Therefore, it is necessary to improve the related technology to overcome the above-mentioned defects in the related technology. Summary of the invention

[0006] The embodiments of the present application provide a video fusion method and device, a storage medium, and an electronic device to at least solve the problem that the same event cannot be intuitively observed through multiple cameras.

[0007] According to one aspect of an embodiment of the present application, a video fusion method is provided, comprising: acquiring N videos captured by N image capture devices, wherein the N image capture areas are located at different positions of a target area and have different image capture angles, and N is an integer greater than or equal to 2; performing fusion processing on the N videos to obtain a target video.

[0008] In an exemplary embodiment, the N videos are fused to obtain a target video, including: in a case where the N videos include video frames at M moments, determining a set of video frames corresponding to the M moments from the N videos, wherein the video frame set includes M subsets, the ith subset of the M subsets includes Z video frames at the ith moment of the M moments in the N videos, where Z is an integer greater than or equal to 1 and less than or equal to N, M is an integer greater than or equal to 2, and i is a positive integer less than or equal to M; determining M video frames based on the video frame set, wherein the ith video frame of the M video frames is a video frame obtained by fusion based on the Z video frames in the ith subset; and obtaining the target video based on the M video frames.

[0009] In an exemplary embodiment, M video frames are determined according to the video frame set, including: obtaining the i-th video frame among the M video frames in the following manner to obtain the M video frames: when the i-th subset includes only one video frame, determining the video frame in the i-th subset as the i-th video frame; when the i-th subset includes multiple video frames, obtaining Z groups of parameters corresponding to Z video frames in the i-th subset, wherein the j-th parameter in the Z group of parameters is parameter information of an image acquisition device that shoots the j-th video frame among the Z video frames; constructing a three-dimensional image in a three-dimensional coordinate system according to the Z video frames in the i-th subset and the Z group of parameters to obtain a target three-dimensional image, wherein the target three-dimensional image has all the elements in the Z video frames; and converting the target three-dimensional image into a two-dimensional image to obtain the i-th video frame.

[0010] In an exemplary embodiment, obtaining the target video according to the M video frames includes: determining target encoding parameters, wherein the target encoding parameters include: frame rate, bit rate, and resolution; and obtaining the target video according to the M video frames based on the target encoding parameters.

[0011] In an exemplary embodiment, the target video is obtained according to the M video frames, including: in a case where there are multiple audios in the N videos, the target video is obtained according to the M video frames and the multiple audios; wherein, in a case where the playback time period of a target audio among the multiple audios in the N videos is a target time period, the playback time period of the target audio in the target video is the target time period.

[0012] In an exemplary embodiment, when the target audio is the audio corresponding to the left picture of the target video, the target audio in the target video is the left channel audio; when the target audio is the audio corresponding to the right picture of the target video, the target audio in the target video is the right channel audio.

[0013] In an exemplary embodiment, when the target audio is the audio corresponding to the left picture of the target video, the target video displays subtitles corresponding to the target audio on the left side of the target video during the time period when the target video is playing the target audio; when the target audio is the audio corresponding to the right picture of the target video, the target video displays subtitles corresponding to the target audio on the right side of the target video during the time period when the target video is playing the target audio.

[0014] According to another aspect of an embodiment of the present application, a video fusion device is also provided, including: an acquisition module, used to acquire N videos captured by N image acquisition devices, wherein the N image acquisition areas are located at different positions of a target area and the image acquisition angles are different, and N is an integer greater than or equal to 2; a fusion module, used to perform fusion processing on the N videos to obtain a target video.

[0015] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned video fusion method when running.

[0016] According to another aspect of the embodiments of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the video fusion method through the computer program.

[0017] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, and the computer program is executed by a processor to implement the above-mentioned video fusion method.

[0018] In the present application, a target video is obtained by fusing multiple videos captured by multiple image acquisition devices. Since the multiple image acquisition devices are located at different positions and capture images at different angles, the target video obtained after the fusion process can more completely and intuitively display the events in the target area, thereby solving the problem of being unable to intuitively observe the same event through multiple cameras. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0021] Figure 1 is a schematic diagram of a hardware environment of a video fusion method according to an embodiment of the present application;

[0022] Figure 2 is a flow chart of a video fusion method according to an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of an optional video fusion according to an embodiment of the present application;

[0024] Figure 4 is a structural block diagram of a video fusion device according to an embodiment of the present application;

[0025] Figure 5 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] According to one aspect of the embodiments of the present application, a video fusion method is provided. The video fusion method is widely used in smart home (Smart Home), smart home, smart home equipment ecology, smart house (IntelligenceHouse) ecology and other whole-house intelligent digital control application scenarios. Optionally, in this embodiment, the above-mentioned video fusion method can be applied to Figure 1 In the hardware environment composed of the terminal device 102 and the server 104 shown in FIG. Figure 1 As shown, the server 104 is connected to the terminal device 102 via a network, and can be used to provide services (such as application services, etc.) for the terminal or a client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data computing services for the server 104.

[0029] The network may include but is not limited to at least one of the following: wired network, wireless network. The wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal device 102 may be but is not limited to a PC, a mobile phone, a tablet computer, a smart air conditioner, a smart range hood, a smart refrigerator, a smart oven, a smart stove, a smart washing machine, a smart water heater, a smart washing device, a smart dishwasher, a smart projection device, a smart TV, a smart clothes drying rack, a smart curtain, a smart audio and video, a smart socket, a smart speaker, a smart fresh air device, a smart kitchen and bathroom device, a smart bathroom device, a smart sweeping robot, a smart window cleaning robot, a smart mopping robot, a smart air purification device, a smart steamer, a smart microwave oven, a smart kitchen treasure, a smart purifier, a smart water dispenser, a smart door lock, etc.

[0030] In order to solve the above problems, a video fusion method is provided in this embodiment, including but not limited to being applied in the whole-house intelligent digital control application scenario. The execution subject may be a video fusion system (which may be located in the above terminal device 102 or the server 104). Figure 2 is a flow chart of a video fusion method according to an embodiment of the present application, the process includes the following steps S202-S206:

[0031] Step S202: Acquire N videos captured by N image capture devices, wherein the N image capture areas are located at different positions of the target area and have different image capture angles, and N is an integer greater than or equal to 2;

[0032] Optionally, the image acquisition device includes but is not limited to a surveillance camera, a network camera, an infrared camera, and a drone camera.

[0033] Optionally, in large target areas that require comprehensive coverage, such as shopping malls, airports, or residential areas, or on smart devices (such as smart door locks), multiple image acquisition devices (cameras) are usually deployed, and each image acquisition device is responsible for acquiring images from different positions and angles.

[0034] Optionally, N video acquisition devices capture N videos, which means that the system can efficiently and in parallel collect video streams from multiple sources, each of which covers a portion of the target area, but is interrelated to each other and together constitutes a panoramic view of the target area. Considering that each camera may have different internal parameters (such as focal length, image sensor size, etc.), the images it captures will produce different degrees of deformation and perspective differences when projected onto a two-dimensional screen. Therefore, when acquiring these video streams, the parameter information of each camera is also recorded for subsequent accurate fusion processing.

[0035] It should be noted that step S202 can ensure the comprehensiveness and flexibility of the monitoring system. Even in a complex and changing environment, all details in the target area can be captured through video streams from multiple angles, providing rich and accurate raw data for subsequent fusion processing. In addition, through parallel acquisition, the blind spot problem that may exist in a single camera can be reduced, and the overall coverage and efficiency of the monitoring system can be improved.

[0036] Step S204: performing fusion processing on the N videos to obtain a target video.

[0037] Optionally, N video streams from different positions and angles are fused to synchronize them in time and coordinate them in space, thereby generating a single target video with a unified perspective. Fusion processing usually includes two parts: time synchronization and spatial stitching. Time synchronization requires the identification and adjustment of the timestamp of each frame in the video stream to ensure that events in different video streams can be presented in the target video in the order in which they actually occurred. Spatial stitching involves converting the two-dimensional images taken by different cameras into a unified three-dimensional coordinate system for alignment and stitching according to their relative positions in three-dimensional space and camera parameters, and finally achieving a seamless and continuous spatial perspective in the target video to restore the original scene of the event.

[0038] It should be noted that step S204 significantly improves the intuitiveness and convenience of event observation. Through the synchronous fusion of time and space, when viewing the target video, the user can obtain a coherent development context of the event without switching between multiple videos. Especially when the event spans multiple camera fields of view, this fusion processing can more accurately track and understand the full picture of the event, greatly optimizing the user experience, and also providing strong technical support for event analysis and security monitoring.

[0039] It should be noted that the above steps, through intelligent time and space synchronous fusion processing, realize a comprehensive, coherent and real observation experience of events, which greatly improves the application value and efficiency of the monitoring system in complex scenarios.

[0040] In the above steps, a target video is obtained by fusing multiple videos captured by multiple image acquisition devices. Since the multiple image acquisition devices are located at different positions and capture images at different angles, the target video obtained after the fusion process can more completely and intuitively display the events in the target area, thereby solving the problem of being unable to intuitively observe the same event through multiple cameras.

[0041] In an exemplary embodiment, the N videos are fused to obtain a target video, which can be achieved by the following steps S11-S13:

[0042] Step S11: in a case where the N videos include video frames at M moments, determine a set of video frames corresponding to the M moments from the N videos, wherein the set of video frames includes M subsets, the i-th subset of the M subsets includes Z video frames at the i-th moment among the M moments in the N videos, Z is an integer greater than or equal to 1 and less than or equal to N, M is an integer greater than or equal to 2, and i is a positive integer less than or equal to M;

[0043] Optionally, when the system faces N video streams, each of which contains M video frames at specific moments, the first task is to identify and locate these video frames to ensure the accuracy of subsequent fusion processing. "Video frames at M moments" means that there are M key points on the timeline, and each point has Z video frames from N cameras. The system needs to accurately capture and record the frame information of each camera at these moments to form M subsets, each of which corresponds to one of the M moments and contains all the video frames captured by all cameras at that moment.

[0044] It should be noted that Z can be less than N, which means that at a certain moment, not all video frames of all cameras may be used for fusion. This may be due to the fact that some cameras fail to capture related events at that moment, there are blind spots, or the captured image quality is poor. This step ensures that a complete video can be fused even when some video frames are missing.

[0045] It should be noted that, through step S11, the system can efficiently locate and filter out all camera video frames at M key moments, laying a solid foundation for subsequent fusion processing. This precise time point positioning and video frame screening ensures the pertinence and effectiveness of fusion processing, avoids unnecessary data processing, saves computing resources, and improves the generation efficiency and quality of fused videos.

[0046] Step S12: determining M video frames according to the video frame set, wherein the i-th video frame in the M video frames is a video frame obtained by fusing the Z video frames in the i-th subset;

[0047] Optionally, step S12 fuses Z video frames at the same time point (any time i among the M time points) in the N video streams to generate a video frame that integrates multiple perspective information. Fusion processing can adopt a variety of technical means, such as time synchronization based on timestamps, spatial stitching based on camera internal and external parameters, reprojection and stitching based on image processing algorithms, image fusion based on deep learning, etc. During the fusion process, the system needs to solve the inconsistency problems between video frames, such as color differences, uneven exposure, perspective misalignment, etc., to ensure that the generated video frames are optimal in image quality and information integrity.

[0048] It should be noted that, by implementing step S12, the system can generate a series of fused video frames, each of which integrates the information of multiple cameras at the same slice time point, achieving seamless connection in perspective and precise synchronization in time. This fusion processing not only optimizes the video quality and enhances the comprehensiveness and accuracy of information, but also provides users with a more intuitive and complete event observation perspective, greatly improving the practical value and user experience of surveillance videos.

[0049] Step S13: Obtain the target video according to the M video frames.

[0050] Optionally, the system encodes the M fused video frames obtained in step S12 in chronological order to synthesize the target video. The synthesis process involves further video compression, format conversion and other processing to ensure that the generated video file can be smoothly played on various playback devices and network environments, while also taking into account the efficiency of storage and transmission.

[0051] It should be noted that step S13 realizes the conversion from the fused video frame to the complete video file, which is convenient for users to replay and analyze events at any time and any place. The generated target video is not only highly synchronized and coordinated in time and space, but also reaches high-quality standards in visual effects, so that users can obtain a smoother and more realistic event playback experience, facilitate in-depth analysis and judgment of events, and enhance the practicality of the system.

[0052] It should be noted that the above steps effectively solve the problem of unintuitive and incoherent event playback in multi-camera monitoring. Through precise time point positioning, video frame screening, multi-view information fusion and high-quality video synthesis, the system can generate a highly synchronized target video with a unified perspective, which significantly improves the intuitiveness of event observation and user experience. At the same time, it also technically realizes efficient data processing and high-quality image fusion, providing strong technical support for intelligent monitoring, security prevention, event analysis and other fields.

[0053] In an exemplary embodiment, determining the M video frames according to the video frame set may be implemented by the following steps: obtaining the i-th video frame among the M video frames by the following steps S21-S24 to obtain the M video frames:

[0054] Step S21: when the i-th subset includes only one video frame, determining the video frame in the i-th subset as the i-th video frame;

[0055] Optionally, when the i-th subset contains only one video frame, it means that at a certain point in time, only one camera captured the relevant scene information. In this case, the system directly uses the video frame as the i-th video frame without fusion processing. In this case, the video frame of a single camera becomes the most direct record of that moment and is directly included in the construction process of the target video.

[0056] It should be noted that this step simplifies the processing flow and avoids unnecessary calculations, especially when some cameras fail or no relevant scenes are captured, which can ensure the continuity and integrity of the video stream.

[0057] Step S22: when the i-th subset includes multiple video frames, obtaining Z groups of parameters corresponding to Z video frames in the i-th subset, wherein the j-th parameter in the Z group of parameters is parameter information of an image acquisition device that shoots the j-th video frame in the Z video frames;

[0058] Optionally, when the i-th subset contains multiple video frames, the system will obtain camera parameter information related to these video frames, including but not limited to: intrinsic parameters (such as focal length, image sensor size, etc.) and extrinsic parameters (such as camera position, pointing direction, etc.) of the image acquisition device (such as camera). These parameter information are crucial for accurately reconstructing the scene in three-dimensional space, and can help the system understand the real position and direction of each video frame in the three-dimensional world, so as to complete the accurate splicing and fusion of video frames in subsequent steps.

[0059] It should be noted that, through step S22, the system can obtain the relative positions and parameters of multiple cameras in space, providing key data for the subsequent construction of a three-dimensional image, ensuring the accuracy of spatial synchronization and the quality of video fusion.

[0060] Step S23: constructing a three-dimensional image in a three-dimensional coordinate system according to the Z video frames in the i-th subset and the Z groups of parameters to obtain a target three-dimensional image, wherein the target three-dimensional image has all the elements in the Z video frames;

[0061] Optionally, each video frame is reprojected in three-dimensional space according to parameter information of the image acquisition device, and then the reprojected image elements are aligned and spliced ​​in a three-dimensional coordinate system, and finally a target three-dimensional image containing all video frame elements is generated. The target three-dimensional image is a three-dimensional reconstruction of the real scene of the monitored area at a certain point in time, which can better restore the original spatial relationship and details of the event.

[0062] It should be noted that step S23 realizes the spatial synchronization and information integration of multi-view video frames. The generated target three-dimensional image is not only visually closer to the real scene, but also more information-rich and coherent, which is of great value for the restoration and analysis of events.

[0063] Step S24: converting the target three-dimensional image into a two-dimensional image to obtain the i-th video frame.

[0064] Optionally, after the target three-dimensional image is generated, step S24 converts it into a two-dimensional image, i.e., the i-th video frame. This process can be implemented in a variety of ways, such as orthogonal projection, perspective projection, etc., to adapt to different display requirements and viewing angles. The generation of a two-dimensional image is the final output form of the fusion process, which is convenient for playing and playback on various display devices, and is also the basic unit of the target video.

[0065] It should be noted that, through step S24, the three-dimensional reconstructed scene is converted into a two-dimensional image, which not only retains the integrity of the three-dimensional information but also facilitates playback in any standard player. At the same time, this process also facilitates the post-editing and sharing of the video.

[0066] It should be noted that through the above steps, not only can the video frames of a single perspective be processed, but also the multi-perspective information can be accurately integrated through 3D reconstruction technology to generate a highly synchronized target video with a unified perspective. This method significantly improves the practical value of surveillance videos, provides a more intuitive and realistic perspective for event observation, and technically achieves efficient processing and high-quality fusion of video information. It is suitable for multiple fields such as intelligent monitoring, security prevention, and event analysis, greatly optimizes the user experience, and improves the accuracy and efficiency of event analysis.

[0067] In an exemplary embodiment, obtaining the target video according to the M video frames can be achieved by following the steps S31-S32:

[0068] Step S31: determining target encoding parameters, wherein the target encoding parameters include: frame rate, bit rate, and resolution;

[0069] It should be noted that the frame rate is the number of images displayed per second, usually measured in frames per second (FPS). A higher frame rate can provide a smoother video playback experience, but it will also increase file size and bandwidth requirements;

[0070] Bit rate: determines the data flow speed and file size after video encoding. The higher the bit rate, the greater the burden of video data storage and transmission, but the video quality is also improved accordingly;

[0071] Resolution: refers to the clarity of the video, usually expressed in pixels of width x height (such as 1080x720 or 1920x1080). High-resolution video can provide more details, but it also means a larger file size and higher storage / transmission costs.

[0072] It should be noted that when determining the target encoding parameters, it is necessary to comprehensively consider multiple factors such as the target video playback device, network environment, storage space, video quality requirements, etc., in order to achieve the best balance between video experience and resource management.

[0073] It should be noted that, through step S31, the system can customize and select the most appropriate encoding parameters according to specific application requirements and resource constraints, ensuring that the generated target video can maintain high-quality playback effects while taking into account storage and transmission efficiency. This flexibility in parameter selection greatly improves the practicality of the target video and enables it to adapt to different scenarios and requirements.

[0074] Step S32: Based on the target encoding parameters, obtain the target video according to the M video frames.

[0075] Optionally, based on the target encoding parameters determined in step S31, the system encodes the fused M video frames in step S32 to generate a continuous video stream, i.e., the target video. The encoding process can adopt a variety of video encoding standards, such as H.264, H.265 (HEVC), VP9, ​​etc. Each standard has its own specific compression efficiency and compatibility characteristics. The purpose of encoding is to convert video frames into a format that is easy to store, transmit, and play, while minimizing information loss.

[0076] It should be noted that step S32 realizes the conversion of video frames to video streams, and the generated target video not only meets the user's needs in terms of quality, but is also optimized in terms of storage and network transmission. The encoded target video is easy to play on various devices and can be efficiently distributed through the network, which greatly enhances the accessibility and application scope of the video.

[0077] It should be noted that the above steps select the best encoding parameters according to specific application requirements, ensuring the balance between quality and resource management of the target video, and enhancing the availability and performance of the video. The generated target video can be played in a professional monitoring system or shared on various devices such as mobile phones and computers through the network, greatly expanding the use scenarios and scope of the video, significantly improving the practicality and user experience of multi-camera surveillance recordings in event observation and analysis, and also reflecting the innovation and efficiency of the technology.

[0078] In an exemplary embodiment, obtaining the target video based on the M video frames can be achieved through the following steps: in a case where there are multiple audios in the N videos, obtaining the target video based on the M video frames and the multiple audios; wherein, in a case where the playback time period of a target audio among the multiple audios in the N videos is a target time period, the playback time period of the target audio in the target video is the target time period.

[0079] It should be noted that not only the video frames need to be fused, but the audio data also needs to be considered to ensure that the generated target video maintains a high degree of synchronization and coherence in audio and video information.

[0080] Optionally, first, the system identifies the correspondence between video frames and audio clips by analyzing the timestamps of video frames and audio, ensuring the temporal consistency of audio and video information. Then, in the presence of multiple audios, the system will determine a "target audio" based on certain rules (such as audio quality, relevance to the event, etc.), and the target audio will be synchronized to the target video as the main source of event audio information. Next, ensure that the playback time period of the target audio is consistent with the time period of the video frame, so that even when multiple cameras capture the same event from different angles, the audio and video information can be fully matched on the timeline to provide a true event playback effect.

[0081] It should be noted that the synchronous presentation of audio and video information allows users to have a more intuitive and immersive experience when viewing multi-angle surveillance videos, which improves the practical value of surveillance videos and user satisfaction. For scenes that require event analysis and investigation, the target video with audio and video synchronization can provide more complete and accurate information, which helps to conduct in-depth analysis and judgment of the event and improves the efficiency of security prevention and event handling.

[0082] In an exemplary embodiment, when the target audio is the audio corresponding to the left picture of the target video, the target audio in the target video is the left channel audio; when the target audio is the audio corresponding to the right picture of the target video, the target audio in the target video is the right channel audio.

[0083] Optionally, the system first determines the cameras and audio sources corresponding to the left and right images of the target video (based on the camera's layout and pointing angle, as well as the association information between the audio source and the camera). Then, based on the division of the video perspective, when the system sets the audio source corresponding to the left image as the target audio, it encodes the target audio as the left channel audio; similarly, when the audio source corresponding to the right image is set as the target audio, it encodes the target audio as the right channel audio. This channel configuration ensures that the audio information is consistent with the perspective of the video image, enhancing the naturalness and immersion of the audio and video fusion.

[0084] It should be noted that the selection and configuration of the audio channels make the audio playback respond to the changes in the viewing angle of the video screen. When the user observes the video screen moving from left to right, the audio will naturally transition from the left channel to the right channel, creating an immersive audio and video experience like being in the scene, improving the viewing experience of the surveillance video and the accuracy of information transmission.

[0085] In an exemplary embodiment, when the target audio is the audio corresponding to the left picture of the target video, the target video displays subtitles corresponding to the target audio on the left side of the target video during the time period when the target video is playing the target audio; when the target audio is the audio corresponding to the right picture of the target video, the target video displays subtitles corresponding to the target audio on the right side of the target video during the time period when the target video is playing the target audio.

[0086] Optionally, the system first determines whether the target audio is associated with the left or right picture according to the left and right picture division of the target video. According to the positioning of the target audio, the system will display subtitles related to the audio on the corresponding side (left or right) of the target video during the time period of playing the target audio. This configuration ensures that when the user hears the audio on one side, he can immediately see the text information related to it, which enhances the intuitiveness and accuracy of information transmission.

[0087] It should be noted that by associating audio with subtitle positions, users can more easily identify sounds from different positions in the video and quickly understand the semantics of audio information through subtitles, which is particularly important for event analysis in fast-paced or complex scenes, and significantly improves the efficiency of information understanding. Moreover, this mechanism of matching audio with subtitle positions can be flexibly applied to various monitoring scenarios, whether indoors or outdoors, whether it is a single event or a complex situation with multiple events concurrently, it can provide clear information presentation to meet the needs of different users.

[0088] In an exemplary embodiment, after obtaining the target video, the method further includes the following steps S41-S43:

[0089] Step S41: obtaining device parameters and network status corresponding to a target device, wherein the target device is a device for playing a target video;

[0090] Optionally, device parameters include, but are not limited to: screen resolution, processor performance (processor model, number of cores, frequency), memory size, display capability (such as color depth), audio decoding capability, storage space, operating system version, etc.

[0091] Optionally, the network conditions include but are not limited to: network bandwidth, delay, packet loss rate, connection stability, etc.

[0092] Step S42: adjusting the playback parameters of the target video according to the device parameters and network conditions corresponding to the target device;

[0093] Optionally, playback parameters include but are not limited to: video bit rate, resolution, frame rate, audio bit rate, buffering strategy, video encoding format (such as converting from H.265 to H.264 to adapt to device decoding capabilities), color configuration, audio output configuration (such as converting stereo to mono, or optimizing surround sound effects).

[0094] Step S43: based on the playback parameters of the target video, play the target video on the target device.

[0095] It should be noted that, through the above steps, the device parameters and network status of the target device are obtained, and the video playback parameters are intelligently adjusted, which not only greatly improves the user experience of watching videos on different devices, but also enhances the system's compatibility, resource utilization efficiency and security, realizes the intelligence and personalization of surveillance video playback, and provides users with a smoother, clearer and safer viewing experience.

[0096] In an exemplary embodiment, after obtaining the target video, the method further includes the following steps: performing noise reduction processing on the target video based on an audio noise reduction processing algorithm.

[0097] Optionally, after acquiring the target video, the system first analyzes the audio stream in the video to identify background noise (such as wind, distant conversations, equipment operation sounds, etc.). Next, it uses advanced audio noise reduction algorithms, such as spectral noise reduction, adaptive filtering, deep learning noise reduction models, etc., to process the audio stream to reduce or eliminate these background noises.

[0098] Optionally, the noise-reduced audio stream is recombined with the video stream to form an optimized target video. The system can also further adjust audio parameters according to the audio output capability of the target device to ensure that the user has the best audio experience on the target device.

[0099] It should be noted that through the above steps, background noise is removed and the audio clarity is significantly improved, which not only improves the user experience, but also enhances the understanding and communication of video content, providing strong support for event analysis, data processing and security.

[0100] Obviously, the embodiments described above are only some embodiments of the present application, not all embodiments. In order to better understand the above method, the above process is described below in conjunction with the embodiments, but it is not intended to limit the technical solutions of the embodiments of the present application. Optionally:

[0101] Figure 3 The following is an example of an optional overall process of video fusion:

[0102] 1. In the video acquisition stage, the start time of a certain video segment must be recorded first;

[0103] 2. When playing the video, parse the frame time mark from the video file and superimpose the video start time, so as to match the video frame with the time when it occurs. In this way, multiple videos are parsed frame by frame, and multiple videos are extracted frame by frame according to the time progress;

[0104] 3. According to the camera layout and camera imaging parameters, the two-dimensional images of each camera are restored to the three-dimensional world coordinate system, and then re-imaged to achieve spatial stitching and restoration of the image;

[0105] 4. Render the processed image;

[0106] 5. Re-encode the images after time and space synchronization into videos for users to download or share on WeChat, so that users can also get the synchronous presentation experience using a general player.

[0107] It should be noted that the present application also has the following advantages: users do not need to view the videos of each camera separately, but can view related videos on the same screen simultaneously, and the time occurrence in the video is synchronized, the spatial position is restored, and the experience is more intuitive.

[0108] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0109] In this embodiment, a video fusion device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0110] Figure 4 : is a structural block diagram of a video fusion device according to an embodiment of the present application, the device comprising:

[0111] An acquisition module 42 is used to acquire N videos acquired by N image acquisition devices, wherein the N image acquisition areas are located at different positions of the target area and have different image acquisition angles, and N is an integer greater than or equal to 2;

[0112] The fusion module 44 is used to perform fusion processing on the N videos to obtain a target video.

[0113] The above-mentioned device obtains a target video by fusing multiple videos collected by multiple image acquisition devices. Since the multiple image acquisition devices are located at different positions and collect data at different angles, the target video obtained after the fusion process can more completely and intuitively display the events in the target area, thereby solving the problem of being unable to intuitively observe the same event through multiple cameras.

[0114] In an exemplary embodiment, the fusion module 44 is also used to determine a set of video frames corresponding to the M moments from the N videos when the N videos include video frames at M moments, wherein the video frame set includes M subsets, the i-th subset of the M subsets includes Z video frames at the i-th moment in the N videos at the M moments, Z is an integer greater than or equal to 1 and less than or equal to N, M is an integer greater than or equal to 2, and i is a positive integer less than or equal to M; determine M video frames based on the video frame set, wherein the i-th video frame of the M video frames is a video frame obtained by fusion of the Z video frames in the i-th subset; and obtain the target video based on the M video frames.

[0115] In an exemplary embodiment, the fusion module 44 is also used to obtain the i-th video frame among the M video frames in the following manner to obtain the M video frames: when the i-th subset includes only one video frame, determine the video frame in the i-th subset as the i-th video frame; when the i-th subset includes multiple video frames, obtain Z groups of parameters corresponding to Z video frames in the i-th subset, wherein the j-th parameter in the Z group of parameters is parameter information of an image acquisition device that shoots the j-th video frame among the Z video frames; construct a three-dimensional image in a three-dimensional coordinate system according to the Z video frames in the i-th subset and the Z group of parameters to obtain a target three-dimensional image, wherein the target three-dimensional image has all the elements in the Z video frames; convert the target three-dimensional image into a two-dimensional image to obtain the i-th video frame.

[0116] In an exemplary embodiment, the fusion module 44 is further used to determine target encoding parameters, wherein the target encoding parameters include: frame rate, bit rate, and resolution; based on the target encoding parameters, the target video is obtained according to the M video frames.

[0117] In an exemplary embodiment, the fusion module 44 is also used to obtain the target video according to the M video frames and the multiple audios when there are multiple audios in the N videos; wherein, when the playback time period of the target audio among the multiple audios in the N videos is the target time period, the playback time period of the target audio in the target video is the target time period.

[0118] In an exemplary embodiment, when the target audio is the audio corresponding to the left picture of the target video, the target audio in the target video is the left channel audio; when the target audio is the audio corresponding to the right picture of the target video, the target audio in the target video is the right channel audio.

[0119] In an exemplary embodiment, when the target audio is the audio corresponding to the left picture of the target video, the target video displays subtitles corresponding to the target audio on the left side of the target video during the time period when the target video is playing the target audio; when the target audio is the audio corresponding to the right picture of the target video, the target video displays subtitles corresponding to the target audio on the right side of the target video during the time period when the target video is playing the target audio.

[0120] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0121] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0122] S1, obtaining N videos captured by N image acquisition devices, wherein the N image acquisition areas are located at different positions of the target area and have different image acquisition angles, and N is an integer greater than or equal to 2;

[0123] S2, performing fusion processing on the N videos to obtain a target video.

[0124] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0125] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0126] The embodiment of the present application also provides an electronic device, such as Figure 5 As shown, the electronic device includes a memory 502 and a processor 504. The memory 502 stores a computer program, and the processor 504 is configured to execute the steps in any of the above method embodiments through the computer program.

[0127] Optionally, in this embodiment, the processor 504 may be configured to perform the following steps through a computer program:

[0128] S1, obtaining N videos captured by N image acquisition devices, wherein the N image acquisition areas are located at different positions of the target area and have different image acquisition angles, and N is an integer greater than or equal to 2;

[0129] S2, performing fusion processing on the N videos to obtain a target video.

[0130] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0131] Alternatively, a person skilled in the art may understand that: Figure 5 The structure shown is for illustration only. Figure 5 The structure of the electronic device is not limited. For example, the electronic device may also include Figure 5 More or fewer components (such as network interfaces, etc.) as shown in, or with Figure 5 Different configurations are shown.

[0132] Among them, the memory 502 can be used to store software programs and modules, such as the program instructions / modules corresponding to the video fusion method and device in the embodiments of the present application. The processor 504 executes various functional applications and data processing by running the software programs and modules stored in the memory 502, that is, realizing the above-mentioned video fusion method. The memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 502 may further include a memory remotely located relative to the processor 504, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 502 can be specifically, but not limited to, used to store information such as system configuration files. As an example, such as Figure 5As shown, the memory 502 may include, but is not limited to, the acquisition module 42 and the fusion module 44 in the video fusion device. In addition, it may also include, but is not limited to, other module units in the video fusion device (such as the first determination module and the second determination module), which will not be repeated in this example.

[0133] Optionally, the transmission device 506 is used to receive or send data via a network. Specific examples of the network may include wired networks and wireless networks. In one example, the transmission device 506 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 506 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0134] In addition, the electronic device further includes: a display 508; and a connection bus 510, which is used to connect various module components in the electronic device.

[0135] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0136] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0137] The embodiments of the present application also provide a computer program, which includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any one of the above method embodiments.

[0138] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0139] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0140] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A video fusion method, characterized in that: include: Acquire N videos captured by N image acquisition devices, wherein the N image acquisition areas are located at different positions of the target area and have different image acquisition angles, and N is an integer greater than or equal to 2; The N videos are fused to obtain a target video.

2. The method according to claim 1, characterized in that The N videos are fused to obtain a target video, including: In a case where the N videos include video frames at M moments, a video frame set corresponding to the M moments is determined from the N videos, wherein the video frame set includes M subsets, an i-th subset of the M subsets includes Z video frames at the i-th moment among the M moments in the N videos, where Z is an integer greater than or equal to 1 and less than or equal to N, M is an integer greater than or equal to 2, and i is a positive integer less than or equal to M; Determine M video frames according to the video frame set, wherein an i-th video frame among the M video frames is a video frame obtained by fusing Z video frames in the i-th subset; The target video is obtained according to the M video frames.

3. The method according to claim 2, characterized in that Determining M video frames according to the video frame set includes: The i-th video frame among the M video frames is obtained in the following manner to obtain the M video frames: In a case where the i-th subset includes only one video frame, determining the video frame in the i-th subset as the i-th video frame; In the case where the i-th subset includes multiple video frames, obtaining Z groups of parameters corresponding to Z video frames in the i-th subset, wherein the j-th parameter in the Z group of parameters is parameter information of an image acquisition device that shoots the j-th video frame in the Z video frames; constructing a three-dimensional image in a three-dimensional coordinate system according to the Z video frames in the i-th subset and the Z groups of parameters to obtain a target three-dimensional image, wherein the target three-dimensional image has all the elements in the Z video frames; The target three-dimensional image is converted into a two-dimensional image to obtain the i-th video frame.

4. The method according to claim 2, characterized in that: Obtaining the target video according to the M video frames includes: Determine target encoding parameters, wherein the target encoding parameters include: frame rate, bit rate, and resolution; Based on the target encoding parameters, the target video is obtained according to the M video frames.

5. The method according to claim 2, characterized in that: Obtaining the target video according to the M video frames includes: In the case where the N videos have multiple audios, obtaining the target video according to the M video frames and the multiple audios; Wherein, when the playing time period of the target audio among the multiple audios in the N videos is the target time period, the playing time period of the target audio in the target video is the target time period.

6. The method according to claim 5, characterized in that When the target audio is the audio corresponding to the left picture of the target video, the target audio in the target video is the left channel audio; when the target audio is the audio corresponding to the right picture of the target video, the target audio in the target video is the right channel audio.

7. The method according to claim 5, characterized in that In the case where the target audio is the audio corresponding to the left screen of the target video, the target video displays subtitles corresponding to the target audio on the left side of the target video during the time period when the target video is playing the target audio; in the case where the target audio is the audio corresponding to the right screen of the target video, the target video displays subtitles corresponding to the target audio on the right side of the target video during the time period when the target video is playing the target audio.

8. A video fusion device, characterized in that: include: An acquisition module, used to acquire N videos acquired by N image acquisition devices, wherein the N image acquisition areas are located at different positions of the target area and have different image acquisition angles, and N is an integer greater than or equal to 2; The fusion module is used to perform fusion processing on the N videos to obtain a target video.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 7 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 7 through the computer program.

Images