Historical data-based audio and video real-time fusion method and system

Abstract

The invention discloses an audio and video real-time fusion method and system based on historical data, and the method comprises the steps: obtaining historical audio and video data, and extracting a first feature group and a second feature group from an image frame and an audio sampling frame; associatively storing the evolution characteristics of the audio and video data of the target object in each scene model; associatively storing each target object image and an audio processing algorithm; identifying and selecting at least one target object as a current target object; judging evolution characteristics and confirming a processing algorithm; processing and coding the image and the audio based on the processing algorithm to form a video segment; and splicing the video segments to generate a target audio / video. The audio and video fusion system obtains audio and video data characteristics of each target object according to historical data, selects audio and video data of one target object from current audio and video data for processing, determines a processing algorithm based on evolution characteristics in the processing process, realizes automatic processing of audio and video data of different sound sources and image sources, and improves the processing efficiency. The treatment efficiency is obviously improved.

Description

Technical Field

[0001] The present application relates to the technical field of multi-source data processing, and in particular to a method and system for real-time fusion of audio and video based on historical data. Background Art

[0002] In practice, a microphone is usually used to collect audio data, and a camera is used to collect image data. After obtaining the audio frames and image frames, they are processed by audio and image processing to form audio files and video files. The two are arranged and fused according to certain rules to form a common video image with sound.

[0003] Usually, when there is only one audio source and image source, it is only necessary to synchronously collect audio and video data, and after encoding, directly integrate the audio packet queue and video packet queue in time sequence to obtain the target audio and video. However, when there are multiple audio sources and image sources, it is necessary to manually process the collected audio data and image data one by one, such as associating each video image with the sound, and finally through a specific arrangement sequence, so that the sound can correspond to the video image, and finally generate a video image with sound that can switch with the video lens.

[0004] Obviously, when there are many audio or image sources, the efficiency of manual audio and video arrangement is very low, and it is impossible to achieve the effect of real-time processing and fusion of audio and video. Summary of the invention

[0005] In order to solve the problem that multi-source audio and video fusion processing is inefficient and cannot be processed in real time in practical applications, the first purpose of this application is to provide a real-time audio and video fusion method based on historical data, which uses historical big data to accurately determine the relationship between different audio data and image data, automatically fuses the related audio and video, and can automatically switch images and sounds according to the switching of actual scenes without manual intervention. It is highly efficient and can process audio and video data in real time. In order to realize the above method, the second purpose of this application is to propose a real-time audio and video fusion system based on historical data, and the third purpose is to propose protection of a computer-readable storage medium. The specific scheme is as follows: A method for real-time fusion of audio and video based on historical data, comprising: S100, acquiring historical audio and video data of each target object, extracting a first feature group and a second feature group for identifying the image and audio of each target object from the image frame and the audio sampling frame, respectively, and storing the two in association; S200, establishing and storing the evolution characteristics of the target object itself and the audio and video data between the target objects in each scene model, and storing the target object name in association with the above-mentioned evolution characteristics and scene model; S300, establishing and associating storage of image and audio processing algorithms for each target object in each scene model; S400, acquiring a plurality of audio and video data at a current moment, identifying and selecting at least one target object from the audio and video data as a current target object; S500, determining a scene model according to the evolution characteristics of the audio and video data of the current target object, and determining an image and audio processing algorithm according to the scene model; S600, generating image frames and audio sampling frames from the audio and video data of the current target object, processing the above images and audio based on the processing algorithm, and then encoding and fusing them to form a video segment; S700, taking the video segment as the starting segment or splicing it with the video segment formed at the previous moment to generate target audio and video, and repeating steps S400-S700; Wherein, the first feature group is a combination of specific image points in the image frame; The second feature group is a combination of specific sampling values ​​in the audio sampling frame; The evolution characteristics include the change trend of the audio and video data of the current target object itself before the transition from the current target object to the next target object, and the correlation trend between the audio and video data of the current target object and other target objects.

[0006] By adopting the above technical scheme, according to the characteristics of the audio and video data of each target object, the audio and video data of a target object is automatically selected from multiple audio and video data for processing, and the evolution characteristics of the current audio and video data are identified and confirmed during the processing, thereby determining the scene model and related processing algorithms, realizing automatic processing of images and audio, and the received audio and video data can be processed continuously with high efficiency.

[0007] Furthermore, establishing and associating and storing the image and audio processing algorithms of each target object in each scene model also includes: Storing rendering rules and background audio of each target object image frame in each scene model; Store the importance ranking of each target object in each scene model; The image and audio processing algorithms include: Render the image frame of the target object according to the set rendering rules; The audio of the target object is filtered and enhanced, and the audio of the remaining target objects is filtered and attenuated based on the importance ranking of each target object, or background audio is added to the audio of the target object according to the scene model.

[0008] Further, identifying and selecting at least one target object from the audio and video data as the current target object includes: Configure a processing priority for each target object and store it in association with the name of each target object; Simultaneously searching for matching first feature groups and second feature groups from image frames and audio sampling frames of a plurality of audio and video data to confirm a target object contained in the current audio and video data; Analyze the audio and video data corresponding to each target object to generate the duration of the audio and video; Select target objects whose audio and video duration exceeds a set value, sort the target objects based on the processing priority of each target object, and form a queue to be processed; At least one target object with a high ranking is selected from the queue to be processed as the current target object.

[0009] By adopting the above technical solution, a target object can be selected for processing based on the duration of the audio and video of each target object and the pre-set processing priority, avoiding the frequent switching of later images and audio due to the short duration of audio and video, thereby ensuring the playback quality of the later target video.

[0010] Further, identifying and selecting at least one target object from the audio and video data as the current target object also includes: Analyze the audio and video data corresponding to each target object, identify the position coordinates of the first feature group from each image frame of the target object, generate a movement trajectory of the first feature group, determine the speed and direction of the movement trajectory, and if the movement trajectory exceeds a set interval, delete the corresponding target object from the queue to be processed; and / or The audio and video data corresponding to each target object are analyzed, and the intensity and amplitude of the target object audio are determined from each audio sampling frame of the target object. If the above intensity and amplitude are lower than the set value, the corresponding target object is deleted from the queue to be processed.

[0011] By adopting the above technical solution, it is possible to avoid the occurrence of blurred image quality and unclear sound in the target audio and video generated later, thereby ensuring the quality of the generated target audio and video.

[0012] Further, determining the evolution characteristics of the audio and video data of the current target object includes: Extract multiple image frames and audio sampling frames of the target object within a set time period before the current moment; Identifying and storing position coordinates of a first feature group from a plurality of image frames, generating a motion trajectory of a target object based on the position coordinates, and determining evolution features of audio and video data of the target object according to the motion trajectory; and / or A second feature group is identified from a plurality of audio sampling frames and stored, an audio change trend of a target object is generated based on the second feature group, and an evolution feature of audio and video data of the target object is determined according to the audio change trend.

[0013] Furthermore, the target object includes a person or object that can make sound and has a visible appearance in the audio or video recording scene; The scene model includes an audio and video recording scene of multiple image sources and audio sources.

[0014] In a second aspect, the present application provides a real-time audio and video fusion system based on historical data, comprising: A data acquisition unit configured to acquire historical audio and video data of each target object and audio and video data to be processed; A target object marking unit is configured to extract a first feature group and a second feature group for identifying each target object image and audio from an image frame and an audio sampling frame respectively based on historical audio and video data, and to mark the target object in the current audio and video data based on the first feature group and the second feature group; A data storage unit configured to store the target object itself in each scene model and the evolution characteristics of the audio and video data between the target objects, store the image and audio processing algorithms of each target object in each scene model, and store the generated video segments; a data processing unit configured to identify and select at least one target object from current audio and video data as a current target object, determine and confirm a scene model based on evolution characteristics of the audio and video data of the current target object and determine an image and audio processing algorithm based on the scene model, process the image and audio of the current target object based on the processing algorithm, and then encode and fuse them to form a video segment; A video segment splicing unit is configured to obtain the above video segment, use the video segment as a starting segment or splice it with an already generated video segment, generate a target audio and video, and output it; Wherein, the first feature group is a combination of specific image points in the image frame; The second feature group is a combination of specific sampling values ​​in the audio sampling frame.

[0015] Furthermore, the data processing unit is configured with: The first target object screening module is configured as follows: Analyze the audio and video data of each target object associated with the current audio and video data, generate the duration of the audio and video corresponding to each target object, select the target object whose audio and video duration exceeds the set value, and sort the target objects based on the processing priority of each target object to form a queue to be processed; Selecting at least one target object with a high ranking from the queue to be processed as the current target object; and The second target object screening module is configured as follows: Analyze the audio and video data corresponding to each target object, identify the position coordinates of the first feature group from each image frame of the target object, generate a movement trajectory of the first feature group, determine the speed and direction of the movement trajectory, and if the movement trajectory exceeds a set interval, delete the corresponding target object from the queue to be processed; and / or The audio and video data corresponding to each target object are analyzed, and the intensity and amplitude of the target object audio are determined from each audio sampling frame of the target object. If the above intensity and amplitude are lower than the set value, the corresponding target object is deleted from the queue to be processed.

[0016] Finally, it is also proposed to protect a computer-readable storage medium, in which a program module for implementing the above-mentioned method for real-time fusion of audio and video based on historical data is loaded.

[0017] To summarize, the present application includes at least one of the following beneficial technical effects: the audio and video fusion system obtains the audio and video data characteristics of each target object based on historical data, automatically selects the audio and video data of a target object from multiple audio and video data for processing, and identifies and confirms the evolution characteristics of the current audio and video data during the processing process, thereby determining the scene model and related processing algorithms, and realizing automatic processing of images and audios. Compared with manual processing, the processing efficiency is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is an overall schematic diagram of the audio and video fusion method of the present application; Figure 2 A schematic diagram of a method for selecting a target object from audio and video data; Figure 3 This is a functional block diagram of the system of this application.

[0019] Figure numerals: 1. Data acquisition unit; 2. Target object marking unit; 3. Data storage unit; 4. Data processing unit; 5. Video segment splicing unit. DETAILED DESCRIPTION

[0020] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings.

[0021] In the description of this specification, the description with reference to the terms "certain embodiments", "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0022] The present application embodiment discloses a method for real-time fusion of audio and video based on historical data, referring to Figure 1 , mainly includes the following steps: S100, acquiring historical audio and video data of each target object, extracting a first feature group and a second feature group for identifying each target object from an image frame and an audio sampling frame, respectively, and storing the first feature group and the second feature group in association with the name of the target object; S200, establishing and storing the evolution characteristics of the target object itself and the audio and video data between the target objects in each scene model, and storing the target object name in association with the above-mentioned evolution characteristics and scene model; S300, establishing and associating storage of image and audio processing algorithms for each target object in each scene model; S400, acquiring a plurality of audio and video data at a current moment, identifying and selecting at least one target object from the audio and video data as a current target object; S500, determining and confirming a scene model according to the evolution characteristics of the audio and video data of the current target object and determining an image and audio processing algorithm according to the scene model; S600, generating image frames and audio sampling frames from the audio and video data of the current target object, processing the above images and audio based on the processing algorithm, and then encoding and fusing them to form a video segment; S700, taking the video segment as the starting segment or splicing it with the video segment formed at the previous moment to generate target audio and video, and repeating steps S400-S700.

[0023] In step S100, the target object includes but is not limited to a person or object that can make a sound and has a visible appearance in the audio and video recording scene. The first feature group is a combination of specific image points in the image frame, such as a combination of specific pixel points. The above combination can accurately reflect the target object to which the current image frame belongs. The second feature group is a combination of specific sampling values ​​in the audio sampling frame, such as speech frequency features, voiceprint features, etc. Based on the above second feature group, the target object to which it belongs can be determined from the sound. By storing the above first feature group and the second feature group, the target object contained in the current audio and video data can be obtained.

[0024] In step S200, the evolutionary features include the image and audio of the current target object transitioning from the current moment to the next moment, the change trend of the current target object's own audio and video data, or the correlation trend between the current target object and the audio and video data of other target objects before the current target object transitions to the next target object. For example, the change trend of the target object sound amplitude or the change trend of the target object action, the above correlation trend includes the coordinate position relationship between different target objects in the image, the proportional relationship of frequency or amplitude between different target objects in the audio, etc. In specific practice, the above target object image or action change trend can be obtained by analyzing the data of the position coordinates of the first feature group in the set number of frame images before and after, and the change trend of the target object audio can be obtained by analyzing the second feature group.

[0025] In step S200, the scene model refers to the audio and video recording environment in which each target object is located. In the embodiment of the present application, it specifically refers to a scene with multiple image sources and audio sources, such as a video conference scene, a variety show recording scene, etc. Since each target object in different scene models has different image and audio features, including the aforementioned evolution features, the aforementioned evolution features and scene models can be associated and stored, and the scene model in which the current target object is located can be inferred from the evolution features.

[0026] In step S300, image and audio processing algorithms for each target object in each scene model are established and stored in association, and the details include: S310, storing the rendering rules and background audio of each target object image frame in each scene model. By pre-storing the image rendering rules of different target objects in different scene models and the background audio that needs to be fused, the algorithm matching time required for the later audio and video data processing can be greatly shortened.

[0027] S320, stores the importance ranking of each target object in each scene model. The above ranking refers to the processing priority of each target object in different scene models. For example, when there are three target objects in the same scene model, the above three target objects can be ranked to determine the priority of audio and video data processing. The audio and video data of an important target object is processed first, and the audio and video data of other target objects are processed slowly or deleted directly.

[0028] In the implementation manner of the present application, the image and audio processing algorithm includes: S330, rendering the image frame of the target object according to the set rendering rules, such as adjusting the image color rendering; S340, filtering and enhancing the audio of the target object, and filtering and attenuating the audio of the remaining target objects based on the importance ranking of each target object, or adding background audio to the audio of the target object according to the scene model. The audio filtering and attenuation processing in the above steps includes filtering out high-frequency and low-frequency components in the audio, reducing the size of the audio data, and reducing the amplitude of the sound.

[0029] Through the above step S300, different adjustments can be made to the images and audios of different target objects, so that the target audio and video layers synthesized later are clearer and the images and sounds of the target objects are highlighted.

[0030] In step S400, at least one target object is identified and selected from the audio and video data as the current target object. Figure 2 As shown, further comprising: S410, configuring a processing priority for each target object and storing it in association with the name of each target object; S420, searching for matching first feature group and second feature group from image frames and audio sampling frames of a plurality of audio and video data, and confirming a target object included in the current audio and video data; S430, analyzing the audio and video data corresponding to each target object to generate the duration of the audio and video; S440, selecting target objects whose audio and video duration exceeds a set value, and sorting the target objects based on the processing priority of each target object to form a queue to be processed; S450: Select at least one target object with a high ranking from the queue to be processed as the current target object.

[0031] The above technical solution can select a target object for processing according to the duration of the audio and video of each target object and the pre-set processing priority, avoiding the frequent switching of later images and audio due to the short duration of audio and video, and ensuring the playback quality of the later target video.

[0032] In order to ensure the quality of the target audio and video generated later, in the implementation mode of the present application, identifying and selecting at least one target object from the audio and video data as the current target object also includes: S460, analyzing the audio and video data corresponding to each target object, identifying the position coordinates of the first feature group from each image frame of the target object, generating a moving trajectory of the first feature group, determining the speed and direction of the moving trajectory, and if the moving trajectory exceeds a set interval, deleting the corresponding target object from the queue to be processed; and / or S470, analyzing the audio and video data corresponding to each target object, determining the intensity and amplitude of the target object audio from each audio sampling frame of the target object, and if the intensity and amplitude are lower than a set value, deleting the corresponding target object from the queue to be processed.

[0033] In a specific implementation, the position coordinates of the first feature group refer to the pixel coordinates of each image point in the image frame.

[0034] In step S500, determining the evolution characteristics of the current target object audio and video data further includes: S510, extracting multiple image frames and audio sampling frames of the target object within a set time period before the current moment; S521, identifying and storing position coordinates of a first feature group from a plurality of image frames, generating a motion trajectory of the target object based on the position coordinates, and determining evolution features of audio and video data of the target object according to the motion trajectory; and / or S522, identifying and storing a second feature group from a plurality of audio sampling frames, generating an audio change trend of the target object based on the second feature group, and determining an evolution feature of the audio and video data of the target object according to the audio change trend.

[0035] This application provides a real-time audio and video fusion system based on historical data, such as Figure 3 As shown, it mainly includes: a data acquisition unit 1, a target object marking unit 2, a data storage unit 3, a data processing unit 4 and a video segment splicing unit 5.

[0036] The data acquisition unit 1 is configured to acquire historical data and audio and video data to be processed.

[0037] The target object marking unit 2 is configured to extract the first feature group and the second feature group for identifying each target object from the image frame and the audio sampling frame based on the historical audio and video data, and to mark the target object in the current audio and video data based on the first feature group and the second feature group, that is, to configure an identity identification reference for each target object. In practice, the target object marking unit 2 includes an image recognition module and an audio recognition module, which are respectively used to identify specific graphics or pixel locations in image frames, and speech features in audio, such as voiceprint features, speech frequency features, etc.

[0038] The data storage unit 3 is data-connected to the data acquisition unit 1 and the target object marking unit 2, and is configured to store historical audio and video data of each target object, associate and store the evolution characteristics of the target object itself in each scene model and the audio and video data between each target object, associate and store the image and audio processing algorithms of each target object in each scene model, associate and store each target object and its corresponding first feature group and second feature group data, and store the generated video segments.

[0039] The data processing unit 4 is configured to identify and select at least one target object from the current audio and video data as the current target object, determine and based on the evolution characteristics of the audio and video data of the current target object, confirm the scene model and determine the image and audio processing algorithm based on the scene model, process the image and audio of the current target object based on the processing algorithm, and then encode and fuse them to form the video segment to be spliced.

[0040] The video segment splicing unit 5 is data-connected to the data processing unit 4, and is configured to obtain the video segment generated in the data processing unit 4, and then use the video segment as the starting segment or splice it with the stored video segment to generate the target audio and video and output it.

[0041] In order to delete the audio and video data of secondary target objects from the audio and video data to be processed, improve the processing efficiency while ensuring the quality of the target audio and video generated later, the above-mentioned data processing unit is configured with a first target object screening module and a second target object screening module.

[0042] The first target object screening module is configured to: analyze the audio and video data of each target object associated with the current audio and video data, generate the duration of the audio and video corresponding to each target object, select the target object whose audio and video duration exceeds the set value, and sort the target objects based on the processing priority of each target object to form a queue to be processed; Then, at least one target object with a high ranking is selected from the queue to be processed as the current target object.

[0043] The second target object screening module is configured to analyze the audio and video data corresponding to each target object, identify the position coordinates of the first feature group from each image frame of the target object, generate a moving trajectory of the first feature group, determine the speed and direction of the moving trajectory, and if the moving trajectory exceeds a set interval, delete the corresponding target object from the queue to be processed; and / or The audio and video data corresponding to each target object are analyzed, and the intensity and amplitude of the target object audio are determined from each audio sampling frame of the target object. If the above intensity and amplitude are lower than the set value, the corresponding target object is deleted from the queue to be processed.

[0044] The present application also provides a computer-readable storage medium, which stores a computer program for executing the audio and video real-time fusion method based on historical data described in any of the above embodiments. The above computer-readable storage medium includes but is not limited to a disk storage, a CD-ROM, an optical storage, etc.

[0045] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A real-time fusion method of audio and video based on historical data, characterized in that: include: S100, acquiring historical audio and video data of each target object, extracting a first feature group and a second feature group for identifying the image and audio of each target object from the image frame and the audio sampling frame, respectively, and storing the two in association; S200, establishing and storing the evolution characteristics of the target object itself and the audio and video data between the target objects in each scene model, and storing the target object name in association with the above-mentioned evolution characteristics and scene model; S300, establishing and associating storage of image and audio processing algorithms for each target object in each scene model; S400, acquiring a plurality of audio and video data at a current moment, identifying and selecting at least one target object from the audio and video data as a current target object; S500, determining and confirming a scene model according to the evolution characteristics of the audio and video data of the current target object and determining an image and audio processing algorithm according to the scene model; S600, generating image frames and audio sampling frames from the audio and video data of the current target object, processing the above images and audio based on the processing algorithm, and then encoding and fusing them to form a video segment; S700, taking the video segment as the starting segment or splicing it with the video segment formed at the previous moment to generate target audio and video, and repeating steps S400-S700; Wherein, the first feature group is a combination of specific image points in the image frame; The second feature group is a combination of specific sampling values ​​in the audio sampling frame; The evolution characteristics include the change trend of the audio and video data of the current target object itself before the transition from the current target object to the next target object, and the correlation trend between the audio and video data of the current target object and other target objects.

2. The method for real-time fusion of audio and video based on historical data according to claim 1, characterized in that: Establishing and associating storage of image and audio processing algorithms for each target object in each scene model, including: Storing rendering rules and background audio of each target object image frame in each scene model; Store the importance ranking of each target object in each scene model; The image and audio processing algorithms include: Render the image frame of the target object according to the set rendering rules; The audio of the target object is filtered and enhanced, and the audio of the remaining target objects is filtered and attenuated based on the importance ranking of each target object, or background audio is added to the audio of the target object according to the scene model.

3. The method for real-time fusion of audio and video based on historical data according to claim 1, characterized in that: Identifying and selecting at least one target object from the audio and video data as the current target object includes: Configure a processing priority for each target object and store it in association with the name of each target object; Simultaneously searching for matching first feature groups and second feature groups from image frames and audio sampling frames of a plurality of audio and video data to confirm a target object contained in the current audio and video data; Analyze the audio and video data corresponding to each target object to generate the duration of the audio and video; Select target objects whose audio and video duration exceeds a set value, sort the target objects based on the processing priority of each target object, and form a queue to be processed; At least one target object with a high ranking is selected from the queue to be processed as the current target object.

4. The method for real-time fusion of audio and video based on historical data according to claim 2, characterized in that: Identifying and selecting at least one target object from the audio and video data as the current target object also includes: Analyze the audio and video data corresponding to each target object, identify the position coordinates of the first feature group from each image frame of the target object, generate a movement trajectory of the first feature group, determine the speed and direction of the movement trajectory, and if the movement trajectory exceeds a set interval, delete the corresponding target object from the queue to be processed; and / or The audio and video data corresponding to each target object are analyzed, and the intensity and amplitude of the target object audio are determined from each audio sampling frame of the target object. If the above intensity and amplitude are lower than the set value, the corresponding target object is deleted from the queue to be processed.

5. The method for real-time fusion of audio and video based on historical data according to claim 1, characterized in that: Determine the evolution characteristics of the audio and video data of the current target object, including: Extract multiple image frames and audio sampling frames of the target object within a set time period before the current moment; Identifying and storing position coordinates of a first feature group from a plurality of image frames, generating a motion trajectory of a target object based on the position coordinates, and determining evolution features of audio and video data of the target object according to the motion trajectory; and / or A second feature group is identified from a plurality of audio sampling frames and stored, an audio change trend of a target object is generated based on the second feature group, and an evolution feature of audio and video data of the target object is determined according to the audio change trend.

6. The method for real-time fusion of audio and video based on historical data according to claim 1, characterized in that: The target object includes a person or object that can make sound and has a visible appearance in the audio or video recording scene.

7. The method for real-time fusion of audio and video based on historical data according to claim 1, characterized in that: The scene model includes an audio and video recording scene of multiple image sources and audio sources.

8. A real-time audio and video fusion system based on historical data, characterized in that: include: A data acquisition unit (1) is configured to acquire historical audio and video data of each target object and audio and video data to be processed; The target object marking unit (2) is configured to extract a first feature group and a second feature group for identifying each target object image and audio from the image frame and the audio sampling frame respectively based on the historical audio and video data, and to mark the identity of the target object in the current audio and video data based on the first feature group and the second feature group; A data storage unit (3) is configured to store historical audio and video data of each target object, associate and store the evolution characteristics of the audio and video data of the target object itself and between the target objects in each scene model, associate and store the image and audio processing algorithms of each target object in each scene model, and store the generated video segments; The data processing unit (4) is configured to identify and select at least one target object from the current audio and video data as the current target object, determine and confirm the scene model based on the evolution characteristics of the audio and video data of the current target object, and determine the image and audio processing algorithm based on the scene model, process the image and audio of the current target object based on the processing algorithm, and then encode and fuse them to form a video segment; A video segment splicing unit (5) is configured to obtain the above video segment, use the video segment as a starting segment or splice it with an already generated video segment, generate a target audio and video, and output it; Wherein, the first feature group is a combination of specific image points in the image frame; The second feature group is a combination of specific sampling values ​​in the audio sampling frame.

9. The real-time audio and video fusion system based on historical data according to claim 8 is characterized in that: The data processing unit is configured with: The first target object screening module is configured as follows: Analyze the audio and video data of each target object associated with the current audio and video data, generate the duration of the audio and video corresponding to each target object, select the target object whose audio and video duration exceeds the set value, and sort the target objects based on the processing priority of each target object to form a queue to be processed; Select at least one target object with a high ranking from the queue to be processed as the current target object; as well as The second target object screening module is configured as follows: Analyze the audio and video data corresponding to each target object, identify the position coordinates of the first feature group from each image frame of the target object, generate a movement trajectory of the first feature group, determine the speed and direction of the movement trajectory, and if the movement trajectory exceeds a set interval, delete the corresponding target object from the queue to be processed; and / or The audio and video data corresponding to each target object are analyzed, and the intensity and amplitude of the target object audio are determined from each audio sampling frame of the target object. If the above intensity and amplitude are lower than the set value, the corresponding target object is deleted from the queue to be processed.

10. A computer-readable storage medium, characterized in that: A program module is loaded therein for implementing the real-time fusion method of audio and video based on historical data as described in any one of claims 1-7.

Images