Code stream fusion method, chip, NVR, server, network and storage medium

Abstract

The invention relates to the technical field of image processing, and provides a code stream fusion method, a chip, an NVR, a server, a network and a storage medium. The method comprises the following steps: acquiring an original code stream generated by a plurality of network cameras IPC deployed in a target scene, wherein the original code stream is generated by collecting a scene picture of the target scene; according to the method, original code streams generated by a plurality of IPCs are fused to obtain at least one fused code stream, the original code streams of at least two IPCs are fused in each fused code stream, and then the at least one fused code stream is transmitted to at least one code stream display network video recorder (NVR), so that on the premise that the number of channels of the NVR is not expanded, the number of the channels of the NVR is increased. More code stream data are accessed into the original channel, more scene pictures are displayed, and the product performance is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology and provides a code stream fusion method, chip, NVR, server, network and storage medium. Background Art

[0002] The main purpose of a network video recorder (NVR) is to centrally process and store data transmitted by front-end IP cameras (IP Cameras, IPCs), and to implement functions such as video preview and playback.

[0003] Only one IPC is deployed in the target scene, and the IPC is controlled to rotate, pitch, and shoot the scene at multiple angles and postures to obtain a panoramic image composed of multi-angle images. However, in actual applications, due to the uneven terrain and many obstacles in the target scene, it is difficult for one IPC to capture a complete scene image. In addition, when there is a dynamically moving object in the target scene, the dynamic object will also appear morphologically distorted in the panoramic image, affecting the visual presentation effect.

[0004] Therefore, in actual applications, multiple IPCs need to be placed for shooting, and then the bitstreams shot by each IPC are transmitted to the NVR through the NVR channel for centralized processing to generate panoramic video in this scene. However, due to the limited decoding capability and number of channels of the NVR, when the number of IPCs connected to the NVR is greater than the maximum number of streams supplied by the NVR, the NVR will be insufficient in supplying streams, and there will be a situation of competing for bitstreams, causing the NVR to have problems such as screen flickering, black screen, and intermittent progress bar.

[0005] At present, when the number of IPCs connected to the NVR is greater than the number of remaining channels of the NVR, the number of channels of the NVR will be expanded, and the bitstream parameters of the access devices in the NVR will be adjusted to alleviate performance pressure and to undertake multi-channel bitstreams shot by multiple IPCs.

[0006] However, expanding channels will cause performance loss, and the number of channel expansions is limited, making it difficult to meet the code stream transmission requirements from multiple IPCs in a wide field of view. Secondly, the operation of modifying the code stream parameters of the access device is highly dependent on the device configuration of the NVR. As the number of access devices reaches the processing limit of the NVR, this adjustment method will also lose the expected adjustment effect.

[0007] Therefore, how to process stream data that exceeds the number of channels without expanding the original number of channels of the NVR has become an urgent problem to be solved. Summary of the invention

[0008] The embodiments of the present application provide a code stream fusion method, chip, NVR, server, network and storage medium to solve the problem of processing code stream data that is greater than the number of channels without expanding the number of original channels of the NVR.

[0009] In a first aspect, an embodiment of the present application provides a code stream fusion method, including: Acquire the original code stream generated by multiple network cameras IPC deployed in the target scene, where the original code stream is generated by capturing the scene picture of the target scene; The original bitstreams generated by the multiple IPCs are merged to obtain at least one merged bitstream, wherein each merged bitstream is merged with the original bitstreams of at least two IPCs; At least one fused code stream is transmitted to at least one code stream display network video recorder NVR.

[0010] Optionally, the original bitstreams generated by multiple IPCs are merged to obtain at least one merged bitstream: Device grouping is performed based on location information of multiple IPCs in the target scene, multiple IPCs with adjacent deployment locations are divided into an associated device group, and at least one IPC with non-adjacent deployment locations is divided into an unrelated device group; The original code streams generated by at least two IPCs in the unrelated device group are merged into a first merged code stream, and / or the original code streams generated by multiple IPCs in at least one associated device group are merged into one or more second merged code streams.

[0011] Optionally, the merging of the original bitstreams generated by the multiple IPCs to obtain at least one fused bitstream includes: Acquire at least one target stream group selected by the target object, wherein the target stream group is composed of original streams generated by at least one IPC selected by the target object; The original code streams not selected by the target object are merged into a third merged code stream, and / or the original code streams in the target code stream group are merged to obtain one or more fourth merged code streams.

[0012] Optionally, perform the following operations for an associated device group: Perform image matching on the original code streams generated by multiple IPCs in an associated device group to determine the overlapping areas between the original code streams; Based on the obtained overlapping areas, multiple original code streams are merged into one merged code stream.

[0013] Optionally, the original code streams generated by multiple IPCs in an associated device group are image matched to determine the overlapped area between the original code streams, wherein the following operations are performed for the original code streams generated by multiple IPCs in an associated device group: Compare the original code stream generated by an IPC in an associated device group with the original code stream generated by at least one other IPC in the same associated device group to obtain the pixel difference between the original code streams; An area composed of pixels whose pixel difference values ​​do not exceed the first set pixel threshold is used as an original code stream overlap area between the one IPC and the other IPC.

[0014] Optionally, after determining the overlapping area between the original code streams and before merging the multiple original code streams into one fused code stream, the method further includes: For two original bitstreams with different video resolutions and / or frame rates, resampling and time synchronizing the two original bitstreams to obtain two adjusted original bitstreams; For two original code streams with different pixel values, image preprocessing and image perspective conversion are performed on the two original code streams to obtain the two adjusted original code streams.

[0015] Optionally, resampling and time synchronizing the two original bitstreams to obtain the two adjusted original bitstreams includes: Resample the two original bitstreams to align the video resolutions and / or frame rates of the two original bitstreams; Adjust the frame rates of the two original bitstreams to align the frame rates of the two original bitstreams; According to the preset reference time, the shooting time of the two original code streams is aligned to obtain the adjusted two original code streams.

[0016] Optionally, performing image preprocessing and image view angle conversion on the two original bitstreams to obtain the two adjusted original bitstreams includes: Performing image preprocessing on the two original code streams, and extracting key pixel points of the two original code streams after preprocessing, respectively, wherein the key pixel points are: pixel points in one original code stream whose pixel difference value with other pixel points exceeds a second set pixel point threshold, and the second set pixel point threshold is greater than the first set pixel point threshold; Based on the extracted multiple key pixel points, the geometric transformation parameters between the two original code streams are determined, and based on the geometric transformation parameters, the image perspective conversion is performed on the two original code streams to obtain the adjusted two original code streams.

[0017] Optionally, the device grouping is performed based on the location information of the multiple IPCs in the target scene, wherein the following operations are performed for the multiple IPCs respectively: Determine the relative distance between an IPC and other IPCs based on the respective location information of an IPC and other IPCs deployed in the target scenario; When all relative distances are greater than the set distance threshold, an IPC is classified into the irrelevant device group; When at least one relative distance does not exceed a set distance threshold, an IPC and other IPCs with the smallest relative distance are grouped into an associated device group.

[0018] In the second aspect, the embodiment of the present application also provides a code stream fusion chip for implementing the code stream fusion method described above, wherein the code stream fusion chip is used to obtain the original code stream generated by multiple network cameras IPC deployed in the target scene, wherein the original code stream is generated by capturing the scene image of the target scene; the original code streams generated by multiple IPCs are merged to obtain at least one merged code stream, wherein each merged code stream merges the original code streams of at least two IPCs; and the at least one merged code stream is transmitted to at least one code stream display network video recorder NVR.

[0019] In a third aspect, an embodiment of the present application further provides a stream fusion network video recorder NVR, which is used to implement the stream fusion method described above, wherein the stream fusion NVR is used to obtain the original streams generated by multiple network cameras IPC deployed in the target scene, wherein the original streams are generated by capturing the scene images of the target scene; the original streams generated by multiple IPCs are merged to obtain at least one merged stream, wherein each merged stream contains the original streams of at least two IPCs; and the at least one merged stream is transmitted to at least one stream display network video recorder NVR.

[0020] In a fourth aspect, an embodiment of the present application further provides a stream fusion server for implementing the stream fusion method described above, wherein the stream fusion server is used to obtain original streams generated by multiple network cameras IPCs deployed in a target scene, wherein the original streams are generated by capturing scene images of the target scene; the original streams generated by multiple IPCs are merged to obtain at least one merged stream, wherein each merged stream is merged with original streams of at least two IPCs; and the at least one merged stream is transmitted to at least one stream display network video recorder NVR.

[0021] In a fifth aspect, the embodiment of the present application further provides a stream fusion network, including at least one network camera IPC, a stream fusion network video recorder NVR, and at least one stream display network video recorder NVR; The IPC is used to collect the scene picture of the target scene and generate an original bit stream; The code stream fusion NVR is used to fuse the original code streams generated by multiple IPCs to obtain at least one fused code stream, and transmit the at least one fused code stream to at least one code stream display NVR; The code stream display NVR is used to connect at least one fused code stream to a corresponding channel and display the picture to a target object.

[0022] In a sixth aspect, the embodiment of the present application further provides a stream fusion network, including at least one network camera IPC, a stream fusion server, and a stream display network video recorder NVR; The IPC is used to collect the scene picture of the target scene and generate an original bit stream; The code stream fusion server is used to fuse the original code streams generated by multiple IPCs to obtain at least one fused code stream, and transmit the at least one fused code stream to at least one code stream display network video recorder NVR; The stream display NVR is used to connect at least one fused stream to the corresponding channel and display the picture to the target object.

[0023] In the seventh aspect, an embodiment of the present application also provides a computer-readable storage medium, which includes a program code. When the program product is running on a stream fusion chip, a stream fusion network video recorder NVR or a stream fusion server, the program code is used to enable the stream fusion chip, the stream fusion network video recorder NVR or the stream fusion server to perform the steps of any one of the above-mentioned stream fusion methods.

[0024] In an eighth aspect, an embodiment of the present application further provides a computer program product, including computer instructions, which are executed by a processor to perform the steps of any one of the above-mentioned code stream fusion methods.

[0025] The beneficial effects of this application are as follows: The embodiment of the present application provides a code stream fusion method, chip, NVR, server, network and storage medium, the method comprising: obtaining the original code stream generated by multiple network camera IPCs deployed in the target scene, the original code stream is generated by collecting the scene picture of the target scene; fusing the original code streams generated by multiple IPCs to obtain at least one fused code stream, wherein each fused code stream is fused with the original code streams of at least two IPCs, and then transmitting the at least one fused code stream to at least one code stream display network video recorder NVR, so as to achieve that without expanding the number of NVR channels, the original channel can access more code stream data, display more scene pictures, and improve product performance. Moreover, since the number of NVR channels is not expanded, the NVR is not limited by the device configuration of the NVR itself when processing the code stream, further improving the work efficiency of processing the code stream and the picture display effect.

[0026] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1A A schematic diagram of a process for fusing multiple original code streams provided in an embodiment of the present application; Figure 1B A logical schematic diagram of fusing multiple original code streams provided in an embodiment of the present application; Figure 1C A schematic diagram of a process for fusing the original code streams generated by multiple IPCs deployed adjacently in an embodiment of the present application; Figure 1D A schematic diagram of the locations of multiple IPCs in a target scene provided in an embodiment of the present application; Figure 1E A logical diagram of connecting the original code streams of multiple IPCs to the code stream display NVR provided in an embodiment of the present application; Figure 2A A schematic diagram of the process of generating the original code stream of multiple IPCs selected by the fusion target object provided in the embodiment of the present application; Figure 2B A schematic diagram of a code stream access selection interface provided in an embodiment of the present application; Figure 2C A logical schematic diagram of the original code streams generated by multiple IPCs selected by the fusion target object provided in the embodiment of the present application; Figure 3 A schematic diagram of the structure of a code stream fusion network provided in an embodiment of the present application; Figure 4 A schematic diagram of the structure of another code stream fusion network provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0029] Some of the terms used in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0030] 1. NVR: A device used in monitoring systems. Unlike traditional digital video recorders, network video recorders are not directly connected to analog cameras, but communicate with IPCs through network interfaces to receive, store, manage and play digital videos from IPCs, providing a highly integrated, networked and intelligent video surveillance management method.

[0031] 2. IPC: It is a digital video camera that can directly transmit the recorded scene video in digital form to a network video recorder, computer, mobile phone or other digital storage device through a wired or wireless network connection. Compared with the analog camera used in traditional closed-circuit television systems, IPC has many characteristics such as digital transmission, high definition, intelligent functions, flexibility and scalability.

[0032] IPC has designed a variety of forms according to different application requirements and scenarios, including spherical cameras and multi-eye cameras. Spherical cameras are named for their rotatable spherical shape. They have universal adjustment functions and support pitch angles and horizontal rotations. Some also support zoom. Spherical cameras are suitable for scenes that require large-scale monitoring or detailed tracking. Multi-eye cameras inherit multiple lenses or sensors, each of which can point in different directions, or provide different viewing angles (such as wide angle and narrow angle) in the same direction. Multi-eye cameras are suitable for intersections, large public places, and other occasions that require wide field of view coverage.

[0033] 3. Video encoding: It converts the original video data into a digital format and processes it through a compression algorithm. It aims to ensure visual quality while significantly reducing the size of the video file so that the video content can be stored, transmitted and played more efficiently.

[0034] 4. In a video stream, I frames, P frames and B frames are three basic frame types. They are the core concepts in video compression coding technology and are mainly used for efficient storage and transmission of video data.

[0035] A key frame (Intra Frame, I frame) is a complete image frame that contains all the necessary information of a scene and can be decoded independently without referring to other frames.

[0036] A predicted frame (P frame) is encoded by referring to its previous frame (usually an I frame or a P frame) and only records the difference information from the previous frame, thereby achieving a higher compression rate.

[0037] Bidirectionally predicted frames (B frames) are encoded by referring to the previous and next frames, and use bidirectional temporal redundancy to further improve compression efficiency. They have the highest compression ratio and the smallest data volume among the three types of frames.

[0038] The following is a brief introduction to the design concept of the embodiment of the present application: The main purpose of NVR is to centrally process and store data transmitted by the IPC connected to the front end, and realize multiple functions such as previewing videos and viewing playback.

[0039] Only one IPC is deployed in the target scene, and the IPC is controlled to rotate, pitch, and shoot the scene at multiple angles and postures to obtain a panoramic image composed of multi-angle images. However, in actual applications, due to the uneven terrain and many obstacles in the target scene, it is difficult for one IPC to capture a complete scene image. In addition, when there is a dynamically moving object in the target scene, the dynamic object will also appear morphologically distorted in the panoramic image, affecting the visual presentation effect.

[0040] Therefore, in actual applications, multiple IPCs need to be placed for shooting, and then the bitstreams shot by each IPC are transmitted to the NVR through the NVR channel for centralized processing to generate panoramic video in this scene. However, due to the limited decoding capability and number of channels of the NVR, when the number of IPCs connected to the NVR is greater than the maximum number of streams supplied by the NVR, the NVR will be insufficient in supplying streams, and there will be a situation of competing for bitstreams, causing the NVR to have problems such as screen flickering, black screen, and intermittent progress bar.

[0041] At present, when the number of IPCs connected to the NVR is greater than the number of remaining channels of the NVR, the number of channels of the NVR will be expanded, and the bitstream parameters of the access devices in the NVR will be adjusted to alleviate performance pressure and to undertake multi-channel bitstreams shot by multiple IPCs.

[0042] However, expanding channels will cause performance loss, and the number of channel expansions is limited, making it difficult to meet the code stream transmission requirements from multiple IPCs in a wide field of view. Secondly, the operation of modifying the code stream parameters of the access device is highly dependent on the device configuration of the NVR. As the number of access devices reaches the processing limit of the NVR, this adjustment method will also lose the expected adjustment effect.

[0043] Therefore, how to process bitstream data that is more than the number of channels without expanding the number of original channels of the NVR has become an urgent problem to be solved. In order to solve the above problem, the present application proposes a bitstream fusion method.

[0044] The method includes: obtaining the original code streams generated by multiple network camera IPCs deployed in the target scene, the original code streams are generated by collecting scene images of the target scene; fusing the original code streams generated by the multiple IPCs to obtain at least one fused code stream, wherein each fused code stream is fused with the original code streams of at least two IPCs, and then transmitting the at least one fused code stream to at least one code stream display network video recorder NVR, so as to allow the original channel to access more code stream data and display more scene images without expanding the number of NVR channels, thereby improving product performance. Moreover, since the number of NVR channels is not expanded, the NVR is not restricted by the device configuration of the NVR itself when processing the code stream, further improving the work efficiency of processing the code stream and the picture display effect.

[0045] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0046] The bitstream fusion method provided in the embodiment of the present application can be stored in a separate bitstream fusion chip in the form of an application program or code, or can be set in a processing chip of a bitstream fusion NVR in the form of an application program or code, or can be set in a bitstream fusion server or other device in the form of an application program or code. Figure 1A-1B The schematic diagram shown introduces the process of fusing the original code streams generated by multiple IPCs.

[0047] S101: Acquire original code streams generated by multiple IPCs deployed in a target scene, where the original code streams are generated by capturing scene images of the target scene.

[0048] Due to the uneven terrain and many obstacles of the target scene, it is difficult for one IPC to capture a complete scene image. In order to ensure that the complete picture of the target scene is captured, multiple IPCs will be placed for image acquisition in actual applications regardless of whether the terrain of the target scene is complex.

[0049] IPC converts the collected scene images into original bitstreams, and transmits them to bitstream fusion chips, bitstream fusion NVRs, bitstream fusion servers and other devices through pre-established transmission channels. These devices perform bitstream fusion operations to obtain at least one fused bitstream.

[0050] S102: The original code streams generated by the multiple IPCs are merged to obtain at least one merged code stream, wherein each merged code stream is merged with the original code streams of at least two IPCs. In the process of merging code streams, it is not limited to merge all the original code streams. There may be a situation where a part of the original code streams are merged and another part of the original code streams are not merged, or there may be a situation where all the original code streams are merged. Therefore, the present application provides two fusion methods, one is to merge the original code streams generated by multiple IPCs with adjacent deployment locations, and the other is to merge the original code stream selected by the target object.

[0051] Combination Figure 1C The flowchart shown introduces the specific implementation process of the fusion method of this embodiment.

[0052] S1021: Grouping devices based on location information of multiple IPCs in the target scene, grouping multiple IPCs with adjacent deployment locations into an associated device group, and grouping at least one IPC with non-adjacent deployment locations into an unrelated device group.

[0053] In order to save the processing resources of the device, improve the efficiency of code stream fusion, and avoid repeated verification of the original code stream overlap area between IPCs, this application adds multiple IPCs to the device through automatic search protocol or manual operation, and the device links the map software. Figure 1D In the map shown, the device information and location information of multiple IPCs connected to the device are marked in the form of pushpins. Then, based on the location information of multiple IPCs in the target scene, the devices are grouped, and multiple IPCs with adjacent deployment locations are divided into an associated device group, and at least one IPC with non-adjacent deployment locations is divided into an unrelated device group.

[0054] Specifically, the following operations are performed for multiple IPCs: based on the respective location information of an IPC and other IPCs deployed in the target scenario, the relative distance between an IPC and other IPCs is determined; when each relative distance is greater than a set distance threshold, an IPC is classified into an unrelated device group; however, when at least one relative distance does not exceed the set distance threshold, an IPC and other IPCs with the smallest relative distance are classified into an associated device group.

[0055] When all relative distances are greater than the set distance threshold, it means that the distance between one IPC and other IPCs is far, and the possibility of overlapping areas between the original code streams generated by this IPC and other IPCs is low. Therefore, this IPC is classified into the irrelevant device group.

[0056] When at least one relative distance does not exceed the set distance threshold, it means that there are other IPCs around the IPC, and there is a high possibility that there is an overlapping area between the original code streams generated by the IPC and other IPCs whose relative distances do not exceed the set distance threshold. Therefore, the IPC and other IPCs with the smallest relative distances are divided into an associated device group.

[0057] S1022: merging original code streams generated by at least two IPCs in an unrelated device group into a first fused code stream, and / or merging original code streams generated by multiple IPCs in at least one associated device group into one or more second fused code streams.

[0058] Since there is no overlapping area between the original code streams generated by at least one IPC in the unrelated device group, the original code stream in the device group can be directly output as one code stream to the code stream display NVR without any processing. Alternatively, an edge splicing algorithm is used to splice the original code streams generated by at least two IPCs in the unrelated device group, and then a compatible encoder (such as H.264, H.265) is used to encode the spliced ​​code stream to obtain a first fused code stream to adapt to the target resolution and expected network transmission quality.

[0059] It can be understood that in this embodiment, all original code streams in the unrelated device group can be merged into one first merged code stream, or multiple first merged code streams, or only part of the original code streams can be merged to obtain one or multiple first merged code streams, and the other part of the original code streams are not merged. Similarly, the original code streams generated by multiple IPCs in the associated device group can be merged into one second merged code stream, or multiple second merged code streams, which is not limited in this embodiment.

[0060] For the fused bitstream obtained by splicing and then encoding, the quality of the fused bitstream will be evaluated based on its I-frame, P-frame and B-frame. If the quality evaluation fails, the encoder parameters will be readjusted by software adaptive parameter adjustment or manual adjustment to optimize network transmission quality and bandwidth utilization.

[0061] However, there are overlapping areas between the original code streams generated by multiple IPCs in the associated device group. In order to ensure the integrity of the picture, the overlapping areas between the two original code streams need to be merged. Therefore, taking an associated device group as an example, the image frame similarity algorithm is first used to match the original code streams generated by multiple IPCs in the associated device group to determine the overlapping areas between the original code streams. Then, based on the obtained overlapping areas, the edge splicing algorithm is used to merge the multiple original code streams into one fused code stream.

[0062] Among them, taking an associated device group as an example, for the original code streams generated by multiple IPCs in the device group, the following operations are performed respectively: the original code stream generated by an IPC in an associated device group is compared with the original code stream generated by at least one other IPC in the same associated device group to obtain the pixel difference between the original code streams; the area composed of pixels whose pixel difference does not exceed the first set pixel threshold is used as the original code stream overlap area between the IPC and the other IPCs.

[0063] When stitching overlapping areas, there are two key issues involved. The first issue is how to merge the bitstreams when the video resolution and / or frame rate between the original bitstreams are different. The second issue is how to merge the bitstreams when the pixel values ​​between the original bitstreams are different due to interference factors such as shooting background and shooting angle.

[0064] Before splicing the overlapping areas, the two code streams with overlapping areas need to be adjusted accordingly according to the above two situations.

[0065] (1) For two original bitstreams with different video resolutions and / or frame rates, the two original bitstreams are resampled and time-synchronized to obtain the two adjusted original bitstreams.

[0066] First, the two original bitstreams are resampled to align the video resolutions and / or frame rates of the two original bitstreams.

[0067] Specifically, based on the video resolution and frame rate of one of the original code streams, up-sample or down-sample the other original code stream, so that the video resolution and frame rate of the two original code streams are kept consistent after adjustment.

[0068] For example, the original bitstream A has a video resolution of 720p and a frame rate of 24fps, and the original bitstream B has a video resolution of 1080p and a frame rate of 30fps. Upsample the original bitstream A to increase its frame rate to 30fps and its video resolution to 1080p. Alternatively, downsample the original bitstream B to reduce its frame rate to 24fps while maintaining its 1080p video resolution.

[0069] Secondly, the frame numbers of the two original code streams are adjusted by adding or reducing frames to align the frame numbers of the two original code streams. Finally, the shooting time of the two original code streams is aligned according to the preset reference time to obtain the adjusted two original code streams. Among them, the network time of the device performing the code stream fusion operation can be used as the preset reference time to adjust the shooting time of the two original code streams so that the adjusted shooting time is aligned with the preset reference time.

[0070] (2) For two original bitstreams with different pixel values, image preprocessing and image perspective conversion are performed on the two original bitstreams to obtain the two adjusted original bitstreams.

[0071] First, the two original code streams are image preprocessed, and the key pixel points of the two original code streams after preprocessing are extracted respectively. The key pixel point is: a pixel point whose pixel difference value with other pixel points in one original code stream exceeds the second set pixel point threshold, that is, a pixel point whose pixel value is greatly different from other pixel points in one original code stream, and the second set pixel point threshold is greater than the first set pixel point threshold.

[0072] By adjusting the brightness, contrast and other image parameters of the two original bitstreams, the two original bitstreams are preprocessed to ensure that the two original bitstreams have the video quality and clarity required for analysis. Then, the feature point detection algorithm is used to extract features from the two original bitstreams and locate the key pixel points in each bitstream.

[0073] For example, the Fast Library for Approximate Nearest Neighbors (FLANN) search algorithm is used to calculate the feature similarity between key pixel points in the two original bitstreams and determine the key pixel points that match in the two original bitstreams. The core advantage of the FLANN search algorithm is that it can quickly find the closest neighbor of a point in a large data set. Although the algorithm sacrifices a little bit of accuracy to improve processing speed, the matching results output by the algorithm may not be completely accurate, but they are close enough to the correct results.

[0074] Secondly, based on the extracted multiple key pixel points, the geometric transformation parameters between the two original bitstreams are determined, and based on the geometric transformation parameters, the image perspective conversion is performed on the two original bitstreams to obtain the adjusted two original bitstreams.

[0075] Based on the key pixel points that have been matched successfully, the optimal geometric transformation parameters between the two original bitstreams are calculated to verify whether the two original bitstreams are from the same scene. When it is determined that they are from the same scene, the image perspective of the two original bitstreams is converted based on the optimal geometric transformation parameters (for example, non-ideal angles such as squint angles are converted to normal angles), so that the image perspectives of the two original bitstreams are similar after adjustment, optimizing the visual stitching effect.

[0076] Furthermore, a threshold may be set on the device that performs the code stream fusion operation to control the quality of the match, so as to achieve a better visual stitching effect.

[0077] After the above adjustment, for the two original bitstreams with overlapping areas, the following operations are performed to splice them: the overlapping areas of the two original bitstreams are traversed frame by frame, and the image frames corresponding to the same frame in the two original bitstreams are merged into one frame.

[0078] Use a compatible encoder (such as H.264, H.265) to encode the spliced ​​bitstream of a group of associated devices to obtain a fused bitstream. Adjust the encoder parameters to suit the target resolution and expected network transmission quality.

[0079] For each fused bitstream obtained by splicing and then encoding, the quality of the fused bitstream will be evaluated based on its I-frame, P-frame and B-frame. If the quality evaluation fails, the encoder parameters will be readjusted by software adaptive parameter adjustment or manual adjustment to optimize network transmission quality and bandwidth utilization.

[0080] S103: Transmit at least one fused code stream to at least one code stream display NVR.

[0081] Next, combine Figure 1E The logical schematic diagram shown takes a specific embodiment as an example to introduce the whole process of connecting the original code streams of multiple IPCs to the code stream display NVR.

[0082] Multiple IPCs are added to the device through automatic search protocols or manual operation. The device is linked to the map software to mark the device information and location information of multiple IPCs connected to the device on the map in the form of pushpins. Then, the devices are grouped based on the location information of multiple IPCs in the target scene. Multiple IPCs with adjacent deployment locations are divided into an associated device group, and at least one IPC with non-adjacent deployment locations is divided into an unrelated device group.

[0083] By using the image frame similarity algorithm and the edge stitching algorithm, the original code streams of multiple IPCs belonging to the same associated device group are merged into one merged code stream, and the original code stream of at least one IPC belonging to an unrelated device group is merged into another merged code stream.

[0084] When the IPC position changes, the device performing the stream fusion operation modifies the corresponding IPC position information stored in itself, and regroups the devices and performs stream fusion. When the IPC position does not change, the stream fusion is still performed according to the original device grouping.

[0085] The stream display NVR accesses the Internet Protocol (IP) address of the device and actively pulls the processed multi-channel fusion stream. The stream display NVR accesses the corresponding number of fusion streams in n channels according to the self-selected stream access options. The stream display NVR decodes the multi-channel fusion streams to obtain multiple video data packets. According to the timestamp of the video data packet and the location information of the IPC, the video data packets are arranged in time and space order to present a panoramic video covering the monitoring range.

[0086] The embodiment of the present application fuses the original bitstreams generated by multiple IPCs to obtain at least one fused bitstream, wherein each fused bitstream fuses the original bitstreams of at least two IPCs, and then transmits the at least one fused bitstream to at least one bitstream display network video recorder NVR, so that the original channel can access more bitstream data and display more scene images without expanding the number of NVR channels, thereby improving product performance. Moreover, since the number of NVR channels is not expanded, the NVR is not restricted by the device configuration of the NVR itself when processing the bitstream, further improving the work efficiency of processing the bitstream and the image display effect.

[0087] Combination Figure 2A As shown in the flowchart, another embodiment of the present application discloses a specific implementation process of the second fusion method.

[0088] S201: Acquire at least one target stream group selected by a target object, where the target stream group is composed of original streams generated by at least one IPC selected by the target object.

[0089] S202: merging the original code streams that are not selected by the target object into a third merged code stream, and / or merging the original code streams in the target code stream group to obtain one or more fourth merged code streams.

[0090] Figure 2B It includes multiple selection controls, and the number of split boxes in each selection control represents the number of channels of an NVR. Target object Click any selection control in the figure to select which original streams generated by IPCs will form a target stream group, and which target stream groups will be connected to a channel, so as to display multiple fourth fusion streams in one channel.

[0091] The original bitstreams not selected by the target object are merged into the third merged bitstream, and the original bitstreams in the target bitstream group selected by the target object are merged into one or more fourth merged bitstreams. Therefore, the number of target bitstream groups and the number of fourth merged bitstreams may be the same or different, which is not limited here.

[0092] Similarly, similar to the case of the first fused code stream, in this embodiment, all the original code streams not selected by the target object can be fused into one third fused code stream, or multiple third fused code streams, or only part of the original code streams can be fused to obtain one or multiple third fused code streams, and the other part of the original code streams are not fused. Similarly, the original code streams generated by multiple IPCs selected by the target object can be fused into one fourth fused code stream, or multiple fourth fused code streams, which is not limited in this embodiment.

[0093] For example, Figure 2C As shown, the target object selects 4 channel selection controls, selects to connect the target stream group 1 composed of IPC 1-9 to channel 1, and selects to connect the target stream group 2 composed of IPC 10-13 to channel 2. The device performing the stream fusion operation fuses the target stream group 1 into a fourth fused stream and connects it to channel 1, fuses the target stream group 2 into another fourth fused stream and connects it to channel 2, fuses the original streams generated by IPC14-20 not selected by the target object into a third fused stream and connects it to channel 3, and uses the original stream generated by IPC21 selected by the target object as an independent stream and connects it to channel 4 separately.

[0094] In this embodiment, an image frame similarity algorithm is used to determine the overlapping areas between the original bitstreams. For the original bitstreams without overlapping areas, an edge splicing algorithm is used for splicing, and then a compatible encoder (such as H.264, H.265) is used to encode the spliced ​​bitstream to obtain a fused bitstream. For the original bitstreams with overlapping areas, when the video resolution and / or frame rate and pixel value of the two original bitstreams are the same, an edge splicing algorithm is used to merge multiple original bitstreams into one fused bitstream.

[0095] For the fused bitstream obtained by splicing and then encoding, the quality of the fused bitstream will be evaluated based on its I-frame, P-frame and B-frame. If the quality evaluation fails, the encoder parameters will be readjusted by software adaptive parameter adjustment or manual adjustment to optimize network transmission quality and bandwidth utilization.

[0096] The specific operations of bitstream fusion, bitstream splicing, bitstream quality assessment, and bitstream parameter adjustment have been introduced in the first fusion method and will not be repeated here.

[0097] Based on the same inventive concept as the above method embodiment, the embodiment of the present application also provides a code stream fusion chip. The code stream fusion chip is used to obtain the original code streams generated by multiple network cameras IPC deployed in the target scene, and the original code streams are generated by capturing the scene images of the target scene; the original code streams generated by multiple IPCs are fused to obtain at least one fused code stream, wherein each fused code stream is fused with the original code streams of at least two IPCs; and the at least one fused code stream is transmitted to at least one code stream display network video recorder NVR.

[0098] The code stream fusion chip may be an independent chip with a code stream fusion function, or the code stream fusion function may be integrated into other chips.

[0099] Based on the same inventive concept as the above method embodiment, the embodiment of the present application also provides a stream fusion NVR. The stream fusion NVR is used to obtain the original streams generated by multiple network cameras IPC deployed in the target scene, and the original streams are generated by capturing the scene images of the target scene; the original streams generated by multiple IPCs are fused to obtain at least one fused stream, wherein each fused stream is fused with the original streams of at least two IPCs; and the at least one fused stream is transmitted to at least one stream display network video recorder NVR.

[0100] The stream fusion NVR may be an independent NVR having only the stream fusion function, or may be a stream display NVR integrated with the stream fusion function, so that the stream fusion NVR has both the stream display function and the stream fusion function.

[0101] Based on the same inventive concept as the above method embodiment, the embodiment of the present application also provides a stream fusion server. The stream fusion server is used to obtain the original streams generated by multiple network cameras IPC deployed in the target scene, and the original streams are generated by capturing the scene images of the target scene; the original streams generated by multiple IPCs are merged to obtain at least one merged stream, wherein each merged stream is merged with the original streams of at least two IPCs; and the at least one merged stream is transmitted to at least one stream display network video recorder NVR.

[0102] The code stream fusion server can be an independent server with only the code stream fusion function, or it can be an existing server in the network, and the server integrates the code stream fusion function, so that the server has both the functions of a conventional network server and the code stream fusion function.

[0103] Based on the same inventive concept as the above method embodiment, the embodiment of the present application also provides a code stream fusion network. Figure 3The stream fusion network shown includes: at least one network camera IPC, a stream fusion network video recorder NVR, and at least one stream display network video recorder NVR; The IPC is used to collect the scene picture of the target scene and generate an original bit stream; The code stream fusion NVR is used to fuse the original code streams generated by multiple IPCs into at least one fused code stream, and transmit the at least one fused code stream to at least one code stream display NVR; The code stream display NVR is used to connect at least one fused code stream to a corresponding channel and display the picture to a target object.

[0104] Based on the same inventive concept as the above method embodiment, the embodiment of the present application also provides a code stream fusion network. Figure 4 The stream fusion network shown includes: at least one network camera IPC, a stream fusion server, and a stream display network video recorder NVR; The IPC is used to collect the scene picture of the target scene and generate an original bit stream; The code stream fusion server is used to fuse the original code streams generated by multiple IPCs into at least one fused code stream, and transmit the at least one fused code stream to at least one code stream display network video recorder NVR; The stream display NVR is used to connect at least one fused stream to the corresponding channel and display the picture to the target object.

[0105] In some possible implementations, various aspects of the bitstream fusion method provided in the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is run on a bitstream fusion chip, a bitstream fusion network video recorder NVR, or a bitstream fusion server, the computer program is used to enable the bitstream fusion chip, the bitstream fusion network video recorder NVR, or the bitstream fusion server to perform the steps of the bitstream fusion method according to various exemplary implementations of the present application described above in this specification. For example, the bitrate fusion chip may perform the following steps: Figure 1A Follow the steps shown in .

[0106] The program product may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0107] The program product of the embodiment of the present application may adopt a portable compact disk read-only memory (CD-ROM) and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, apparatus, or device.

[0108] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0109] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0110] The computer program for performing the operation of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer program can be executed entirely on the user's computer device, partially on the user's computer device, as a separate software package, partially on the user's computer device and partially on a remote computer device, or completely on the remote computer device. In the case of a remote computer device, the remote computer device can be connected to the user's computer device through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer device (for example, using an Internet service provider to connect through the Internet).

[0111] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0112] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0113] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain a computer-usable computer program.

[0114] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program commands. These computer program commands can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the commands executed by the processor of the computer or other programmable data processing device generate a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0115] These computer program commands may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the commands stored in the computer-readable memory produce a manufactured product including a command device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0116] These computer program commands may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the commands executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0117] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0118] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A code stream fusion method, characterized in that: include: Acquire the original code stream generated by multiple network cameras IPC deployed in the target scene, where the original code stream is generated by capturing the scene picture of the target scene; The original bitstreams generated by the multiple IPCs are merged to obtain at least one merged bitstream, wherein each merged bitstream is merged with the original bitstreams of at least two IPCs; At least one fused code stream is transmitted to at least one code stream display network video recorder NVR.

2. The method according to claim 1, characterized in that The merging of the original bitstreams generated by the multiple IPCs to obtain at least one fused bitstream includes: Device grouping is performed based on location information of multiple IPCs in the target scene, multiple IPCs with adjacent deployment locations are divided into an associated device group, and at least one IPC with non-adjacent deployment locations is divided into an unrelated device group; The original code streams generated by at least two IPCs in the unrelated device group are merged into a first merged code stream, and / or the original code streams generated by multiple IPCs in at least one associated device group are merged into one or more second merged code streams.

3. The method according to claim 1, characterized in that The merging of the original bitstreams generated by the multiple IPCs to obtain at least one fused bitstream includes: Acquire at least one target stream group selected by the target object, wherein the target stream group is composed of original streams generated by at least one IPC selected by the target object; The original code streams not selected by the target object are merged into a third merged code stream, and / or the original code streams in the target code stream group are merged to obtain one or more fourth merged code streams.

4. The method according to claim 2, characterized in that For a linked device group, do the following: Perform image matching on the original code streams generated by multiple IPCs in an associated device group to determine the overlapping areas between the original code streams; Based on the obtained overlapping areas, multiple original code streams are merged into one merged code stream.

5. The method according to claim 4, characterized in that The original code streams generated by multiple IPCs in an associated device group are image matched to determine the overlapped areas between the original code streams, wherein the following operations are performed for the original code streams generated by multiple IPCs in an associated device group: Compare the original code stream generated by an IPC in an associated device group with the original code stream generated by at least one other IPC in the same associated device group to obtain the pixel difference between the original code streams; An area composed of pixels whose pixel difference values ​​do not exceed the first set pixel threshold is used as an original code stream overlap area between the one IPC and the other IPC.

6. The method according to claim 4, characterized in that After determining the overlapped area between the original code streams and before merging the multiple original code streams into one fused code stream, the method further includes: For two original bitstreams with different video resolutions and / or frame rates, resample and time synchronize the two original bitstreams to obtain two adjusted original bitstreams; For two original code streams with different pixel values, image preprocessing and image perspective conversion are performed on the two original code streams to obtain the two adjusted original code streams.

7. The method according to claim 6, characterized in that The resampling and time synchronization of the two original bit streams to obtain the adjusted two original bit streams includes: Resample the two original bitstreams to align the video resolutions and / or frame rates of the two original bitstreams; Adjust the frame rates of the two original bitstreams to align the frame rates of the two original bitstreams; According to the preset reference time, the shooting time of the two original code streams is aligned to obtain the adjusted two original code streams.

8. The method according to claim 6, characterized in that The performing image preprocessing and image perspective conversion on the two original bitstreams to obtain the adjusted two original bitstreams includes: Performing image preprocessing on the two original code streams, and extracting key pixel points of the two original code streams after preprocessing, respectively, wherein the key pixel points are: pixel points in one original code stream whose pixel difference value with other pixel points exceeds a second set pixel point threshold, and the second set pixel point threshold is greater than the first set pixel point threshold; Based on the extracted multiple key pixel points, the geometric transformation parameters between the two original code streams are determined, and based on the geometric transformation parameters, the image perspective conversion is performed on the two original code streams to obtain the adjusted two original code streams.

9. The method according to any one of claims 5 to 8, characterized in that: The device grouping is performed based on the location information of the multiple IPCs in the target scene, wherein the following operations are performed for the multiple IPCs respectively: Determine the relative distance between an IPC and other IPCs based on the respective location information of an IPC and other IPCs deployed in the target scenario; When all relative distances are greater than the set distance threshold, an IPC is classified into the irrelevant device group; When at least one relative distance does not exceed a set distance threshold, an IPC and other IPCs with the smallest relative distance are grouped into an associated device group.

10. A code stream fusion chip, characterized in that: Used to implement the method described in any one of claims 1 to 9, the code stream fusion chip is used to obtain the original code stream generated by multiple network cameras IPC deployed in the target scene, and the original code stream is generated by capturing the scene picture of the target scene; the original code streams generated by multiple IPCs are fused to obtain at least one fused code stream, wherein each fused code stream is fused with the original code streams of at least two IPCs; and the at least one fused code stream is transmitted to at least one code stream display network video recorder NVR.

11. A code stream fusion network video recorder NVR, characterized in that: Used to implement the method described in any one of claims 1 to 9, the stream fusion NVR is used to obtain the original streams generated by multiple network cameras IPC deployed in the target scene, and the original streams are generated by capturing the scene images of the target scene; the original streams generated by multiple IPCs are merged to obtain at least one merged stream, wherein each merged stream is merged with the original streams of at least two IPCs; and the at least one merged stream is transmitted to at least one stream display network video recorder NVR.

12. A code stream fusion server, characterized in that: Used to implement the method described in any one of claims 1 to 9, the stream fusion server is used to obtain original streams generated by multiple network cameras IPC deployed in the target scene, and the original streams are generated by capturing scene images of the target scene; the original streams generated by multiple IPCs are merged to obtain at least one merged stream, wherein each merged stream is merged with original streams of at least two IPCs; and the at least one merged stream is transmitted to at least one stream display network video recorder NVR.

13. A code stream fusion network, characterized in that: It comprises at least one network camera IPC, the code stream fusion network video recorder NVR as claimed in claim 11, and at least one code stream display network video recorder NVR; The IPC is used to collect the scene picture of the target scene and generate an original bit stream; The code stream fusion NVR is used to fuse the original code streams generated by multiple IPCs to obtain at least one fused code stream, and transmit the at least one fused code stream to at least one code stream display NVR; The code stream display NVR is used to connect at least one fused code stream to a corresponding channel and display the picture to a target object.

14. A code stream fusion network, characterized in that: It includes at least one network camera IPC, the stream fusion server as claimed in claim 12, and a stream display network video recorder NVR; The IPC is used to collect the scene picture of the target scene and generate an original bit stream; The code stream fusion server is used to fuse the original code streams generated by multiple IPCs into at least one fused code stream, and transmit the at least one fused code stream to at least one code stream display network video recorder NVR; The code stream display NVR is used to connect at least one fused code stream to the corresponding channel and display the picture to the target object.

15. A computer-readable storage medium, characterized in that: It includes program code, and when the program code runs on a stream fusion chip, a stream fusion network video recorder NVR or a stream fusion server, the program code is used to enable the stream fusion chip, the stream fusion network video recorder NVR or the stream fusion server to perform the steps of the method described in any one of claims 1 to 9.

16. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9.

Images