Multi-camera splicing fusion method and system

Abstract

The invention provides a multi-camera splicing and fusing method and a multi-camera splicing and fusing system. The method comprises the following steps: correcting a video picture acquired by a multi-view camera; splicing the corrected video pictures into a panoramic picture with a wider visual angle; splicing seams between every two adjacent video pictures are fused; and outputting the processed panoramic picture. According to the multi-camera splicing and fusing method and system provided by the invention, the technology of splicing the video pictures collected by the multi-camera through the splicing angular points is combined with the technology of fusing the splicing seams of the video pictures through the splicing and fusing algorithm, so that the user experience is improved on the basis of improving the immersive experience of the user in a video conference; and the requirements on the performance of a video chip and the video resolution can be reduced in the splicing and fusion of the panoramic picture.

Description

Technical Field

[0001] The present invention relates to the field of video communication technology, and in particular to a multi-camera splicing and fusion method and a system thereof. Background Art Multi-camera stitching and fusion technology is a technology that combines images or video streams captured by multiple cameras into a unified panoramic image with a wider viewing angle. This technology is widely used in many fields such as video surveillance, virtual reality, telemedicine, intelligent traffic management, and live broadcast of sports events. The development of multi-camera stitching and fusion technology not only improves the coverage and efficiency of video surveillance systems, but also provides a richer and more realistic visual experience for emerging fields such as virtual reality and augmented reality. With the continuous improvement of image processing technology and computing power, the application scope and effect of this technology have been further expanded and improved.

[0002] In the existing technology, in order to achieve the stitching and fusion of panoramic videos, the commonly used methods include hardware-based panoramic stitching algorithms (such as Any View Stitching, referred to as AVS) and software-based solutions (such as OpenCV Stitch). These methods are usually performed inside the video device, using the video streams captured by multiple cameras and processed through the built-in processor or external server.

[0003] Among them, hardware solutions such as the AVS algorithm are usually integrated into dedicated video processing chips, which can complete image preprocessing, feature point detection, image transformation, and splicing and fusion operations inside the camera device. The advantage of this solution is that it can achieve real-time video splicing, but it is often difficult to support high-resolution and complex scene panoramic video output due to hardware performance and cost constraints.

[0004] Software solutions such as OpenCV Stitch use software libraries on general computing platforms for processing. This solution has high flexibility and scalability, and the algorithm and processing flow can be optimized by upgrading the software. However, the Stitch class in OpenCVStitch is mainly used for panoramic stitching of static images, rather than directly for real-time stitching of video frames. Therefore, if you want to use it in a video, you may need to write additional code to automate the above process and implement the processing of continuous video frames. Therefore, this solution has a high demand for computing resources, especially when processing high-resolution video streams in real time, you may encounter performance bottlenecks, resulting in a drop in the video output frame rate or failure to achieve the expected resolution.

[0005] Whether it is a hardware solution or a software solution, the existing panoramic stitching and fusion technology has certain limitations in implementation. First of all, these solutions have high requirements for the number of cameras and the resolution of the fusion. With the increase in the number of cameras and the substantial improvement in resolution, the amount of data processed has increased exponentially, which has put forward higher requirements on the computing power and memory bandwidth of the processor. Secondly, the existing technology has high requirements on the selection and performance of video processing chips. High-performance chips can provide faster image processing speeds and higher video output quality, but the corresponding costs will also increase significantly. Under limited budgets and chip performance, it is often necessary to make a trade-off between real-time performance, resolution, and cost. Summary of the invention

[0006] The present invention provides a multi-camera splicing and fusion method and system thereof, which completes the splicing of video images by calculating the positions of splicing corner points in advance, reduces the requirements on the performance of video chips, and fuses the splicing seams of video images through a splicing and fusion algorithm, thereby improving the immersive experience of users in video conferencing.

[0007] In a first aspect, the present invention provides a multi-camera stitching and fusion method, characterized in that the method comprises: Correct the video images collected by multiple cameras; Stitch the corrected video images into a panoramic image with a wider viewing angle; Merge the seams between two adjacent video images; Output the processed panoramic image.

[0008] In a second aspect, the present invention further provides a multi-camera stitching system, characterized in that the system comprises a correction module, a stitching module, a fusion module and an output module, wherein: The correction module is used to correct the video images collected by the multi-camera; The stitching module is used to stitch the corrected video images into a panoramic image with a wider viewing angle; The fusion module is used to fuse the joints between two adjacent video images; The output module is used to output the processed panoramic picture.

[0009] The advantages of the multi-camera stitching and fusion method and system provided by the present invention are: first, the stitching of multiple video images is completed according to the stitching corner points calculated in advance, and the steps of extracting and detecting local invariant key points in the panoramic image stitching process are omitted, thereby reducing the burden of video chip calculation; second, the performance requirements of the video chip are relatively low, and high frame rate and resolution video output can be achieved under limited video chip performance, while reducing the cost of panoramic video image stitching and ensuring the user's viewing experience; third, in the stitching seam fusion stage, the edge fusion processing is performed on the stitched panoramic image, further reducing the color difference of the stitching seams between cameras, weakening the presence of the stitching seams, and improving the user's immersive conference viewing experience; fourth, the present invention formulates different calibration points for the correction of the video image according to the different postures of the participants, which can make the correction effect more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0011] Figure 1 is a flow chart of a multi-camera stitching and fusion method provided by an embodiment of the present invention; Figure 2 is a multi-camera video picture correction flow chart provided by an embodiment of the present invention; Figure 3 This is a multi-camera video image stitching flow chart provided by an embodiment of the present invention; Figure 4 This is a flow chart of multi-camera video image stitching and fusion provided by an embodiment of the present invention; Figure 5 is a block diagram of a multi-camera stitching system provided by an embodiment of the present invention; Figure 6 This is a demonstration diagram of multi-camera video image stitching provided by an embodiment of the present invention; Figure 7 is a panoramic synthesis flow chart provided by an embodiment of the present invention; Figure 8 is a schematic diagram of a calibration diagram on a calibration board provided in an embodiment of the present invention; Fig. 9 is a diagram of an example of seam fusion provided by an embodiment of the present invention; Figures 10a-10d It is a comparison diagram before and after splicing and fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0012] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. SUMMARY OF THE INVENTION As mentioned above, the present invention provides a flexible management method and system for multi-camera stitching and fusion, which can improve the panoramic video stitching and fusion effect and video resolution and enhance the user's conference experience when the video chip performance is limited.

[0014] Exemplary Methods Figure 1 : is a flow chart of a multi-camera stitching and fusion method provided by an embodiment of the present invention, and the embodiment includes the following steps: S101: Correcting the video images captured by the multi-camera. Different cameras may have different perspectives and directions when capturing images, so correction is needed to unify the perspectives of different cameras into a common reference plane or coordinate system for subsequent processing. In addition, since each camera lens may have distortion, especially in wide-angle lenses, correction is needed to correct these distortions and ensure the geometric accuracy of the image.

[0015] like Figure 2 As shown, step S101 specifically includes: S201: Fix the position and angle of each camera.

[0016] Before correcting the video image, in order to ensure data consistency and reduce the complexity of calculation, the position and angle of the camera need to be fixed.

[0017] S202: Calibrate each camera individually to determine the positions of four splicing corner points of the video images captured by each camera.

[0018] Specifically, you first need to check the number of cameras, and prepare the same number of calibration plates according to the number of cameras for calibrating the cameras. The calibration plate has a calibration diagram, such as Figure 8 As shown. The calibration map has calibration points 811, 812 and 813 and corner point 814. The calibration plate is then placed at a specific location, for example, in front of a conference room table, and then the camera shoots the conference room, and the image captured by the camera is imported into a stitching tool such as OpenCV or PTGui, and the position of the stitching corner point is automatically calculated by the stitching tool. In addition, the stitching corner point needs to be further adjusted based on the position information of the stitching corner point and the calibration map, and the final position of the stitching corner point is determined.

[0019] S203: Determine the position of the calibration point according to the participant information.

[0020] The calibration points are used to correct the distortion of the video images to ensure that the video images are restored as realistically as possible. In addition, because the facial positions of the participants when standing and sitting are different during the correction process, in order to ensure the restoration of the participants' faces in the video images as much as possible during the correction process, two different sets of calibration points need to be formulated according to the participants' postures for more accurate correction. Figure 8 The middle calibration point 812 is the calibration point for the standing posture, and the calibration point 813 is the calibration point for the sitting posture. It can be seen that the calibration point for the sitting posture is slightly lower than the calibration point for the standing posture. By judging whether the posture of the participant is standing or sitting and selecting different calibration points for calibration and fusion, the head image of the participant can be more accurately placed in the center of the image, so that the facial distortion of the participant is reduced.

[0021] Specifically, it is first necessary to detect whether the posture of the participants is standing or sitting. When the number of standing participants is greater than that of sitting participants, a higher standing calibration point 812 is selected; when the number of standing participants is less than that of sitting participants, a lower sitting calibration point 813 is selected.

[0022] S204: Crop the video frame according to the splicing corner point 814.

[0023] Cropping the video images before stitching them together can reduce the total number of pixels that the stitching algorithm needs to process, improve stitching efficiency, and optimize stitching quality.

[0024] S205: The cropped video images are subjected to distortion correction and geometric correction according to the calibration points to obtain rectangular images of the same size.

[0025] Specifically, first, you need to match the positions of the matched calibration points with the actual geometric layout of the calibration object, find the correspondence between the calibration points in the video and the real world, and then use an appropriate algorithm (such as OpenCV's calibrateCamera function) to estimate the distortion parameters of the camera. Next, based on the estimated distortion parameters, select a distortion model (common models include pinhole models and fisheye models) to perform distortion correction on the video captured by the camera. The correction process usually involves reverse mapping, that is, mapping the distorted picture points back to their correct positions in the undistorted state. Finally, the distortion-corrected video picture needs to be scaled, shifted, and other geometrically corrected so that the seams between adjacent video pictures can be aligned as much as possible when they are spliced.

[0026] S102: stitching the corrected video images into a panoramic image with a wider viewing angle.

[0027] like Figure 3 As shown, step S102 specifically includes: S301: Find the splicing corner points with the same corresponding positions in two adjacent video frames.

[0028] Specifically, before stitching two adjacent video screens, it is first necessary to detect four stitching corner points with corresponding positions in the two video screens. For example, stitching corner point 1 in video screen A and stitching corner point 2 in video screen B both represent the same position of the conference room, so these two stitching corner points are corresponding.

[0029] S302: splicing two adjacent video frames using an operator according to the information of the splicing corner points.

[0030] Multiple video images are stitched together based on the stitching corner point information calculated in advance, omitting the steps of extracting and detecting local invariant key points in the panoramic image stitching process, reducing the computational burden of the video chip and achieving high frame rate and resolution video output under limited video chip performance.

[0031] Specifically, the opencv affine transformation or hardware correction operator can be used to stitch adjacent video frames according to the position information of the stitching corner points.

[0032] Figure 6 This is a demonstration diagram of correcting and stitching video images using a three-eye camera as an example, where cornorA-cornorD represent the four stitching corner points of a single video image.

[0033] Fig.10a and Fig.10b These are comparison pictures before and after stitching of the video images taking a three-eye camera as an example.

[0034] S103: Merge the seams between two adjacent video images.

[0035] Before fusing the seam between two adjacent video images, it is necessary to first determine whether the left and right sides of the seam represent the same object. If the left and right sides of the seam represent the same object, the seam is fused; if the left and right sides of the seam do not represent the same object, the seam is not fused, and the adjacent video images need to be re-aligned before the seam is fused.

[0036] Methods for determining whether the left and right sides of a seam represent the same object include color matching and edge detection. Color matching is to use a color matching algorithm to compare whether the colors of objects on both sides of the seam are similar or consistent. Color matching can be performed by calculating the mean, median or mode of the color; edge detection is to use an edge detection algorithm to identify the boundary of an object in a video screen. If the edge is continuous on both sides of the seam, it can be considered that the two sides may belong to the continuation of the same object.

[0037] like Figure 4 As shown, step S103 specifically includes: S401: Determine a color difference calculation range between the two adjacent video pictures.

[0038] Specifically, the color difference calculation range is the width of the area where the color difference needs to be calculated when the stitching seam of two adjacent video images is fused. The color difference calculation range is adjusted according to the resolution of the video image. The higher the resolution of the video image, the larger the color difference calculation range. For example, if the resolution of the video image is 1080P, the color difference calculation range is the width of 16 pixels of the edge of each image that needs to be fused.

[0039] Fig. 9 This is an example diagram of stitching seam fusion using a three-eye camera as an example. Taking the first video screen as an example, the narrow strip on the far right of the first video screen is the color difference calculation range, the width of the color difference calculation range is L, and the larger shadow area is the fusion range, and the width of the fusion range is w.

[0040] S402: Calculate the color difference value between two adjacent video pictures within a color difference calculation range.

[0041] The color difference between two adjacent video images refers to the color difference between two consecutive different spliced ​​images in the video sequence. Due to differences in camera parameters, the adjacent images captured by two adjacent cameras have a relatively fixed color difference as a whole, and the color difference is particularly obvious at the seam of two adjacent images. Therefore, the color difference between two adjacent images can be calculated by averaging the pixel values ​​of a narrow strip of predetermined pixels with a narrow range of width at the seam of adjacent images. Assume that for the edge of a 1080p image, all pixel values ​​in a narrow strip of 16*1080 are selected to calculate the average value of the YUV color space, and then the color difference average value within the color difference calculation range of two adjacent images is calculated.

[0042] Specifically, it is necessary to first extract the pixel value of each pixel within the color difference calculation range and convert it from the RGB color space to the YUV color space, where Y represents brightness and U and V represent chrominance. The reason for converting the RGB color space to the YUV color space is that the YUV color space separates brightness and chrominance, which can more accurately simulate human visual contrast sensitivity; in addition, in video compression and processing, YUV can independently compress brightness and chrominance components, which can simplify color difference calculation. Finally, according to the value of the YUV color space, a suitable color difference formula is used to calculate the average color difference between the pixels of adjacent video images within the color difference calculation range.

[0043] For example, within the color difference calculation range, there are a total of 16*1080 pixels, and the average value of the RGB color space of the pixels within the color difference calculation range of the first video screen is converted into the average value of the corresponding YUV color space, for example, by the following conversion algorithm: Y = 0.257R + 0.504G + 0.098B + 16 U = 0.148R - 0.291G + 0.439B + 128 V = 0.439R - 0.368G - 0.071B + 128 Assume the conversion results are: 50.55, -45.23, and 64.12; The average values ​​of the RGB color space of the pixels in the color difference calculation range of the second video screen converted into the corresponding average values ​​of the YUV color space are: 57.64, -48.73 and 69.54; The average value of the color difference in the YUV color space between the first video screen and the second video screen is calculated to be approximately equal to 7.09, -3.5, and 5.42. The color difference value ΔE between the first video screen and the second video screen is calculated based on the color difference formula.

[0044] S403: Calculate the fusion range w between the two adjacent video images according to the color difference value △E. The color difference value between the two adjacent video images is proportional to the fusion range. The larger the color difference value, the larger the fusion range. For example, the fusion range w is determined to be 120 pixels based on the color difference value △E, that is, all pixel points within 120 pixels from the joint seam in the first video image and the second video image need to adjust the value of the YUV component of the pixel.

[0045] S404: Correcting the pixel values ​​between the two adjacent video frames within the fusion range.

[0046] Specifically, the modification amount of the pixel value is calculated based on the distance d between the pixel points in the fusion range and the stitching seam and the average color difference of the YUV color space.

[0047] The calculation formula of the pixel value modification amount is:

[0048] in represents the modification amount of the pixel value within the fusion range w, w represents the fusion range of a single video screen, d represents the distance between the pixel points that need to be adjusted within the fusion range and the stitching seam, 2w represents the sum of the fusion ranges of two adjacent video screens, Represents the average color difference between adjacent video images within the color difference calculation range.

[0049] According to the modification amount of the pixel value, the pixel values ​​of the pixel points of the two adjacent video pictures within the fusion range are modified.

[0050] Specifically, when It means that when the average pixel value of the left video picture in the adjacent video picture within the color difference calculation range is subtracted from the average pixel value of the right video picture in the color difference calculation range, for the left video picture in the adjacent video picture, the modified pixel value I`=I- , for the right video picture in the adjacent video picture, the modified pixel value I`=I+

[0051] Where I` represents the pixel value of the video picture after modification, and I represents the original pixel value of the video picture.

[0052] S104: Outputting the processed panoramic image. The processed video image is output in multiple video streams, rather than being synthesized into one video stream for input. For example, when a three-eye camera is used to capture the video image at the beginning, the processed video image is also output in three streams.

[0053] Fig.10c and Fig.10d These are comparison pictures before and after fusion of the stitching seams of adjacent video images taking a three-eye camera as an example.

[0054] Figure 7 This is a panoramic synthesis flow chart taking a three-eye camera as an example. First, the three cameras are used to collect images separately, then the images of the three cameras are corrected separately, and then the adjacent video images are spliced ​​and fused. Finally, the processed video images are output in three video streams.

[0055] Exemplary Systems Correspondingly, an embodiment of the present invention also provides a multi-camera stitching system. Figure 5 : is a block diagram of a multi-camera stitching system provided by an embodiment of the present invention, such as Figure 5 As shown, the system 500 provided in this embodiment includes a correction module 501, a splicing module 502, a fusion module 503 and an output module 504, wherein: The correction module 501 is used to correct the video images collected by the multi-camera; The stitching module 502 is used to stitch the corrected video images into a panoramic image with a wider viewing angle; The fusion module 503 is used to fuse the seams between two adjacent video images; The output module 504 is used to output the processed panoramic image.

[0056] The correction module 501 specifically includes: A fixing module 505, used to fix the position and angle of each camera; A splicing corner point determination module 506 is used to calibrate each camera individually to determine the positions of four splicing corner points of the video images captured by each camera; The module 507 for determining the calibration point is used to determine the position of the calibration point according to the information of the participants; A cropping module 508, configured to crop the video image according to the splicing corner points; Obtaining rectangle module 509 obtains a rectangular picture of the same size by performing distortion correction and geometric correction on the cropped video picture according to the calibration points.

[0057] The splicing corner point determination module 506 specifically includes: The preparation module is used to prepare the same number of calibration plates according to the number of cameras. The calibration plates have calibration diagrams, such as Figure 8 The calibration diagram has calibration points 811, 812, 813 and a corner point 814; A placement module is used to place the calibration plates at specific locations; A corner point calculation module, used for calculating the position of the splicing corner point by using a splicing tool; An adjustment module, for further adjusting the stitching corner points based on the position information of the stitching corner points and the calibration map; The module for determining the splicing corner points is used to determine the positions of the final splicing corner points.

[0058] The module 507 for determining the calibration points specifically includes: A detection module is used to detect the posture of the current participants; A first selection module, for selecting a higher position calibration point when the number of standing participants is greater than the number of sitting participants; The second selection module is used to select a lower position calibration point when the number of standing participants is less than the number of sitting participants.

[0059] The splicing module 502 specifically includes: A search module 510 is used to find the same splicing corner points in corresponding positions in two adjacent video frames; The picture splicing module 511 splices two adjacent video pictures through an operator according to the information of the splicing corner points.

[0060] The fusion module 503 specifically includes: A color difference calculation range determination module 512 is used to determine a color difference calculation range between the two adjacent video pictures; A color difference calculation module 513, used to calculate the color difference between two adjacent video pictures within a color difference calculation range; A fusion range calculation module 514 is used to calculate a fusion range that needs to be fused between the two adjacent video images according to the color difference value; The correction module 515 is used to correct the pixel value between the two adjacent video frames within the fusion range.

[0061] The color difference value between the two adjacent video images is proportional to the fusion range. The larger the color difference value is, the larger the fusion range is.

[0062] The color difference value calculation module 513 specifically includes: An extraction module, used to extract corresponding pixel points within the color difference calculation range; A conversion module, used for converting the pixel value of the pixel point from the RGB color space into the YUV color space; The color difference average value calculation module is used to calculate the color difference average value between pixel points of adjacent video images within the color difference calculation range using a suitable color difference formula according to the value of the YUV color space.

[0063] The correction module 515 specifically includes: A modification amount calculation module, used for calculating the modification amount of the pixel value based on the distance between the pixel points in the fusion range and the stitching seam and the color difference average value; The calculation formula of the pixel value modification amount is:

[0064] in represents the modification amount of the pixel value within the fusion range w, w represents the fusion range of a single video screen, d represents the distance between the pixel points that need to be adjusted within the fusion range and the stitching seam, 2w represents the sum of the fusion ranges of two adjacent video screens, Represents the average color difference between adjacent video images within the color difference calculation range.

[0065] The pixel value modification module is used to modify the pixel values ​​of the pixel points of the two adjacent video images within the fusion range according to the modification amount of the pixel value.

[0066] The processed panoramic picture is outputted in a manner of dividing the processed video picture into multiple video streams and outputting them separately.

[0067] It should be noted that although the operations of the multi-camera stitching and fusion method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0068] In addition, although several devices, units or modules of the multi-camera stitching system are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules described above can be concretized in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules for concretization.

[0069] The present invention also relates to the following scheme: 1. A multi-camera splicing and fusion method, characterized in that the method comprises: Correct the video images collected by multiple cameras; Stitch the corrected video images into a panoramic image with a wider viewing angle; Merge the seams between two adjacent video images; Output the processed panoramic image.

[0070] 2. The multi-camera stitching and fusion method according to item 1 is characterized in that the step of correcting the video images collected by the multi-cameras specifically comprises: Fix the position and angle of each camera; Calibrate each camera individually to determine the positions of the four corner points of the video captured by each camera; Determine the location of the calibration point based on the information of the participants; Cropping the video image according to the splicing corner points; The cropped video images are subjected to distortion correction and geometric correction according to the calibration points to obtain rectangular images of the same size.

[0071] 3. The multi-camera stitching and fusion method according to item 2 is characterized in that the step of individually calibrating each camera to determine the positions of the four stitching corner points of the video images captured by each camera specifically includes: Prepare the same number of calibration plates according to the number of cameras, wherein the calibration plates have calibration diagrams, and the calibration diagrams have calibration points and corner points; Place the calibration plates at specific locations respectively; Calculate the position of the splicing corner point by using a splicing tool; Further adjusting the splicing corner points based on the position information of the splicing corner points and the calibration map; Determine the position of the final stitching corner point.

[0072] 4. The multi-camera stitching and fusion method according to item 2 is characterized in that the step of determining the position of the calibration point according to the information of the participants specifically includes: Detect the posture of the current participants; When the number of standing participants is greater than the number of seated participants, select a higher calibration point; When the number of standing participants is less than that of seated participants, a lower calibration point is selected.

[0073] 5. The multi-camera stitching and fusion method according to any one of items 1 to 3, characterized in that the step of stitching the corrected video images into a panoramic image with a wider viewing angle specifically comprises: Find the same splicing corner points in corresponding positions in two adjacent video images; According to the information of the splicing corner points, two adjacent video images are spliced ​​through an operator.

[0074] 6. The multi-camera stitching and fusion method according to any one of items 1 to 5, characterized in that the step of fusing the stitching seams between two adjacent video images specifically comprises: Determining a color difference calculation range between the two adjacent video pictures; Calculate the color difference value between two adjacent video images within the color difference calculation range; Calculating a fusion range between the two adjacent video images that needs to be fused according to the color difference value; Within the fusion range, the pixel values ​​between the two adjacent video frames are corrected.

[0075] 7. The multi-camera stitching and fusion method according to item 6 is characterized in that the color difference value between the two adjacent video images is proportional to the fusion range, and the larger the color difference value, the larger the fusion range.

[0076] 8. The multi-camera stitching and fusion method according to item 7 is characterized in that the step of calculating the color difference value between two adjacent video images within the color difference calculation range specifically includes: Extract corresponding pixel points within the color difference calculation range; Convert the pixel value of the pixel point from the RGB color space to the YUV color space; According to the value of the YUV color space, a suitable color difference formula is used to calculate the average color difference between pixel points of adjacent video images within a color difference calculation range.

[0077] 9. The multi-camera stitching and fusion method according to item 8 is characterized in that the step of correcting the pixel value between the two adjacent video images within the fusion range specifically includes: Calculate the modification amount of the pixel value based on the distance between the pixel point and the stitching seam within the fusion range and the average value of the color difference; The pixel values ​​of the pixel points of the two adjacent video images within the fusion range are modified according to the modification amount of the pixel value.

[0078] 10. The multi-camera stitching and fusion method according to item 9 is characterized in that the calculation formula of the modification amount of the pixel value is:

[0079] in represents the modification amount of the pixel value within the fusion range w, w represents the fusion range of a single video screen, d represents the distance between the pixel points that need to be adjusted within the fusion range and the stitching seam, 2w represents the sum of the fusion ranges of two adjacent video screens, Represents the average color difference between pixels of adjacent video images within the color difference calculation range.

[0080] 11. The multi-camera stitching and fusion method according to item 1 is characterized in that the step of outputting the processed panoramic picture specifically includes: dividing the processed video picture into multiple video streams and outputting them separately.

[0081] 12. A multi-camera stitching system, characterized in that the system comprises a correction module, a stitching module, a fusion module and an output module, wherein: The correction module is used to correct the video images collected by the multi-camera; The stitching module is used to stitch the corrected video images into a panoramic image with a wider viewing angle; The fusion module is used to fuse the joints between two adjacent video images; The output module is used to output the processed panoramic picture.

[0082] 13. The multi-camera stitching system according to item 12, wherein the correction module specifically comprises: A fixing module, used to fix the position and angle of each camera; A splicing corner point determination module is used to calibrate each camera individually and determine the positions of the four splicing corner points of the video images captured by each camera; A module for determining the calibration points is used to determine the locations of the calibration points according to the information of the participants; A cropping module, used for cropping the video image according to the splicing corner points; A rectangular module is obtained, and a rectangular image of the same size is obtained by performing distortion correction and geometric correction on the cropped video image according to the calibration points.

[0083] 14. The multi-camera stitching system according to item 13, wherein the stitching corner point determination module specifically comprises: A preparation module, used for preparing the same number of calibration plates according to the number of cameras, wherein the calibration plates have calibration diagrams, and the calibration diagrams have calibration points and corner points; A placement module is used to place the calibration plates at specific locations; A corner point calculation module, used for calculating the position of the splicing corner point by using a splicing tool; An adjustment module, used for further adjusting the splicing corner point based on the position information of the splicing corner point and a calibration map; The module for determining the splicing corner points is used to determine the positions of the final splicing corner points.

[0084] 15. The multi-camera stitching system according to item 13 is characterized in that the module for determining the calibration points specifically comprises: A detection module is used to detect the posture of the current participants; A first selection module, for selecting a higher position calibration point when the number of standing participants is greater than the number of sitting participants; The second selection module is used to select a lower position calibration point when the number of standing participants is less than the number of sitting participants.

[0085] 16. The multi-camera stitching system according to any one of items 12 to 14, wherein the stitching module specifically comprises: Find the module with To find the same splicing corner points in the corresponding positions of two adjacent video frames; The picture splicing module splices two adjacent video pictures through an operator according to the information of the splicing corner points.

[0086] 17. The multi-camera stitching system according to any one of items 12 to 16, wherein the fusion module specifically comprises: A color difference calculation range determination module is used to determine the color difference calculation range between the two adjacent video pictures; A color difference value calculation module, used to calculate the color difference value between two adjacent video images within a color difference calculation range; A fusion range calculation module, used for calculating a fusion range that needs to be fused between the two adjacent video images according to the color difference value; The correction module is used to correct the pixel value between the two adjacent video images within the fusion range.

[0087] 18. The multi-camera stitching system according to item 17 is characterized in that the color difference value between the two adjacent video images is proportional to the fusion range, and the larger the color difference value, the larger the fusion range.

[0088] 19. The multi-camera stitching system according to item 18, wherein the color difference value calculation module specifically comprises: An extraction module, used to extract corresponding pixel points within the color difference calculation range; A conversion module, used for converting the pixel value of the pixel point from the RGB color space into the YUV color space; The color difference average value calculation module is used to calculate the color difference average value between pixel points of adjacent video images within the color difference calculation range using a suitable color difference formula according to the value of the YUV color space.

[0089] 20. The multi-camera stitching system according to item 19, wherein the correction module specifically comprises: A modification amount calculation module, used for calculating the modification amount of the pixel value based on the distance between the pixel points in the fusion range and the stitching seam and the color difference average value; The pixel value modification module is used to modify the pixel values ​​of the pixel points of the two adjacent video images within the fusion range according to the modification amount of the pixel value.

[0090] 21. The multi-camera stitching system according to item 20 is characterized in that the calculation formula of the modification amount of the pixel value is:

[0091] in represents the modification amount of the pixel value within the fusion range w, w represents the fusion range of a single video screen, d represents the distance between the pixel points that need to be adjusted within the fusion range and the stitching seam, 2w represents the sum of the fusion ranges of two adjacent video screens, Represents the average color difference between pixels of adjacent video images within the color difference calculation range.

[0092] 22. The multi-camera stitching system according to item 12 is characterized in that the step of outputting the processed panoramic picture specifically includes a sub-module for dividing the processed video picture into multiple video streams and outputting them separately.

[0093] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. A multi-camera splicing and fusion method, characterized in that: The method comprises: Correct the video images collected by multiple cameras; Stitch the corrected video images into a panoramic image with a wider viewing angle; Merge the seams between two adjacent video images; Output the processed panoramic image.

2. The multi-camera splicing and fusion method according to claim 1, characterized in that: The step of correcting the video images collected by the multi-cameras specifically includes: Fix the position and angle of each camera; Calibrate each camera individually to determine the positions of the four corner points of the video captured by each camera; Determine the location of the calibration point based on the information of the participants; Cropping the video image according to the splicing corner points; The cropped video images are subjected to distortion correction and geometric correction according to the calibration points to obtain rectangular images of the same size.

3. The multi-camera splicing and fusion method according to claim 2 is characterized in that: The step of individually calibrating each camera to determine the positions of four splicing corner points of the video images captured by each camera specifically includes: Prepare the same number of calibration plates according to the number of cameras, wherein the calibration plates have calibration diagrams, and the calibration diagrams have calibration points and corner points; Place the calibration plates at specific locations respectively; Calculate the position of the splicing corner point by using a splicing tool; Further adjusting the stitching corner points based on the position information of the stitching corner points and the calibration map; Determine the position of the final stitching corner point.

4. The multi-camera splicing and fusion method according to claim 2, characterized in that: The step of determining the position of the calibration point according to the information of the participants specifically includes: Detect the posture of the current participants; When the number of standing participants is greater than the number of seated participants, select a higher calibration point; When the number of standing participants is less than that of seated participants, a lower calibration point is selected.

5. The multi-camera stitching and fusion method according to any one of claims 1 to 3, characterized in that: The step of stitching the corrected video images into a panoramic image with a wider viewing angle specifically includes: Find the same splicing corner points in corresponding positions in two adjacent video images; According to the information of the splicing corner points, two adjacent video images are spliced ​​through an operator.

6. A multi-camera splicing system, characterized in that: The system includes a correction module, a splicing module, a fusion module and an output module, wherein: The correction module is used to correct the video images collected by the multi-camera; The stitching module is used to stitch the corrected video images into a panoramic image with a wider viewing angle; The fusion module is used to fuse the joints between two adjacent video images; The output module is used to output the processed panoramic picture.

7. The multi-camera stitching system according to claim 6, characterized in that: The correction module specifically includes: A fixing module, used to fix the position and angle of each camera; A splicing corner point determination module is used to calibrate each camera individually and determine the positions of the four splicing corner points of the video images captured by each camera; A module for determining the calibration points is used to determine the positions of the calibration points according to the information of the participants; A cropping module, used for cropping the video image according to the splicing corner points; A rectangular module is obtained, and a rectangular image of the same size is obtained by performing distortion correction and geometric correction on the cropped video image according to the calibration points.

8. The multi-camera stitching system according to claim 7, characterized in that: The module for determining the splicing corner points specifically includes: A preparation module, used for preparing the same number of calibration plates according to the number of cameras, wherein the calibration plates have calibration diagrams, and the calibration diagrams have calibration points and corner points; A placement module is used to place the calibration plates at specific locations; A corner point calculation module, used for calculating the position of the splicing corner point by using a splicing tool; An adjustment module, for further adjusting the stitching corner points based on the position information of the stitching corner points and the calibration map; The module for determining the splicing corner points is used to determine the positions of the final splicing corner points.

9. The multi-camera stitching system according to claim 7, characterized in that: The module for determining the calibration points specifically includes: A detection module is used to detect the posture of the current participants; A first selection module, for selecting a higher position calibration point when the number of standing participants is greater than the number of sitting participants; The second selection module is used to select a lower position calibration point when the number of standing participants is less than the number of sitting participants.

10. The multi-camera stitching system according to any one of claims 6 to 9, characterized in that: The splicing module specifically includes: A search module is used to find the same splicing corner points in corresponding positions in two adjacent video images; The picture splicing module splices two adjacent video pictures through an operator according to the information of the splicing corner points.

Images