Video image stitching method and apparatus, terminal device, and storage medium

Abstract

The present application is suitable for the technical field of video processing, and provides a video image stitching method and apparatus, a terminal device, and a storage medium. The video image stitching method comprises: acquiring a first video image and a second video image which are to be stitched; performing homography transformation on the second video image and then determining overlapping regions, wherein the overlapping regions include a first overlapping region and a second overlapping region; calculating the similarities between corresponding overlapping sub-regions in the first overlapping region and corresponding overlapping sub-regions in the second overlapping region; on the basis of overlapping sub-regions and the similarities, unifying an image tone of the overlapping regions to obtain a target overlapping region; determining an optimal seam line of the target overlapping region; and on the basis of the optimal seam line, performing video image stitching on the first video image and the second video image to obtain a stitched video image.

Description

Video image splicing method and device, terminal equipment and storage medium

[0001] Cross-reference to Related Applications

[0002] This application claims priority to Chinese Patent Application No. 202410645284.3, filed May 21, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the field of video processing, in particular to a video image splicing method, device, terminal equipment and storage medium. BACKGROUND

[0004] Ordinary cameras often cannot completely include the required picture in an excessively large scene. Real-time splicing technology for videos can splice the pictures of multiple cameras into a complete image presented to the user, which can compensate for the defect of not being able to completely include the required picture. Real-time splicing of pictures of multiple cameras can significantly improve the picture presentation quality and visual experience in application scenarios such as video conferencing, online teaching, unmanned aerial photography, and intelligent driving.

[0005] In related technologies, objects in images captured by different cameras that are closer to the cameras can be misaligned, resulting in misalignment or artifacts of objects located in the overlapping area of the images after splicing, which not only adversely affects the visual experience of the video images, but also is not conducive to accurate extraction of the content of the video images by the user.

[0006] Therefore, how to accurately and effectively splice video images to avoid misalignment or artifacts is a problem that needs to be considered. SUMMARY

[0007] The embodiments of the present application provide a video image splicing method, device, terminal equipment and storage medium, which can accurately and effectively splice video images to avoid misalignment or artifacts, optimize the visual experience of the user for the video images, and improve the user experience.

[0008] In a first aspect, the embodiments of the present application provide a video image splicing method, comprising:

[0009] obtaining a first video image and a second video image to be spliced;

[0010] performing homographic transformation on the second video image, and determining an overlapping area of the first video image and the second video image after homographic transformation, the overlapping area including a first overlapping area of the first video image overlapping with the second video image, and a second overlapping area of the second video image overlapping with the first video image;

[0011] perform region division on the first and second overlap regions respectively, and calculate similarity of corresponding overlap sub-regions in the first and second overlap regions after the region division;

[0012] unify image tones of the overlap regions based on the overlap sub-regions and the similarity to obtain a target overlap region;

[0013] determine an optimal seam line of the target overlap region;

[0014] perform video image stitching on the first and second video images according to the optimal seam line to obtain a stitched video image.

[0015] In a possible implementation manner of the first aspect, the overlap sub-regions include first overlap sub-regions and second overlap sub-regions.

[0016] The performing region division on the first and second overlap regions respectively, and calculating similarity of corresponding overlap sub-regions in the first and second overlap regions after the region division, includes:

[0017] perform region division on the first and second overlap regions respectively to obtain a plurality of first overlap sub-regions and a plurality of second overlap sub-regions;

[0018] determine whether there is edge information in the first overlap sub-region and the corresponding second overlap sub-region;

[0019] if there is edge information, calculate binary similarity between the first overlap sub-region and the corresponding second overlap sub-region;

[0020] if there is no edge information, calculate color similarity between the first overlap sub-region and the corresponding second overlap sub-region.

[0021] In a possible implementation manner of the first aspect, the determining whether there is edge information in the first overlap sub-region and the corresponding second overlap sub-region includes:

[0022] obtain a total sum of non-zero pixel values in the first and second overlap sub-regions;

[0023] if the total sum of non-zero pixel values is less than a preset pixel threshold, it is determined that there is no edge information in the first and second overlap sub-regions;

[0024] if the total sum of non-zero pixel values is greater than or equal to the preset pixel threshold, it is determined that there is edge information in the first and second overlap sub-regions.

[0025] In a possible implementation manner of the first aspect, the overlapping sub-regions include a first overlapping sub-region and a second overlapping sub-region; and the uniformizing the image color tones of the overlapping region based on the overlapping sub-regions and the similarity to obtain a target overlapping region includes:

[0026] adding a mask to the first overlapping sub-region and the second overlapping sub-region in the first overlapping region and the second overlapping region that have a similarity less than a preset similarity threshold;

[0027] performing color migration on the second overlapping region based on target parameter information to uniformize the image color tones of all the first overlapping sub-regions and corresponding second overlapping sub-regions, to obtain a target overlapping region, where the target parameter information is determined according to the first overlapping sub-region without the mask and the second overlapping sub-region without the mask.

[0028] In a possible implementation manner of the first aspect, the determining the optimal seam line of the target overlapping region includes:

[0029] if the first video image and the second video image are the first frame images in the video stream, determining the optimal seam line of the target overlapping region according to a preset seam line searching algorithm;

[0030] if the first video image and the second video image are non-first frame images in the video stream, obtaining dynamic information in the target overlapping region, and determining the optimal seam line of the target overlapping region according to the dynamic information.

[0031] In a possible implementation manner of the first aspect, the dynamic information includes a foreground expansion item, and the obtaining the dynamic information in the target overlapping region and determining the optimal seam line of the target overlapping region according to the dynamic information includes:

[0032] obtaining an energy value of the foreground expansion item in the target overlapping region, and counting a number of foreground expansion items with an energy value greater than 0 on a seam line of a previous frame spliced video image;

[0033] if the number of foreground expansion items does not satisfy a preset expansion energy value condition, using the seam line of the previous frame spliced video image as the optimal seam line of the current target overlapping region;

[0034] if the number of foreground expansion items satisfies the preset expansion energy value condition, determining the optimal seam line of the target overlapping region according to the preset seam line searching algorithm.

[0035] In a possible implementation manner of the first aspect, the determining the optimal seam line of the target overlapping region according to the preset seam line searching algorithm includes:

[0036] obtaining an energy matrix of the target overlap region;

[0037] dividing the target overlap region into a plurality of target overlap sub-regions;

[0038] determining a comprehensive energy average value of each target overlap sub-region based on the energy matrix;

[0039] constructing a rough energy matrix according to the comprehensive energy average value of each target overlap sub-region;

[0040] determining an optimal seam line of the target overlap region based on the rough energy matrix and a dynamic programming algorithm.

[0041] In a second aspect, an embodiment of the present application provides a video image splicing device, comprising:

[0042] a video image obtaining unit, configured to obtain a first video image and a second video image to be spliced;

[0043] an overlap region determining unit, configured to perform homographic transformation on the second video image, and determine an overlap region of the first video image and the second video image after homographic transformation, the overlap region comprising a first overlap region of the first video image overlapping with the second video image, and a second overlap region of the second video image overlapping with the first video image;

[0044] a region similarity calculating unit, configured to divide the first overlap region and the second overlap region respectively, and calculate a similarity of corresponding overlap sub-regions in the first overlap region and the second overlap region after division;

[0045] a hue unifying unit, configured to unify image hues of the overlap region based on the overlap sub-regions and the similarity, to obtain a target overlap region;

[0046] a seam line determining unit, configured to determine an optimal seam line of the target overlap region;

[0047] a video image splicing unit, configured to perform video image splicing on the first video image and the second video image according to the optimal seam line, to obtain a spliced video image.

[0048] In a third aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the video image splicing method of the first aspect.

[0049] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the video image splicing method according to the first aspect.

[0050] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the video image splicing method according to the first aspect.

[0051] In the embodiment of the present application, the first video image and the second video image to be spliced are obtained, a homographic transformation is performed on the second video image, and an overlapping region of the first video image and the second video image after the homographic transformation is determined. The overlapping region includes a first overlapping region and a second overlapping region. The first overlapping region and the second overlapping region are respectively divided into regions, and the similarity of corresponding overlapping sub-regions in the first overlapping region and the second overlapping region after the division is calculated. Then, based on the overlapping sub-regions and the similarity, the image color tone of the overlapping region is unified to obtain a target overlapping region. The best seam line of the target overlapping region is determined, and the first video image and the second video image are spliced according to the best seam line. In the present application, the homographic transformation is performed on the second video image, and the image perspective is unified. The image splicing in the same perspective helps to reduce the occurrence of misplacement or artifacts. The similarity of the corresponding overlapping sub-regions in the two video images is used to unify the color tone of the overlapping region. The effectiveness of the color tone unification is improved while the color tone of the spliced image is unified. Thus, the video image splicing can be accurately and effectively implemented, the user's visual perception of the video image is optimized, and the user experience is improved. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiments or related technical descriptions will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0053] FIG. 1 is an implementation flowchart of the video image splicing method according to an embodiment of the present application;

[0054] FIG. 2 is a specific implementation flowchart of step S103 in the video image splicing method according to an embodiment of the present application;

[0055] FIG. 3 is a specific implementation flowchart of edge information judgment in the video image splicing method according to an embodiment of the present application;

[0056] FIG. 4 is a flow chart illustrating a specific implementation of step S104 in the method for stitching video images according to an embodiment of the present application;

[0057] FIG. 5 is a flow chart illustrating a specific implementation of step S105 in the method for stitching video images according to an embodiment of the present application;

[0058] FIG. 6 is a flow chart illustrating a specific implementation of determining the optimal seam line of the target overlapping region in the method for stitching video images according to an embodiment of the present application;

[0059] FIG. 6.1 is a schematic diagram illustrating the determination of the seam line selection region in the method for stitching video images according to an embodiment of the present application;

[0060] FIG. 6.2 is a schematic diagram illustrating the arrangement of the sub-regions after the division of the seam line selection region in the method for stitching video images according to an embodiment of the present application;

[0061] FIG. 7 is a flow chart illustrating a specific implementation of determining the optimal seam line of the non-first frame image in the method for stitching video images according to an embodiment of the present application;

[0062] FIG. 8 is a block diagram illustrating the structure of a video image stitching apparatus according to an embodiment of the present application;

[0063] FIG. 9 is a schematic diagram illustrating a terminal device according to an embodiment of the present application. DETAILED DESCRIPTION

[0064] In the following description, specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, persons skilled in the art will understand that the present application can be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure aspects of the present application.

[0065] It should be understood that the term "comprising" as used in the specification and in the claims indicates the presence of the recited features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0066] It should also be understood that the term "and / or" as used in the specification and in the claims indicates any combination of one or more of the associated listed items and all possible combinations of the items.

[0067] As used in this application and the appended claims, the term “if’ can be construed to mean “when” or “once” or “in response to a determination” or “in response to a detection” that a stated condition precedent has been met, depending on the context. Similarly, the phrase “if it is determined” or “if [a stated condition or event] is detected” can be construed to mean “once it is determined” or “in response to a determination” or “once [the stated condition or event] is detected” or “in response to the detection [of the stated condition or event],” depending on the context.

[0068] In addition, in the description of the present application and the appended claims, the terms “first”, “second”, “third”, etc. are used only to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0069] Reference in the specification to “one embodiment” or “some embodiments” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrases “in one embodiment”, “in some embodiments”, “in other embodiments”, “in additional embodiments”, etc. in various places in the specification are not necessarily all referring to the same embodiment, although they can. The terms “comprising”, “including”, “having” and their variants mean “including but not limited to”, unless otherwise expressly specified and limited by the context.

[0070] The video image splicing method provided by the embodiments of the present application is suitable for various types of intelligent terminals and servers that need to perform video image splicing. The intelligent terminals can specifically include mobile phones, tablet computers, wearable devices, notebook computers, desktop computers, etc. The embodiments of the present application do not make any limitation on the specific types of intelligent terminals.

[0071] FIG. 1 shows the implementation flow of the video image splicing method provided by the embodiments of the present application, which includes steps S101 to S106. The specific implementation principles of each step are as follows:

[0072] Step S101: Obtain a first video image and a second video image to be spliced.

[0073] The first video image is a video image in a first video stream collected in real time, and the second video image is a video image in a second video stream collected in real time.

[0074] Taking a left-right binocular camera as an example, two side video streams are collected in real time by the binocular camera. The first camera collects the first video stream in real time, and the second camera collects the second video stream in real time. Since the orientations of the first camera and the second camera are consistent, there will usually be an overlapping area in the first video image and the second video image.

[0075] In the embodiment, the first video stream and the second video stream are collected simultaneously, and the first video image in the first video stream and the second video image in the second video stream are video images collected at the same time. The video images in the first video stream and the second video stream correspond to each other according to the collection time.

[0076] Step S102: Homographic transformation is performed on the second video image, and an overlapping region of the first video image and the second video image after the homographic transformation is determined.

[0077] The overlapping region includes a first overlapping region in the first video image overlapping with the second video image, and a second overlapping region in the second video image overlapping with the first video image.

[0078] In the embodiment, the homographic transformation is performed on the second video image based on the perspective of the first video image, and the perspectives of the first video image and the second video image are unified, which is helpful to accurately splice the images and reduce the generation of misplacement and artifacts.

[0079] In a possible implementation, a homographic matrix between the first video image and the second video image is calculated, the homographic transformation is performed on the second video image based on the homographic matrix, and then the overlapping region of the first video image and the second video image after the homographic transformation is determined.

[0080] The overlapping region includes the first overlapping region and the second overlapping region, the first overlapping region is determined based on the pixels in the overlapping region in the first video image, and the second overlapping region is determined based on the pixels in the overlapping region in the second video image. The sizes of the first overlapping region and the second overlapping region are consistent.

[0081] Step S103: The first overlapping region and the second overlapping region are respectively divided into regions, and the similarity of corresponding overlapping sub-regions in the divided first overlapping region and second overlapping region is calculated.

[0082] In a possible implementation, before the first overlapping region and the second overlapping region are respectively divided into regions, the first overlapping region and the second overlapping region are subjected to Gaussian filter denoising and edge detection processing after being converted into grayscale images, to obtain the processed first overlapping region and second overlapping region, and then the processed first overlapping region and second overlapping region are respectively divided into regions, which is helpful to improve the accuracy of image processing.

[0083] As a possible implementation of the present application, the overlapping sub-region includes the first overlapping sub-region and the second overlapping sub-region. FIG. 2 shows a specific implementation process of step S103 in the video image splicing method provided by the embodiment of the present application, which is described in detail as follows A1-A4.

[0084] A1: performing region division on the first overlapping region and the second overlapping region respectively to obtain a plurality of first overlapping sub-regions and a plurality of second overlapping sub-regions.

[0085] The first overlapping region is divided into a plurality of first overlapping sub-regions, and the second overlapping region is divided into a plurality of second overlapping sub-regions. In this embodiment, the division standards of the first overlapping region and the second overlapping region are unified, and the first overlapping region and the second overlapping region are divided according to the specified division number and the specified division mode. After the region division, the first overlapping sub-regions and the second overlapping sub-regions at the same position correspond to each other.

[0086] For example, the image width of the first overlapping region and the second overlapping region is w, the image height is h, the preset division number of each row / column is n, and after the region division, the width and the height of the first overlapping sub-region and the second overlapping sub-region are still the same, the width of each overlapping sub-region is w / n, and the height is h / n, and n is generally an integer close to 10. After the region division of the first overlapping region, a first overlapping sub-region set R1 is obtained, R1={r1,...,rm}, where r1,...,rm are m first overlapping sub-regions. After the region division of the second overlapping region, a second overlapping sub-region set R2 is obtained, R2={r1',...,rm'}, where r1',...,rm' are m second overlapping sub-regions. The first overlapping sub-region and the second overlapping sub-region correspond to each other based on the position in the overlapping region.

[0087] A2: determining whether there is edge information in the first overlapping sub-region and the corresponding second overlapping sub-region.

[0088] The edge information is the feature information of the image edge, the image edge is important feature information of the image, is a discontinuous place of the characteristic (such as pixel gray, texture, etc.) distribution in the image, and is a pixel set with step change or ridge-shaped change around the image. The edge part of the image concentrates most of the information of the image, and the edge structure and characteristics of an image are often an important part of determining the characteristics of the image. Another definition of the image edge is a set of pixels with discontinuous pixel gray around the image. Edges exist widely between objects and backgrounds, between objects, and edge information is an important feature of image segmentation, image understanding and image recognition.

[0089] As a possible implementation manner of the present application, as shown in FIG. 3, determining whether there is edge information in the first overlapping sub-region and the corresponding second overlapping sub-region includes A21-A23.

[0090] A21: obtaining the total sum of non-zero pixel values in the first overlapping sub-region and the second overlapping sub-region.

[0091] A22: If the sum of non-zero pixel values is less than the preset pixel threshold value, it is determined that there is no edge information in the first overlapping sub-region and the second overlapping sub-region.

[0092] A23: If the sum of non-zero pixel values is greater than or equal to the preset pixel threshold value, it is determined that there is edge information in the first overlapping sub-region and the second overlapping sub-region.

[0093] The preset pixel threshold value is a positive integer close to 0.

[0094] In this embodiment, whether there is edge information in the first overlapping sub-region and the corresponding second overlapping sub-region is judged according to the sum of non-zero pixel values. When the sum of non-zero pixel values in the first overlapping sub-region and the second overlapping sub-region is less than the preset pixel threshold value, it is determined that there is no edge information in the group of overlapping sub-regions. On the contrary, when the sum of non-zero pixel values in the first overlapping sub-region and the second overlapping sub-region is greater than or equal to the preset pixel threshold value, it is determined that there is edge information in the group of overlapping sub-regions.

[0095] A3: If there is edge information, the binary similarity between the first overlapping sub-region and the corresponding second overlapping sub-region is calculated.

[0096] In a possible implementation, the binary similarity between the first overlapping sub-region and the second overlapping sub-region is calculated, including:

[0097] The sum of the first number is calculated, which is the sum of the number of non-zero pixel points in the first overlapping sub-region and the second overlapping sub-region. The sum of the second number is calculated, which is the sum of the number of non-zero pixel points after the pixel values of the corresponding positions in the first overlapping sub-region and the second overlapping sub-region are XORed. The first ratio is obtained, which is the ratio of the sum of the second number to the sum of the first number.

[0098] The first overlapping sub-region is cropped according to a preset cropping ratio to obtain a first cropped sub-region, and the size of the first cropped sub-region is proportional to the resolution of the overlapping region.

[0099] The first coordinate position (0, j) is taken as the top left corner of the cropped image, and an image of the same size as the first cropped sub-region is cropped in the second overlapping sub-region to obtain a second cropped sub-region. The sum of the third number is calculated, which is the sum of the number of non-zero pixels after the pixel values of the corresponding positions in the first cropped sub-region and the second cropped sub-region are XORed. The second ratio is obtained, which is the ratio of the sum of the third number to the sum of the first number.

[0100] Taking the second coordinate position (j, 0) as the top-left corner of the cropped image, an image of the same size as the first cropped sub-region is cropped in the second overlapping sub-region to obtain a third cropped sub-region. A fourth number sum is calculated, which is the total number of non-zero pixels after the XOR operation of the pixel values at the corresponding positions in the first and third cropped sub-regions. A third ratio is obtained, which is the ratio of the fourth number sum to the first number sum.

[0101] Taking the third coordinate position (2j, j) as the top-left corner of the cropped image, an image of the same size as the first cropped sub-region is cropped in the second overlapping sub-region to obtain a fourth cropped sub-region. A fifth number sum is calculated, which is the total number of non-zero pixels after the XOR operation of the pixel values at the corresponding positions in the first and fourth cropped sub-regions. A fourth ratio is obtained, which is the ratio of the fifth number sum to the first number sum.

[0102] Taking the fourth coordinate position (j, 2j) as the top-left corner of the cropped image, an image of the same size as the first cropped sub-region is cropped in the second overlapping sub-region to obtain a fifth cropped sub-region. A sixth number sum is calculated, which is the total number of non-zero pixels after the XOR operation of the pixel values at the corresponding positions in the first and fifth cropped sub-regions. A fifth ratio is obtained, which is the ratio of the sixth number sum to the first number sum.

[0103] Based on the first ratio, the second ratio, the third ratio, the fourth ratio, and the fifth ratio, the binary similarity between the first overlapping sub-region and its corresponding second overlapping sub-region is determined, denoted as bsml.

[0104] Specifically, (1-MIN(S0, S1, S2, S3, S4)) is determined as the binary similarity between the first overlapping sub-region and the second overlapping sub-region, S0 is the first ratio, S1 is the second ratio, S2 is the third ratio, S3 is the fourth ratio, and S4 is the fifth ratio.

[0105] A4: If there is no edge information, the color similarity between the first overlapping sub-region and its corresponding second overlapping sub-region is calculated.

[0106] In this embodiment, the image of the first overlapped sub-region at the corresponding position in the first overlapped region is intercepted and recorded as a first intercepted image; the image of the second overlapped sub-region at the corresponding position in the second overlapped region is intercepted and recorded as a second intercepted image; the average value of the pixel value difference of the corresponding pixel points in the first and second intercepted images is calculated respectively, and recorded as the first channel pixel value difference average value a1, the second channel pixel value difference average value a2, and the third channel pixel value difference average value a3 respectively; and the color similarity between the first overlapped sub-region and the second overlapped sub-region is determined according to the first channel pixel value difference average value a1, the second channel pixel value difference average value a2, and the third channel pixel value difference average value a3.

[0107] Specifically, (1-MAX(a1, a2, a3) / 255) is determined as the color similarity of the first overlapped sub-region and the second overlapped sub-region, and recorded as csml.

[0108] Step S104: based on the overlapped sub-regions and the similarity, the image color tone of the overlapped region is unified to obtain a target overlapped region.

[0109] As a possible implementation of the present application, FIG. 4 shows a specific implementation process of S104 in the video image splicing method provided by the embodiments of the present application, which is described in detail as follows B1-B2.

[0110] B1: masks are added to the first overlapped sub-region and the second overlapped sub-region in the first overlapped region and the second overlapped region with a similarity less than a preset similarity threshold. The similarity includes binary similarity and color similarity, and the preset similarity threshold includes a preset binary similarity threshold and a preset color similarity threshold.

[0111] If there is edge information in the first overlapped sub-region and the second overlapped sub-region, it is determined whether the binary similarity of the first overlapped sub-region and the second overlapped sub-region is less than the preset binary similarity threshold, and masks are added to the first overlapped sub-region and the second overlapped sub-region with a binary similarity less than the preset binary similarity threshold.

[0112] If there is no edge information in the first overlapped sub-region and the second overlapped sub-region, it is determined whether the color similarity of the first overlapped sub-region and the second overlapped sub-region is less than the preset color similarity threshold, and masks are added to the first overlapped sub-region and the second overlapped sub-region with a color similarity less than the preset color similarity threshold.

[0113] B2: color migration is performed on the second overlapped region based on target parameter information, the image color tone of all the first overlapped sub-regions and the corresponding second overlapped sub-regions is unified, and a target overlapped region is obtained. The target parameter information is determined according to the first overlapped sub-region without a mask and the second overlapped sub-region without a mask.

[0114] In the embodiments of the present application, the target parameter information includes color average value and standard deviation; the color average value and standard deviation in all first sub-overlapping regions and second sub-overlapping regions without added masks are calculated; color migration is performed on the entire second overlapping region based on the color average value and standard deviation; the image tone of all first sub-overlapping regions and corresponding second sub-overlapping regions is unified, that is, the image tone of the first overlapping region and the second overlapping region is unified, to obtain a target overlapping region.

[0115] In some embodiments, a scaling factor can be calculated according to the color average value and standard deviation; each pixel value of each pixel in the second overlapping region is adjusted based on the scaling factor to realize color migration and unify the image tone of all first sub-overlapping regions and corresponding second sub-overlapping regions, thereby obtaining a target overlapping region.

[0116] The specific manner of color migration can refer to related technologies, for example, Reinhard color migration is adopted.

[0117] In some embodiments, color migration is performed on the second video image according to the parameter information of the second sub-overlapping region without added masks to unify the image tone of the first video image and the second video image. When video image splicing is performed subsequently, the first video image and the second video image after color migration are spliced.

[0118] In a possible embodiment, before color migration is performed, the first overlapping region and the second overlapping region are converted to the lαβ color space, or the first video image and the second video image are converted to the lαβ color space; the parameter information in all second sub-overlapping regions without added masks in the lαβ color space is calculated; and color migration is performed on the second overlapping region or the second video image based on the parameter information in the lαβ color space, which can improve the effect of color migration.

[0119] Due to light and angle problems, color difference exists in the shooting of a picture by a binocular camera. In the embodiments, a mask is added to a region with a similarity lower than a preset similarity threshold, and then color migration is performed on the second overlapping region based on the non-masked region to unify the tone of the overlapping region and reduce the adverse effect of shooting color on video image splicing, which can guarantee the uniformity of the tone of the spliced image and improve the effectiveness of tone unification.

[0120] Step S105: determining the best seam line of the target overlapping region.

[0121] As a possible embodiment of the present application, FIG. 5 shows a specific implementation process of step S105 of the video image splicing method provided in the embodiments of the present application, which is described in detail as follows.

[0122] S201: If the first video image and the second video image are the first frame images in the video streams, determining the optimal seam line of the target overlap region according to a preset seam line searching algorithm.

[0123] When the first video image and the second video image are the first frame images in the first video stream and the second video stream respectively, the optimal seam line of the target overlap region is directly determined according to the preset seam line searching algorithm.

[0124] As a possible implementation of the present application, FIG. 6 shows a specific implementation process of determining the optimal seam line of the target overlap region according to a preset seam line searching algorithm in the video image splicing method provided by the embodiments of the present application, which is described in detail as follows C1-C5.

[0125] C1: Obtain the energy matrix of the target overlap region.

[0126] In the embodiment, the energy function is constructed in advance, and the energy values of each term in the energy function are weighted and averaged to obtain the comprehensive energy value of each pixel point in the target overlap region.

[0127] In a possible implementation, the energy function includes a color term, a structure term, a foreground term and a foreground expansion term. The color term is the gray difference of the pixel points at the same position in the two video images; the structure term is to find the edge structure commonly possessed by the two video images in the target overlap region; the foreground term is the foreground mask image constructed by the background modeling of the first video image and the second video image; and the foreground expansion term is obtained by radiating the foreground in the foreground term to the outside of the contour, wherein the foreground region has the maximum energy value, in the neighborhood outside the foreground contour, the closer to the foreground contour, the greater the energy value of the foreground expansion term, and the energy value of the foreground expansion term at the edge of the neighborhood is 0. The foreground term and the foreground expansion term are used to help better calculate the position of the updated seam line when there is a moving object passing through the overlap region in the subsequent video image frame, so as to help update the seam line when needed. The energy values of the color term, the structure term, the foreground term and the foreground expansion term are weighted and averaged to obtain the comprehensive energy value, and then the energy matrix of the target overlap region is constructed according to the comprehensive energy value of each pixel point position in the target overlap region.

[0128] C2: Divide the target overlap region to obtain a plurality of target overlap sub-regions. The division manner can refer to the division manner of the overlap sub-region described above.

[0129] In a possible implementation, before the target overlapping region is regionally divided, a preset region selection algorithm is used to determine a seam line selection region in the target overlapping region; and the target overlapping region is regionally divided to obtain a plurality of target overlapping sub-regions. By determining the seam line selection region in the target overlapping region, the range of searching for the optimal seam line is reduced, which is conducive to improving the searching efficiency and further improving the efficiency of video image splicing.

[0130] For example, as shown in FIG. 6.1, a portion with a proportion t1 in the middle of the target overlapping region is determined as the seam line selection region, with the center line of the target overlapping region as a reference. t1 is a preset proportion threshold, and t1 is greater than 1 / 2.

[0131] For example, the seam line selection region is divided by n partitions per row / column, to obtain a set R3 of selected sub-regions after division in the seam line selection region, R3={r00, r01, …, r10, r11, …, r(n-1)0, …, r(n-1)(n-1)}, where the division number n should not be too large, and is preferably an integer near 10. When n=10, rn1n2 is arranged in the seam line selection region as shown in FIG. 6.2. In this case, the overall effect of the selected seam line is better.

[0132] C3: Based on the energy matrix, the average value of the comprehensive energy of each target overlapping sub-region is determined.

[0133] C4: According to the average value of the comprehensive energy of each target overlapping sub-region, a rough energy matrix is constructed.

[0134] C5: Based on the rough energy matrix and the dynamic programming algorithm, the optimal seam line of the target overlapping region is determined. The dynamic programming algorithm is used to search for the optimal seam line.

[0135] In this embodiment, a seam line with the minimum average value of the comprehensive energy of all pixels on the line, i.e., a seam line with the minimum difference, is found in the target overlapping region as an initial seam line. Each element on the initial seam line is traversed to find the corresponding target overlapping sub-region (or seam line selection region) corresponding to the rough energy matrix; and the dynamic programming algorithm is used again to calculate the local optimal seam line in each corresponding target overlapping sub-region. The local optimal seam lines in all corresponding target overlapping sub-regions constitute the optimal seam line of the target overlapping region. The initial seam line is obtained according to the rough energy matrix, the rough energy matrix is related to the divided overlapping sub-regions, and the elements of the initial seam line are the overlapping sub-regions located on the initial seam line.

[0136] For example, the target overlapping region is divided into n x n target overlapping sub-regions, a new n x n rough energy matrix is constructed according to the average value of the comprehensive energy of each target overlapping sub-region, the rough optimal seam line is calculated based on the dynamic programming algorithm on the rough energy matrix, each element on the rough optimal seam line is traversed to find the selected sub-region in R3 corresponding to the rough energy matrix as the candidate sub-region, and the dynamic programming algorithm is used again in each candidate sub-region to calculate the local optimal seam line in each candidate sub-region, and the local optimal seam line of all candidate sub-regions is used to construct the optimal seam line of the target overlapping region.

[0137] S202: If the first video image and the second video image are non-first frame images in the video stream, dynamic information in the target overlapping region is acquired, and the optimal seam line of the target overlapping region is determined according to the dynamic information. The dynamic information refers to information that can identify a dynamic object in the target overlapping region.

[0138] In the embodiments of the present application, the first video image and the second video image acquired for the first time after the terminal device starts to perform the video image splicing method each time can be recorded as the first frame image in the video stream, that is, the first frame video image, and the subsequent acquired first video image and second video image are recorded as non-first frame images, specifically, the second frame video image, the third frame video image, and so on.

[0139] In the embodiments of the present application, for the first frame video image, the optimal seam line of the target overlapping region is directly determined according to the preset seam line searching algorithm. For the non-first frame video image, it is needed to determine whether the current optimal seam line needs to be updated according to the dynamic information. The current optimal seam line refers to the seam line of the previous frame spliced video image before the current first video image and the second video image are spliced.

[0140] As a possible implementation manner of the present application, the above dynamic information includes a foreground inflation term in the energy function. As shown in FIG. 7, the dynamic information in the target overlapping region is acquired, and the optimal seam line of the target overlapping region is determined according to the dynamic information, including D1-D3.

[0141] D1: The energy value of the foreground inflation term in the target overlapping region is acquired, and the number of foreground inflation terms with energy value greater than 0 on the seam line of the previous frame spliced video image is counted. Specifically, the number of foreground inflation terms with energy value greater than 0 in the seam line of the previous frame spliced video image is counted.

[0142] D2: If the number of foreground inflation terms does not satisfy the preset inflation energy value condition, the seam line of the previous frame spliced video image is used as the optimal seam line of the current target overlapping region.

[0143] D3: if the number of foreground expansion items meets the preset expansion energy value condition, determining the optimal seam line of the target overlap region according to the preset seam line searching algorithm.

[0144] The preset expansion energy value condition is set according to the foreground expansion item. In this embodiment, the number of pixels with energy values greater than 0 in the seam line of the previous frame spliced video image in the target overlap region is determined. If the number of pixels is greater than or equal to a preset number threshold, it is determined that the energy value of the foreground expansion item meets the preset expansion energy value condition. At this time, the seam line of the previous frame spliced video image needs to be updated, and the optimal seam line of the target overlap region is determined again according to the preset seam line searching algorithm. If the number of pixels is less than the preset number threshold, it is determined that the energy value of the foreground expansion item does not meet the preset expansion energy value condition, and the seam line of the previous frame spliced video image is directly used as the optimal seam line of the current target overlap region.

[0145] In this embodiment, whether the seam line needs to be updated is determined according to the number of foreground expansion items with non-zero energy values across the seam line of the previous frame spliced video image. This can reduce the possibility of misalignment and artifacts caused by dynamic objects interfering with the overlap region, thereby accurately and effectively implementing video image splicing, optimizing user perception of video images, and improving user experience.

[0146] Step S106: performing video image splicing on the first video image and the second video image according to the optimal seam line to obtain a spliced video image.

[0147] In this embodiment, the first overlap region and the second overlap region are spliced according to the optimal seam line, and the non-overlapping parts of the first video image and the second video image are connected to obtain a spliced video image, which is output to a display device as a current frame video image. The process of video image splicing based on the optimal seam line can refer to existing algorithms, which will not be described here.

[0148] It can be seen from the above that, in the embodiment of the application, the first video image and the second video image to be spliced are acquired, the second video image is subjected to homographic transformation, and the overlapping region of the first video image and the second video image subjected to the homographic transformation is determined. The overlapping region includes a first overlapping region and a second overlapping region. The first overlapping region and the second overlapping region are respectively subjected to region division, and the similarity of the corresponding overlapping sub-regions in the divided first overlapping region and second overlapping region is calculated. Then, based on the overlapping sub-regions and the similarity, the image color tones of the overlapping region are unified to obtain a target overlapping region. The best seam line of the target overlapping region is determined, and the first video image and the second video image are spliced according to the best seam line. The embodiment of the application unifies the image perspective by subjecting the second video image to homographic transformation, which helps to reduce the generation of misplacement or artifacts when the image is spliced in the same perspective. The unification of the color tones of the overlapping region is realized based on the similarity of the corresponding overlapping sub-regions in the two video images, which can guarantee the unification of the color tones of the spliced image and improve the effectiveness of the unification of the color tones, so that the video image can be accurately and effectively spliced, the user's visual perception of the video image is optimized, and the user experience is improved.

[0149] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution. The execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the application.

[0150] Corresponding to the video image splicing method of the above embodiment, FIG. 8 shows a structural block diagram of a video image splicing apparatus provided by the embodiment of the application. For ease of illustration, only the parts related to the embodiment of the application are shown.

[0151] Referring to FIG. 8, the video image splicing apparatus includes a video image acquisition unit 81, an overlapping region determination unit 82, a region similarity calculation unit 83, a color tone unification unit 84, a seam line determination unit 85, and a video image splicing unit 86, wherein:

[0152] The video image acquisition unit 81 is configured to acquire a first video image and a second video image to be spliced.

[0153] The overlapping region determination unit 82 is configured to subject the second video image to homographic transformation, and determine the overlapping region of the first video image and the second video image subjected to the homographic transformation. The overlapping region includes a first overlapping region in the first video image overlapping with the second video image, and a second overlapping region in the second video image overlapping with the first video image.

[0154] The region similarity calculation unit 83 is configured to subject the first overlapping region and the second overlapping region to region division respectively, and calculate the similarity of the corresponding overlapping sub-regions in the divided first overlapping region and second overlapping region.

[0155] a hue unifying unit 84, configured to unify image hues of the target overlapping region based on the overlapping sub-regions and the similarity;

[0156] a seam line determining unit 85, configured to determine an optimal seam line of the target overlapping region;

[0157] a video image splicing unit 86, configured to perform video image splicing on the first video image and the second video image according to the optimal seam line, to obtain a spliced video image.

[0158] In a possible implementation of the present application, the overlapping sub-regions include first overlapping sub-regions and second overlapping sub-regions; and the region similarity calculating unit 83 includes:

[0159] a first region dividing module, configured to divide the first overlapping region and the second overlapping region respectively to obtain a plurality of first overlapping sub-regions and a plurality of second overlapping sub-regions;

[0160] an edge information detecting module, configured to determine whether there is edge information in the first overlapping sub-regions and the corresponding second overlapping sub-regions;

[0161] a first similarity calculating module, configured to calculate a binary similarity between the first overlapping sub-regions and the corresponding second overlapping sub-regions if there is edge information;

[0162] a second similarity calculating module, configured to calculate a color similarity between the first overlapping sub-regions and the corresponding second overlapping sub-regions if there is no edge information.

[0163] In a possible implementation of the present application, the edge information detecting module is specifically configured to:

[0164] obtain a total sum of non-zero pixel values in the first overlapping sub-regions and the second overlapping sub-regions;

[0165] if the total sum of non-zero pixel values is less than a preset pixel threshold, it is determined that there is no edge information in the first overlapping sub-regions and the second overlapping sub-regions;

[0166] if the total sum of non-zero pixel values is greater than or equal to the preset pixel threshold, it is determined that there is edge information in the first overlapping sub-regions and the second overlapping sub-regions.

[0167] In a possible implementation of the present application, the overlapping sub-regions include first overlapping sub-regions and second overlapping sub-regions; and the hue unifying unit 84 includes:

[0168] a mask adding module, configured to add a mask to the first overlapping sub-regions and the second overlapping sub-regions in the first overlapping region and the second overlapping region, which have a similarity less than a preset similarity threshold.

[0169] a region color tone unification module, configured to perform color migration on the second overlapping region based on target parameter information, unify image color tones of all the first overlapping sub-regions and corresponding second overlapping sub-regions, and obtain a target overlapping region, wherein the target parameter information is determined according to the first overlapping sub-region without the mask and the second overlapping sub-region without the mask.

[0170] As a possible implementation of the present application, the seam line determination unit 85 comprises:

[0171] a first seam line determination module, configured to, if the first video image and the second video image are the first frame images in the video stream, determine the best seam line of the target overlapping region according to a preset seam line searching algorithm;

[0172] a second seam line determination module, configured to, if the first video image and the second video image are non-first frame images in the video stream, acquire dynamic information in the target overlapping region, and determine the best seam line of the target overlapping region according to the dynamic information.

[0173] As a possible implementation of the present application, the dynamic information comprises a foreground expansion item, and the second seam line determination module is specifically configured to:

[0174] acquire an energy value of the foreground expansion item in the target overlapping region, and count a number of foreground expansion items with energy values greater than 0 on the seam line of the previous frame spliced video image;

[0175] if the number of foreground expansion items does not satisfy a preset expansion energy value condition, use the seam line of the previous frame spliced video image as the best seam line of the current target overlapping region;

[0176] if the number of foreground expansion items satisfies the preset expansion energy value condition, determine the best seam line of the target overlapping region according to the preset seam line searching algorithm.

[0177] As a possible implementation of the present application, the seam line determination unit 85 comprises:

[0178] an energy matrix determination module, configured to acquire an energy matrix of the target overlapping region;

[0179] a target sub-region division module, configured to divide the target overlapping region to obtain a plurality of target overlapping sub-regions;

[0180] a comprehensive average value determination module, configured to determine a comprehensive energy average value of each target overlapping sub-region based on the energy matrix;

[0181] a coarse matrix construction module, configured to construct a coarse energy matrix according to the comprehensive energy average value of each target overlapping sub-region;

[0182] The seam line searching module is configured to determine the optimal seam line of the target overlap region based on the coarse energy matrix and a dynamic programming algorithm.

[0183] As can be seen from the above, in the embodiments of the present application, the first video image and the second video image to be spliced are acquired, the second video image is subjected to homographic transformation, and the overlap region of the first video image and the second video image subjected to homographic transformation is determined. The overlap region includes a first overlap region and a second overlap region. The first overlap region and the second overlap region are respectively subjected to region division, and the similarity of the corresponding overlap sub-regions in the divided first overlap region and second overlap region is calculated. Then, based on the overlap sub-regions and the similarity, the image color tone of the overlap region is unified to obtain a target overlap region. The optimal seam line of the target overlap region is determined, and the first video image and the second video image are spliced according to the optimal seam line. In the present application, the second video image is subjected to homographic transformation, and the image perspective is unified. Splicing the images in the same perspective helps to reduce the generation of misplacement or artifacts. The similarity of the corresponding overlap sub-regions in the two video images is used to realize the unification of the color tone of the overlap region. The effectiveness of the unification of the color tone is improved while the unification of the color tone of the spliced image is ensured. Thus, the splicing of the video images can be accurately and effectively realized, the user's visual perception of the video images is optimized, and the user experience is improved.

[0184] It should be noted that the information interaction, execution process and the like between the above-mentioned devices / units are based on the same concept as the method embodiments of the present application. For specific functions and brought technical effects, refer to the method embodiments part, which will not be repeated here.

[0185] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of any one of the video image splicing methods represented by FIG. 1 to FIG. 7 are implemented.

[0186] The embodiments of the present application also provide a terminal device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of any one of the video image splicing methods represented by FIG. 1 to FIG. 7 are implemented.

[0187] The embodiments of the present application also provide a computer program product, which, when executed on a terminal device, causes the terminal device to perform the steps of any one of the video image splicing methods represented by FIG. 1 to FIG. 7.

[0188] FIG. 9 is a schematic diagram of a terminal device according to an embodiment of the present application. As shown in FIG. 9, the terminal device 9 according to the embodiment includes a processor 90, a memory 91, and a computer program 92 stored in the memory 91 and executable on the processor 90. The processor 90 implements the steps in the above video image stitching method embodiments when executing the computer program 92, e.g., steps S101-S106 shown in FIG. 1. Alternatively, the processor 90 implements the functions of the modules / units in the above device embodiments when executing the computer program 92, e.g., the functions of the video image acquisition unit 81-video image stitching unit 86 shown in FIG. 8.

[0189] For example, the computer program 92 can be divided into one or more modules / units, which are stored in the memory 91 and executed by the processor 90 to complete the present application. The one or more modules / units can be a series of computer readable instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 92 in the terminal device 9.

[0190] The terminal device 9 can include, but is not limited to, the processor 90 and the memory 91. Those skilled in the art can understand that FIG. 9 is only an example of the terminal device 9, and does not limit the terminal device 9, which can include more or fewer components than those shown, or combine certain components, or different components, e.g., the terminal device 9 can also include an input / output device, a network access device, a bus, etc.

[0191] The processor 90 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0192] The memory 91 can be an internal storage unit of the terminal device 9, for example, a hard disk or a memory of the terminal device 9. The memory 91 can also be an external storage device of the terminal device 9, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 9. Further, the memory 91 can also include both the internal storage unit and the external storage device of the terminal device 9. The memory 91 is used to store computer programs and other programs and data required by the terminal device. The memory 91 can also be used to temporarily store data that has been output or will be output.

[0193] It should be noted that the information interaction, execution process, etc. between the above apparatuses / units, since based on the same concept as the method embodiments of the present application, the specific functions and the brought technical effects can be referred to the method embodiments part, and will not be repeated here.

[0194] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific name of each functional unit and module is only for convenient distinction, and does not limit the protection scope of the present application. The specific working process of the unit and module in the above system can refer to the corresponding process in the method embodiments, and will not be repeated here.

[0195] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The computer program can be stored in a computer readable storage medium, and when executed by a processor, the computer program can implement the steps of each method embodiment. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms. The computer readable medium at least includes any entity or device capable of carrying the computer program code to the device / terminal equipment, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, the computer readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0196] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0197] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A video image splicing method, comprising: obtaining a first video image and a second video image to be spliced; homographic transformation is performed on the second video image, and an overlapping area of the first video image and the homographic transformed second video image is determined, the overlapping area comprising a first overlapping area of the first video image overlapping with the second video image, and a second overlapping area of the second video image overlapping with the first video image; region division is performed on the first overlapping area and the second overlapping area respectively, and similarity of corresponding overlapping sub-areas in the divided first overlapping area and the second overlapping area is calculated; based on the overlapping sub-areas and the similarity, image tones of the overlapping area are unified to obtain a target overlapping area; a best seam line of the target overlapping area is determined; video image splicing is performed on the first video image and the second video image according to the best seam line to obtain a spliced video image. The overlapping sub-areas comprise first overlapping sub-areas and second overlapping sub-areas.

2. The method of claim 1, wherein, The region division and similarity calculation of the first overlapping area and the second overlapping area respectively comprises: region division is performed on the first overlapping area and the second overlapping area to obtain a plurality of first overlapping sub-areas and a plurality of second overlapping sub-areas; it is judged whether there is edge information in the first overlapping sub-area and the corresponding second overlapping sub-area; if there is edge information, binary similarity between the first overlapping sub-area and the corresponding second overlapping sub-area is calculated; if there is no edge information, color similarity between the first overlapping sub-area and the corresponding second overlapping sub-area is calculated. The judgment of whether there is edge information in the first overlapping sub-area and the corresponding second overlapping sub-area comprises:

3. The method of claim 2, wherein, the sum of non-zero pixel values in the first overlapping sub-area and the second overlapping sub-area is obtained; if the sum of non-zero pixel values is less than a preset pixel threshold, it is determined that there is no edge information in the first overlapping sub-area and the second overlapping sub-area; if the sum of non-zero pixel values is greater than or equal to the preset pixel threshold, it is determined that there is edge information in the first overlapping sub-area and the second overlapping sub-area. The overlapping sub-areas comprise first overlapping sub-areas and second overlapping sub-areas; the unification of image tones of the overlapping area to obtain a target overlapping area based on the overlapping sub-areas and the similarity comprises:

4. The method of claim 1, wherein, a mask is added to the first overlapping sub-area and the second overlapping sub-area with similarity less than a preset similarity threshold in the first overlapping area and the second overlapping area; color migration is performed on the second overlapping area based on target parameter information to unify image tones of all the first overlapping sub-areas and the corresponding second overlapping sub-areas, to obtain a target overlapping area, wherein the target parameter information is determined according to the first overlapping sub-area without the mask and the second overlapping sub-area without the mask. The determination of the best seam line of the target overlapping area comprises:

5. The method of claim 1, wherein, ​ if the first video image and the second video image are the first frame images in the video stream, determining the optimal seam line of the target overlap region according to a preset seam line searching algorithm; if the first video image and the second video image are non-first frame images in the video stream, obtaining dynamic information in the target overlap region, and determining the optimal seam line of the target overlap region according to the dynamic information.

6. The method of claim 5, wherein, The dynamic information includes a foreground expansion item, and the obtaining of the dynamic information in the target overlap region and the determination of the optimal seam line of the target overlap region according to the dynamic information include: obtaining an energy value of the foreground expansion item in the target overlap region, and counting a number of foreground expansion items with energy values greater than 0 on a seam line of a previous frame of the spliced video image; if the number of the foreground expansion items does not satisfy a preset expansion energy value condition, using the seam line of the previous frame of the spliced video image as the optimal seam line of the current target overlap region; if the number of the foreground expansion items satisfies the preset expansion energy value condition, determining the optimal seam line of the target overlap region according to the preset seam line searching algorithm.

7. The method of claim 5 or 6, wherein, The determination of the optimal seam line of the target overlap region according to the preset seam line searching algorithm includes: obtaining an energy matrix of the target overlap region; performing region division on the target overlap region to obtain a plurality of target overlap sub-regions; determining a comprehensive energy average value of each target overlap sub-region based on the energy matrix; constructing a rough energy matrix according to the comprehensive energy average values of the target overlap sub-regions; determining the optimal seam line of the target overlap region based on the rough energy matrix and a dynamic programming algorithm.

8. A video image splicing apparatus, comprising: a video image obtaining unit configured to obtain a first video image and a second video image to be spliced; an overlap region determining unit configured to perform homographic transformation on the second video image, and determine an overlap region of the first video image and the second video image after the homographic transformation, the overlap region including a first overlap region of the first video image overlapping with the second video image, and a second overlap region of the second video image overlapping with the first video image; a region similarity calculating unit configured to perform region division on the first overlap region and the second overlap region respectively, and calculate a similarity of corresponding overlap sub-regions in the first overlap region and the second overlap region after the division; a hue unifying unit configured to unify image hues of the overlap region based on the overlap sub-regions and the similarity to obtain a target overlap region; a seam line determining unit configured to determine an optimal seam line of the target overlap region; a video image splicing unit configured to perform video image splicing on the first video image and the second video image according to the optimal seam line to obtain a spliced video image.

9. The apparatus of claim 8, wherein, The overlap sub-regions include a first overlap sub-region and a second overlap sub-region. The region similarity calculating unit includes: The first region division module is configured to divide the first overlapping region and the second overlapping region respectively to obtain a plurality of first overlapping sub-regions and a plurality of second overlapping sub-regions. The edge information detection module is configured to determine whether there is edge information in the first overlapping sub-region and the corresponding second overlapping sub-region. The first similarity calculation module is configured to calculate the binary similarity between the first overlapping sub-region and the corresponding second overlapping sub-region if there is edge information. The second similarity calculation module is configured to calculate the color similarity between the first overlapping sub-region and the corresponding second overlapping sub-region if there is no edge information.

10. The apparatus of claim 9, wherein, The edge information detection module is specifically configured to: obtain the total sum of non-zero pixel values in the first overlapping sub-region and the second overlapping sub-region; if the total sum of non-zero pixel values is less than a preset pixel threshold, it is determined that there is no edge information in the first overlapping sub-region and the second overlapping sub-region; if the total sum of non-zero pixel values is greater than or equal to the preset pixel threshold, it is determined that there is edge information in the first overlapping sub-region and the second overlapping sub-region.

11. The apparatus of claim 8, wherein, The overlapping sub-regions include first overlapping sub-regions and second overlapping sub-regions; and the hue unification unit includes: The mask adding module is configured to add a mask to the first overlapping sub-region and the second overlapping sub-region in the first overlapping region and the second overlapping region whose similarity is less than a preset similarity threshold. The region hue unification module is configured to perform color migration on the second overlapping region based on target parameter information to unify the image hue of all the first overlapping sub-regions and the corresponding second overlapping sub-regions, and obtain a target overlapping region, wherein the target parameter information is determined according to the first overlapping sub-region without the mask and the second overlapping sub-region without the mask.

12. The apparatus of claim 8, wherein, The seam line determination unit includes: The first seam line determination module is configured to determine the best seam line of the target overlapping region according to a preset seam line searching algorithm if the first video image and the second video image are the first frame image in the video stream. The second seam line determination module is configured to obtain dynamic information in the target overlapping region and determine the best seam line of the target overlapping region according to the dynamic information if the first video image and the second video image are not the first frame image in the video stream.

13. The apparatus of claim 12, wherein, The dynamic information includes a foreground expansion term; and the second seam line determination module is specifically configured to: obtain the energy value of the foreground expansion term in the target overlapping region, and count the number of foreground expansion terms with energy value greater than 0 on the seam line of the previous frame of spliced video image; if the number of foreground expansion terms does not satisfy a preset expansion energy value condition, the seam line of the previous frame of spliced video image is used as the best seam line of the current target overlapping region; if the number of foreground expansion terms satisfies the preset expansion energy value condition, the best seam line of the target overlapping region is determined according to the preset seam line searching algorithm.

14. The apparatus of claim 12 or 13, wherein, The seam line determination unit includes: The energy matrix determination module is configured to obtain the energy matrix of the target overlapping region. a target sub-region division module configured to divide the target overlapping region into a plurality of target overlapping sub-regions; a comprehensive average value determination module configured to determine a comprehensive energy average value of each target overlapping sub-region based on the energy matrix; a coarse matrix construction module configured to construct a coarse energy matrix according to the comprehensive energy average value of each target overlapping sub-region; a seam line searching module configured to determine the optimal seam line of the target overlapping region based on the coarse energy matrix and a dynamic programming algorithm.

15. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the video image stitching method according to any one of claims 1 to 7 when executing the computer program.

16. A computer readable storage medium, storing a computer program, wherein the computer program is executed by a processor to implement the video image stitching method according to any one of claims 1 to 7.

17. A computer readable storage medium, storing a computer program, wherein the computer program is executed by a processor to implement the video image stitching method according to any one of claims 1 to 7.

Images