Method and device for constructing image splicing model of hundred million-level pixel calculation imaging system

By adjusting the matching area of ​​the local camera in the million-pixel computing imaging system and using the normal matching area to fit the correction position, the problem of low accuracy of the stitching model caused by the lack of feature information in the target scene is solved, and the imaging quality of the fusion video is improved.

CN119090713AActive Publication Date: 2024-12-06SUZHOU YIJI INTELLIGENT TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202311541589.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-11-17
Publication Date
2024-12-06
Estimated Expiration
2043-11-17

AI Technical Summary

Technical Problem

In a million-pixel computing imaging system, the area where the feature information is missing in the target scene causes the local image captured by the local camera to lack feature information, and then matches the area image with a large position error during the construction of the image stitching model, resulting in the stitching model Low adaptability and accuracy affect the imaging quality of the fusion video.

Method used

By acquiring the array camera to match multiple local sample images and global sample images of the target scene, determine the position abnormality of the initial matching area, use the normal matching area of ​​the same row or column to correct the position, adjust the target matching area, and then perform the image Register to build an image stitching model.

Benefits of technology

The matching and accuracy of the image stitching model is improved, and the fusion video imaging quality of the million-pixel computing imaging system is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090713A_ABST
    Figure CN119090713A_ABST
Patent Text Reader

Abstract

The invention relates to an image splicing model construction method and device of a hundred-million-level pixel calculation imaging system, a hundred-million-level pixel fusion video generation method and device, the hundred-million-level pixel calculation imaging system, computer equipment, a storage medium and a computer program product. The method comprises the following steps: performing region matching on each local sample image and a global sample image shot by an array camera to obtain an initial position of each initial matching region; taking the initial position of each candidate region pair as a reference, and determining a reference region pair with the minimum initial position loss value and an initial matching region with the abnormal initial position; according to the initial positions of other initial matching areas which are in the same row or in the same column and have normal initial positions, fitting to obtain a correction position of the initial matching area with the abnormal position; and carrying out image registration on the region image of the target matching region of the correction position and the local sample image, and constructing an image splicing model. By adopting the method, the adaptation degree and the accuracy of the image splicing model can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for constructing an image stitching model for a billion-pixel computational imaging system, a method and device for generating a billion-pixel fusion video, a billion-pixel computational imaging system, a computer device, a storage medium, and a computer program product. Background Art

[0002] Video surveillance has been widely used in the security field due to its advantages of high reliability, strong timeliness, and easy viewing. When the monitoring range is large, in order to achieve ultra-high-definition video monitoring of large scenes, a billion-pixel computational imaging system based on array cameras has emerged. The array camera uses multiple local cameras to shoot multiple local high-definition videos of the target scene, and then uses an image stitching model to stitch and fuse the local videos to obtain a large field of view, up to a billion-pixel level of ultra-high-definition fusion video, which is suitable for video shooting and security monitoring of large scenes such as airports, highways, parks, sports stadiums, borders, and sea surfaces.

[0003] In the related art, the array camera usually includes multiple local cameras with long focal length and narrow field of view and a global camera with short focal length and wide field of view. Before shooting the video of the target scene, an image stitching model can be constructed based on the local sample images taken by the local cameras and the global sample images taken by the global camera. The construction process mainly includes the steps of regional matching, image registration, and establishing transformation relationships. Regional matching refers to matching the regional images corresponding to each local sample image in the global sample image, and then performing image registration on each local sample image and the corresponding regional image, determining the transformation relationship and fusion parameters corresponding to each local camera, and obtaining the image stitching model.

[0004] However, when some areas in the target scene are pictures such as the sky and water surface that lack feature information, the local images taken by some local cameras will lack feature information. Therefore, in the process of establishing the image stitching model, when the local image is matched with the global image, it is easy to match the regional image with a large position error. As a result, the image stitching model established based on the erroneous regional image has low adaptability and accuracy, resulting in poor fusion video imaging quality of the billion-pixel computational imaging system. Summary of the invention

[0005] Based on this, it is necessary to provide an image stitching model construction method and device, a billion-pixel fused video generation method and device, a billion-pixel computational imaging system, computer equipment, storage medium and computer program product, which can improve the image stitching model accuracy and fused video imaging quality of the billion-pixel computational imaging system in response to the above-mentioned technical problems.

[0006] In a first aspect, the present application provides a method for constructing an image stitching model for a billion-pixel computational imaging system. The method comprises:

[0007] Acquire multiple local sample images and a global sample image taken by the array camera of the target scene, and perform region matching on each of the local sample images and the global sample image to obtain an initial position of an initial matching region corresponding to each of the local sample images in the global sample image;

[0008] Any two initial matching regions with different target row numbers and target column numbers are formed into candidate region pairs, and the initial position loss values ​​of the other initial matching regions are determined based on the initial position of each candidate region pair, and the candidate region pair corresponding to the minimum sum value is determined as the reference region pair, and the initial matching region with abnormal initial position and the initial matching region with normal initial position are determined based on the initial position loss values ​​of the other initial matching regions when the initial position of the reference region pair is used as the reference;

[0009] For an initial matching region with an abnormal initial position, a correction position corresponding to the initial matching region is obtained by fitting according to the initial positions of other initial matching regions in the same row or column as the initial matching region and with normal initial positions, and the region of the correction position in the global sample image is determined as the target matching region, and the initial matching region with normal initial position is determined as the target matching region;

[0010] The regional images of each target matching region and the local sample images are registered to construct an image stitching model of the target scene.

[0011] In one embodiment, the initial position includes coordinate information of pixel points included in the initial matching area; the initial matching area with abnormal initial position is fitted to obtain the correction position corresponding to the initial matching area according to the initial positions of other initial matching areas in the same row or column as the initial matching area and with normal initial positions, including:

[0012] For each initial matching region with an abnormal initial position, according to the target number of rows and the target number of columns of each initial matching region, in each initial matching region with a normal initial position, a first reference region in the same row as the initial matching region and a second reference region in the same column as the initial matching region are determined;

[0013] When the number of the first reference areas and the number of the second reference areas are both greater than a preset threshold, a first image fitting line and a second image fitting line are obtained by fitting according to the coordinate information of the pixel points at the first position and the second position in each of the first reference areas, and a third image fitting line and a fourth image fitting line are obtained by fitting according to the coordinate information of the pixel points at the first position and the second position in each of the second reference areas;

[0014] The coordinate information of the intersection of the first image fitting line and the third image fitting line, and the coordinate information of the second image fitting line and the fourth image fitting line are respectively used as the coordinate information of the pixel points of the first position and the second position in the initial matching area to obtain the corrected position corresponding to the initial matching area.

[0015] In one embodiment, the method further comprises:

[0016] When the number of the first reference area or the second reference area is not greater than a preset threshold, taking the initial position of the reference area pair as a reference, calculating the position to be corrected corresponding to the initial matching area, and determining the coordinate information to be corrected of the pixel points at the first position and the second position in the initial matching area;

[0017] According to the coordinate information of the pixel points at the first position and the second position in each of the first reference areas and the coordinate information to be corrected, the first image fitting line and the second image fitting line are fitted, and according to the coordinate information of the pixel points at the first position and the second position in each of the second reference areas and the coordinate information to be corrected, the third image fitting line and the fourth image fitting line are fitted.

[0018] In one embodiment, the step of taking the initial positions of the candidate region pairs as references, determining the sum of the initial position loss values ​​of the other initial matching regions, and determining the candidate region pair corresponding to the minimum sum as the reference region pair includes:

[0019] For each pair of candidate regions, any initial matching region in the candidate region pair is used as a candidate region benchmark, and a unit spacing benchmark is calculated according to a spacing between two initial matching regions in the candidate region pair and a number of row and column spacings between the two initial matching regions;

[0020] Calculate the comparison positions of the other initial matching regions according to the initial position of the candidate region benchmark, the unit spacing benchmark, and the number of row and column intervals between the other initial matching regions and the candidate region benchmark, and use the difference between the comparison position and the initial position as the initial position loss value;

[0021] The sum of the initial position loss values ​​of the other initial matching regions is calculated to determine the minimum sum, and the candidate region pair corresponding to the minimum sum is determined as the reference region pair.

[0022] In one embodiment, determining the initial matching region with abnormal initial position and the initial matching region with normal initial position based on the initial position loss values ​​of the other initial matching regions when the initial position of the reference region pair is taken as the reference comprises:

[0023] Determine the initial position loss values ​​of each of the other initial matching areas when the initial position of the reference area pair is used as a reference, and determine the initial matching area whose initial position loss value is greater than a preset threshold as an initial matching area with an abnormal initial position, and determine the initial matching area whose initial position loss value is not greater than the preset threshold and the two initial matching areas in the reference area pair as initial matching areas with normal initial positions.

[0024] In a second aspect, the present application also provides a method for generating a billion-pixel fusion video. The method comprises:

[0025] Acquire multiple local video streams of the target scene captured by the array camera;

[0026] Using an image stitching model to stitch and fuse each group of local video images in the multiple local video streams to obtain multiple fused video images, and generating a fused video stream based on the fused video images;

[0027] The image stitching model is constructed according to the image stitching model construction method described in the first aspect.

[0028] In a third aspect, the present application also provides an image stitching model construction device. The device comprises:

[0029] A matching module, used to obtain multiple local sample images and a global sample image taken by the array camera of the target scene, and perform region matching on each of the local sample images and the global sample image to obtain an initial position of an initial matching region corresponding to each of the local sample images in the global sample image;

[0030] an abnormality determination module, used to form a candidate region pair from any two initial matching regions with different target row numbers and target column numbers, determine the sum of initial position loss values ​​of other initial matching regions based on the initial position of each candidate region pair, determine the candidate region pair corresponding to the minimum sum as the reference region pair, and determine the initial matching region with an abnormal initial position and the initial matching region with a normal initial position based on the initial position loss values ​​of other initial matching regions based on the initial position of the reference region pair;

[0031] A correction module is used for fitting, for an initial matching region with an abnormal initial position, a correction position corresponding to the initial matching region according to the initial positions of other initial matching regions in the same row or column as the initial matching region and with normal initial positions, determining the region of the correction position in the global sample image as a target matching region, and determining the initial matching region with a normal initial position as a target matching region;

[0032] The registration module is used to perform image registration on the regional images of each target matching area and the local sample image to construct an image stitching model of the target scene.

[0033] In a fourth aspect, the present application also provides a device for generating billion-level pixel fusion videos. The device comprises:

[0034] An acquisition module is used to acquire multiple local video streams captured by the array camera of the target scene;

[0035] A fusion module is used to use an image stitching model to perform image stitching and fusion on each group of local video images in the multiple local video streams to obtain multiple fused video images, and generate a fused video stream based on the fused video images; wherein the image stitching model is constructed according to the image stitching model construction method described in the first aspect.

[0036] In a fifth aspect, the present application also provides a billion-pixel computational imaging system. The billion-pixel computational imaging system includes an array camera and a server, wherein the array camera includes a plurality of local cameras and a global camera, wherein:

[0037] The array camera is used to capture a local sample image of the target scene through each of the local cameras, capture a global sample image through the global camera, and send the local sample image and the global sample image to the server;

[0038] The server is used to perform region matching on each of the local sample images and the global sample image to obtain the initial position of the initial matching region corresponding to each of the local sample images in the global sample image; any two initial matching regions with different target row numbers and target column numbers are formed into a candidate region pair, and the sum of the initial position loss values ​​of the other initial matching regions is determined based on the initial position of each of the candidate region pairs, and the candidate region pair corresponding to the minimum sum is determined as the reference region pair, and the initial matching region with an abnormal initial position and the initial matching region with a normal initial position are determined based on the initial position loss values ​​of the other initial matching regions based on the initial position of the reference region pair; for the initial matching region with an abnormal initial position, the correction position corresponding to the initial matching region is obtained by fitting based on the initial positions of other initial matching regions in the same row or column as the initial matching region and with normal initial positions, and the region of the correction position in the global sample image is determined as the target matching region, and the initial matching region with a normal initial position is determined as the target matching region; the region images of the target matching regions are registered with the local sample image to construct an image stitching model of the target scene.

[0039] In a sixth aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in the first aspect or the second aspect are implemented.

[0040] In a seventh aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in the first aspect or the second aspect are implemented.

[0041] In an eighth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect or the second aspect are implemented.

[0042] The image stitching model construction method and device of the above-mentioned billion-pixel computational imaging system, the billion-pixel fusion video generation method and device, the billion-pixel computational imaging system, computer equipment, storage medium and computer program product determine whether the initial position of each initial matching area is abnormal through the initial position of each initial matching area obtained by area matching. For the initial matching area with an abnormal initial position, the corrected position is obtained by fitting the initial position of the initial matching area with a normal initial position, and then the target matching area corresponding to the corrected position and the initial matching area with a normal initial position are aligned with each local sample image to determine the transformation relationship and fusion parameters to obtain the image stitching model.

[0043] Among them, to judge whether the initial position of each initial matching area is abnormal, the initial position loss value of each initial matching area is calculated based on the initial position of each pair of candidate areas, and then the candidate area pair with the smallest sum of initial position loss values ​​is used as the reference area pair. The matching accuracy of the initial position of the reference area pair is the highest, so that according to the initial position loss values ​​of other initial matching areas calculated under this reference, it is possible to accurately judge whether the initial position of each initial matching area is abnormal. Then, based on the initial position of the normal initial matching area, the corrected position of the abnormal initial matching area is fitted to achieve position correction of the abnormal matching area. The matching accuracy of the target matching area at the corrected position is higher, so that the matching degree and accuracy of the image stitching model and the target scene can be improved, which is conducive to improving the fusion video imaging quality of the billion-pixel computational imaging system. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of a billion-pixel computational imaging system in an example;

[0045] Figure 2 A schematic diagram of a process of constructing an image stitching model in one embodiment;

[0046] Figure 3a is a schematic diagram of an array camera in an example;

[0047] Figure 3b A schematic diagram of the target matching area in a global sample image in an example;

[0048] Figure 3c A schematic diagram of the initial matching area in a global sample image in an example;

[0049] Figure 4 A structural block diagram of an image stitching model building device in one embodiment;

[0050] Figure 5 It is a structural block diagram of a device for generating billion-level pixel fusion video in an embodiment;

[0051] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] First, before specifically introducing the technical solution of the embodiment of the present application, the technical background or technical evolution context on which the embodiment of the present application is based is introduced. In order to realize ultra-high-definition video surveillance of large scenes, a billion-pixel computational imaging system based on array cameras has emerged. The array camera usually includes multiple local cameras with long focal lengths and narrow fields of view and a global camera with short focal lengths and wide fields of view. The field of view of the global camera can cover the field of view of each local camera. The local camera captures a local high-definition image rich in detail information of the target scene, and the global camera captures a global image with a lower resolution. Before shooting a video of the target scene, an image stitching model can be constructed based on the local sample images captured by the local camera and the global sample images captured by the global camera. When shooting a video of the target scene, the image stitching model can be used to stitch and fuse the multiple local high-definition videos captured by each local camera, and output ultra-high-definition fused videos with a large field of view or a wide field of view in real time, which can reach the billion-pixel level. It is suitable for video shooting and security monitoring of large scenes such as airports, highways, parks, sports stadiums, borders, and sea surfaces.

[0054] The process of establishing the image stitching model of the billion-pixel computational imaging system mainly includes the steps of region matching, image registration, determining transformation relationships and fusion parameters. Region matching refers to matching the region images corresponding to each local sample image in the global sample image, and then performing image registration on each local sample image and the corresponding region image to determine the transformation relationship between the two images. The transformation relationship is used to geometrically transform the local sample image to obtain a transformation image consistent with the shooting angle of the region image (i.e., the shooting angle of the global sample image), and complete the coordinate unified transformation, thereby determining the fusion parameters based on the transformation images with unified coordinates. The fusion parameters are used to fuse the transformation images to obtain a fused image with a large field of view and ultra-high resolution. The transformation relationship and fusion parameters corresponding to each local camera can be constructed as an image stitching model to stitch and fuse the local videos shot by the array camera to output an ultra-high-definition fused video.

[0055] However, in certain scenarios, such as when some areas in the target scene are the sky, water surface, etc. that lack feature information or contour information, the local images taken by some local cameras lack feature information. Therefore, in the process of establishing the image stitching model, when the local image is matched with the global image, it is easy to match the regional image with a large position error, and then the erroneous regional image is matched with the local image to determine the transformation relationship and fusion parameters. The resulting image stitching model has low adaptability and accuracy, resulting in poor imaging quality of the fused video image obtained by stitching and fusion, which affects the imaging quality of the billion-pixel computational imaging system.

[0056] Based on this background, the applicant, through long-term research and development and experimental verification, proposed the image stitching model construction method of the billion-pixel computational imaging system of this application, which can improve the matching degree and accuracy of the image stitching model with the target scene, and is conducive to improving the fusion video imaging quality of the billion-pixel computational imaging system. In addition, it should be noted that the applicant has made a lot of creative efforts in the discovery of the technical problem of this application and the technical solutions introduced in the following embodiments.

[0057] The image stitching model construction method provided in the embodiment of the present application can be applied to Figure 1 The billion-pixel computational imaging system shown in FIG. Array camera 102 and display terminal 106 communicate with server 104 through a network respectively. Display terminal 106 may be, but is not limited to, various electronic devices with display units such as personal computers, laptops, smart phones, tablet computers, smart displays, IoT devices, portable wearable devices, etc. Server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0058] In one embodiment, Figure 2 As shown, a method for constructing an image stitching model is provided, which can be applied to Figure 1 In this embodiment, the method includes the following steps:

[0059] Step 201, obtain multiple local sample images and global sample images taken by the array camera of the target scene, and perform region matching on each local sample image and the global sample image to obtain the initial position of the initial matching region corresponding to each local sample image in the global sample image.

[0060] In implementation, the array camera can be installed in the environment of the target scene to shoot a video of the target scene. Before the formal video shooting of the target scene, the array camera can first collect sample images of the target scene to build an image stitching model, so that the image stitching model can be used to perform real-time image stitching and fusion in the subsequent video shooting process, thereby improving the generation efficiency of the fused video.

[0061] The array camera may include a plurality of local cameras distributed in multiple rows and columns and a global camera. Figure 3a The array camera shown includes 18 local cameras (or local lenses) distributed in 3 rows and 6 columns and 1 global camera (or global lens) in the center. Each local camera can be identified according to the row and column information of the local camera. For example, looking from the rear end to the front end of the camera, the local camera in the first row and the first column can be identified as A11. For the convenience of unified expression, the local image taken by the local camera Aij (i represents the number of rows, j represents the number of columns) and the regional image corresponding to the local image in the global image can be identified as Aij.

[0062] The shooting angles of each local camera are different. Usually, the lens optical axes of two local cameras in the same row have a certain angle in the horizontal direction, and the lens optical axes of two local cameras in the same column have a certain angle in the vertical direction (understandably, due to Figure 3a A global camera is set in the middle row of the array camera shown. In order to make the overall structure compact, the local cameras in this row are not on the same vertical line as the local cameras in the same column of other rows, but this does not affect the row and column relationship of each local camera. Usually, the field of view of the global camera is larger than the field of view of the local camera, and the focal length is smaller than the focal length of the local camera. The field of view of the global camera can cover the field of view of each local camera, and the field of view of adjacent local cameras can have a certain overlapping area.

[0063] The server can use template matching algorithm or other methods to perform regional matching between each local sample image (which can be reduced by a certain ratio) and the global sample image, and can use the matched area with the greatest similarity as the initial matching area. The purpose of regional matching is to match the target matching area image of the same local scene (part of the target scene) captured by the global camera and the local camera. The target position of each local sample image in the target matching area in the global sample image A0 (if the match is correct) can be as follows: Figure 3b As shown, it can be understood that, since the local camera and the global camera have different parameters such as shooting angle, focal length, resolution, etc., the images taken of the same local scene are only similar images, not completely the same.

[0064] In actual application, if some local sample images and the corresponding regional images in the global sample image lack feature information, the initial position of the matched initial matching area may have a large error. The initial matching area of ​​each local sample image in the global sample image can be as follows: Figure 3c As shown in the example, the initial position errors of the initial matching areas A13 and A15 are large and seriously deviate from the correct position (refer to Figure 3b The initial position of the initial matching area can be represented by the coordinates of the upper left and lower right pixels of the area (x0, y0, x1, y1). It should be noted that in order to facilitate the stitching and fusion to obtain a seamless fusion image, the shooting fields of adjacent local cameras usually have overlapping areas, and the corresponding matching areas also have overlapping areas. In addition, due to factors such as the different distances between the photographed objects in each local scene of the target scene and the array camera, the sizes of the matching areas corresponding to each local sample image are not necessarily the same. In order to clearly show the row and column relationship of each matching area, Figure 3b and Figure 3c The overlapping area is not shown, nor is the size of each matching area distinguished. This is just a simple illustration.

[0065] In step 202, any two initial matching areas with different target row numbers and target column numbers are grouped into candidate area pairs, and the sum of the initial position loss values ​​of other initial matching areas is determined based on the initial position of each candidate area pair, and the candidate area pair corresponding to the minimum sum is determined as the benchmark area pair, and the initial matching areas with abnormal initial positions and the initial matching areas with normal initial positions are determined based on the initial position loss values ​​of other initial matching areas based on the initial position of the benchmark area pair.

[0066] Among them, the target number of rows and the target number of columns refer to the number of rows and the number of columns where each matching area is located when the area matching is correct. Usually, when the installation angle (shooting angle) of each local camera of the array camera is determined, the number of rows and the number of columns where the matching area corresponding to the local image captured by each local camera can be determined, and the target number of rows and the target number of columns corresponding to each local camera can be pre-stored. It should be noted that since the initial position of the initial matching area may have a large error, the actual number of rows or columns where each initial matching area is located may be different from the target number of rows and the target number of columns.

[0067] The target number of rows and target number of columns of the two initial matching regions in the candidate region pair are different. Figure 3c As shown, the initial matching area A11 (in order to clearly show the matching area with the wrong position, the identification of the matching area with the correct position is omitted in this figure. The identification of each matching area can be seen in Figure 3b ) can form candidate area pairs with matching areas A22 to A26 and A32 to A36. For the array camera with 3 rows and 6 columns, 90 candidate area pairs can be formed.

[0068] The initial position loss value is used to describe the difference between the initial position of the initial matching area and the comparison position determined based on the initial position of the candidate area pair. The initial position of the candidate area pair includes the initial positions of the two initial matching areas, which can reflect the coordinate information of the two initial matching areas in the panoramic image, and the distance (spacing) between the two initial matching areas.

[0069] In one implementation, the process of calculating the sum of the initial position loss values ​​and determining the reference region pairs based on each candidate region pair specifically includes the following steps:

[0070] Step 2021: for each candidate region pair, any initial matching region in the candidate region pair is used as a candidate region benchmark, and a unit spacing benchmark is calculated according to the spacing between the two initial matching regions in the candidate region pair and the number of row and column spacings between the two initial matching regions.

[0071] Among them, the spacing between the two initial matching areas includes row spacing and column spacing, which can be specifically calculated based on the coordinate information of the pixel point at the center position of the initial matching area. The unit spacing benchmark may include a unit row spacing benchmark and a unit column spacing benchmark. The unit row spacing benchmark may be obtained by dividing the row spacing by the number of row spacings, and the unit column spacing benchmark may be obtained by dividing the column spacing by the number of column spacings. The number of row and column spacings may be calculated based on the target number of rows and target number of columns of each initial matching area (or each local camera). For example, the coordinates of the center point of the initial matching area A11 are (x c11 ,y c11 ), the coordinates of the center point of the initial matching area A23 are (x c23 ,y c23 ), the line spacing between the two is Lr = |x c11 -x c23 |, row spacing Nr = 1, column spacing Lc = |y c11 -y c23 |, the number of column intervals Nc = 2. The center point coordinates can be calculated based on the pixel coordinates (x0, y0) and (x1, y1) of the upper left corner and lower right corner of the region, or the pixel coordinates (x2, y2) and (x3, y3) of the lower left corner and upper right corner.

[0072] The candidate region benchmark may be any initial matching region in the candidate region pair, and may be randomly selected, or the initial matching region with the smaller number of rows in the region pair may be used as the candidate region benchmark when the candidate region pairs corresponding to the initial matching regions are traversed in sequence.

[0073] Step 2022, based on the initial position of the candidate area benchmark, the unit spacing benchmark, and the number of row and column intervals between other initial matching areas and the candidate area benchmark, calculate the comparison position of other initial matching areas, and use the difference between the comparison position and the initial position as the initial position loss value.

[0074] Among them, the initial position loss value can be calculated based on the coordinates of one or more pixel points at the center point, four corner points (upper left, lower left, upper right, lower right) or other positions. In one example, since the initial position of the initial matching area can be represented by the coordinates of the upper left corner and lower right corner pixel points of the area (x0, y0, x1, y1), the initial position loss value can be calculated based on the coordinates of the upper left corner and lower right corner. The difference can be obtained based on the coordinate difference of the pixel points at the same position (such as the upper left corner and the lower right corner).

[0075] In one example, the initial position of the candidate region reference in the candidate region pair can be expressed as (x 0_b ,y 0_b ,x 1_b ,y 1_b), the number of rows of the candidate region reference is br, the number of columns is bc, the unit row spacing reference is Sr, the unit column spacing reference is Sc, and the other initial matching regions A except the candidate region pair ij The initial position of (row number is i, column number is j) can be expressed as (x 0_Aij ,y 0_Aij ,x 1_Aij ,y 1_Aij ), the initial matching area A calculated based on the candidate area pair ij The reference position can be expressed as The calculation formula is as follows:

[0076]

[0077]

[0078]

[0079]

[0080] Initial matching area A ij The calculation formula of the initial position loss value Lij is as follows:

[0081]

[0082] The server can calculate other initial matching areas A ij The sum of the loss values ​​Lij is taken as the sum of the initial position loss values ​​corresponding to the candidate region pair.

[0083] Step 2023, calculate the sum of the initial position loss values ​​of other initial matching regions, determine the minimum sum, and determine the candidate region pair corresponding to the minimum sum as the reference region pair.

[0084] In implementation, the server may compare the sums of the initial position loss values ​​respectively calculated when each candidate area pair is a reference, determine the minimum sum, and determine the area pair corresponding to the minimum sum as the reference area pair.

[0085] Afterwards, the server can compare the initial position loss values ​​of other initial matching areas (except the two initial matching areas in the reference area pair) calculated based on the reference area pair with the preset thresholds. If the initial position loss values ​​are greater than the preset thresholds, it means that the comparison position calculated based on the position and spacing of the reference area pair with higher matching accuracy is significantly different from the initial position, so the initial position can be judged to be abnormal. If the initial position loss values ​​are not greater than the preset thresholds, the initial position is judged to be normal. In addition, the two initial matching areas in the reference area pair are also judged to be normal in the initial position.

[0086] Step 203, for the initial matching area with an abnormal initial position, the correction position corresponding to the initial matching area is fitted according to the initial positions of other initial matching areas in the same row or column as the initial matching area and with normal initial positions, the area of ​​the correction position in the global sample image is determined as the target matching area, and the initial matching area with normal initial position is determined as the target matching area.

[0087] In implementation, for the initial matching area with an abnormal initial position, the server can perform least squares fitting based on the coordinate information of the target position pixel points of the initial matching area with a normal initial position in the same row (same target row number) as the initial matching area to obtain a first fitting relationship, and perform least squares fitting based on the coordinate information of the target position pixel points of the initial matching area with a normal initial position in the same column (same target column number) as the initial matching area to obtain a second fitting relationship, and then determine the coordinate information of the corresponding position pixel points of the initial matching area based on the first fitting relationship and the second fitting relationship as the correction position.

[0088] In one implementation, the process of determining the corrected position of the initial matching area with an abnormal initial position includes the following steps:

[0089] Step 2031, for each initial matching area with an abnormal initial position, according to the target number of rows and the target number of columns of each initial matching area, in each initial matching area with a normal initial position, determine a first reference area in the same row and a second reference area in the same column as the initial matching area.

[0090] For example, Figure 3c As shown, for the initial matching area A13 with an abnormal initial position, its first reference area includes A11, A12, A14, and A16, and its second reference area includes A23 and A33.

[0091] In step 2032, when the number of first reference areas and second reference areas is greater than a preset threshold, the first image fitting line and the second image fitting line are fitted according to the coordinate information of the pixel points at the first position and the second position in each first reference area, and the third image fitting line and the fourth image fitting line are fitted according to the coordinate information of the pixel points at the first position and the second position in each second reference area.

[0092] Among them, the preset threshold can be a directly set number, or it can be a preset ratio of the set initial position normal area to the initial position abnormal area or the total number of areas, and then the preset threshold is calculated based on the total number of matching areas in each row and column and the preset ratio.

[0093] If the number of the first reference area and the second reference area is greater than the preset threshold, the abnormal area can be corrected based on the reference area with normal initial position. Specifically, the coordinate information of the pixel points at the first position and the second position in each first reference area is respectively subjected to least square fitting to obtain the first image fitting line and the second image fitting line, and the coordinate information of the pixel points at the first position and the second position in each second reference area is respectively subjected to least square fitting to obtain the third image fitting line and the fourth image fitting line. The coordinate information of the pixel points at the first position and the second position can uniquely determine the position information of a rectangular area, for example, it can be the upper left corner and the right small corner, the upper right corner and the lower left corner, or any corner point and the center point of the rectangular area.

[0094] Step 2033, using the coordinate information of the intersection of the first image fitting line and the third image fitting line, and the coordinate information of the second image fitting line and the fourth image fitting line as the coordinate information of the pixel points at the first position and the second position in the initial matching area, respectively, to obtain the correction position corresponding to the initial matching area.

[0095] In implementation, after determining the first image fitting line (horizontally) and the third image fitting line (vertically) corresponding to the first position (such as the upper left corner), the coordinate information of the first intersection of the two fitting lines can be obtained, and after determining the second image fitting line (horizontally) and the fourth image fitting line (vertically) corresponding to the second position (such as the lower right corner), the coordinate information of the second intersection of the two fitting lines can be obtained. The server can use the rectangular area determined by the two intersections as the target matching area after position correction.

[0096] In another example, if it is determined that the number of the first reference area or the second reference area is not greater than the preset threshold, that is, the number of initial matching areas with the correct initial position of the row or the column is small, a large deviation will occur for position fitting, so the server can use the initial position of the reference area pair as a reference, calculate the position to be corrected corresponding to the initial matching area, and make a preliminary adjustment to the position of the initial matching area. Specifically, the server can calculate the reference coordinates (coordinate information to be corrected) of the pixel points at the first position and the second position of the initial matching area according to the coordinate information of the pixel points at the first position and the second position of the candidate area benchmark of the reference area pair, the unit spacing reference (row spacing reference and column spacing reference) of the reference area pair, and the number of row and column spacings between the initial matching area and the candidate area benchmark, and determine the position to be corrected of the initial matching area using the reference coordinates of the two pixel points. Then, the server can fit the first image fitting line and the second image fitting line according to the coordinate information of the pixel points at the first position and the second position of each first reference area, and the coordinate information to be corrected, and fit the third image fitting line and the fourth image fitting line according to the coordinate information of the pixel points at the first position and the second position of each second reference area, and the coordinate information to be corrected. Then, the correction position of the initial matching area is determined based on the intersection of each fitting line. That is, if the number of normal matching areas in the same row or column is small, the server can use the initial position of the abnormal matching area, which has been preliminarily adjusted, for the correction position, for fitting, which can improve the correction accuracy when the number of normal matching areas is small.

[0097] Then, the server may determine the area at the corrected position in the global sample image as the target matching area, and determine the initial matching area with a normal initial position as the target matching area, thereby obtaining target matching areas with higher matching accuracy corresponding to each local sample image.

[0098] Step 204 , performing image registration on the regional images of each target matching region and the local sample images to construct an image stitching model of the target scene.

[0099] During implementation, the server can perform image registration on each local sample image and its corresponding regional image of the target matching area (which can be enlarged to the same size as the local sample image), determine the transformation relationship or image mapping relationship and corresponding fusion parameters between the two, and obtain the image stitching model corresponding to the target scene.

[0100] In the above-mentioned image stitching model construction method, the initial positions of the initial matching regions obtained by region matching are used to determine whether the initial positions of the initial matching regions are abnormal. For the initial matching regions with abnormal initial positions, the correction positions are obtained according to the initial position fitting of the initial matching regions with normal initial positions, and then the target matching regions corresponding to the correction positions and the initial matching regions with normal initial positions are image-aligned with each local sample image to determine the transformation relationship and fusion parameters to obtain the image stitching model. In particular, to determine whether the initial positions of the initial matching regions are abnormal, the initial position loss values ​​of the initial matching regions are calculated based on the initial positions of each pair of candidate regions, and then the candidate region pair with the smallest sum of the initial position loss values ​​is used as the reference region pair. The matching accuracy of the initial position of the reference region pair is the highest, so that the initial positions of the initial matching regions can be accurately determined based on the initial position loss values ​​of other initial matching regions calculated under the reference. Then, based on the initial position of the normal initial matching area, the corrected position of the abnormal initial matching area is fitted to realize the position correction of the abnormal matching area. The matching accuracy of the target matching area at the corrected position is higher, thereby improving the matching degree and accuracy of the image stitching model and the target scene, which is beneficial to improving the fused video imaging quality of the billion-pixel computational imaging system.

[0101] The present application also provides a method for generating a billion-pixel fusion video. The method can be applied to Figure 1 In this embodiment, the method includes the following steps:

[0102] Step 1: Obtain multiple local video streams captured by array cameras of a target scene.

[0103] Step 2: Use an image stitching model to perform image stitching and fusion on each group of local video images in the multiple local video streams to obtain multiple fused video images, and generate a fused video stream based on the fused video images.

[0104] The image stitching model is constructed according to the image stitching model construction method provided in the above embodiments.

[0105] In implementation, before the server shoots a video of the target scene, it can first use the image stitching model construction method in the above embodiment to construct an image stitching model corresponding to the target scene. Then, when officially shooting the video, the array camera can shoot local video streams respectively through multiple local cameras, and then send the local video streams to the server. The server can use the image stitching model to stitch and fuse a group of local video images shot by each local camera at the same time or similar time (there may be an acceptable time difference due to the inconsistent exposure time of each local camera), including performing image transformation on each local video image based on the transformation relationship corresponding to each local camera in the image stitching model, obtaining a transformation image corresponding to each local video image with unified coordinates, and then based on the fusion parameters in the image stitching model, fuse each transformation image to obtain a fused video image of the current frame. The server can generate a fused video stream from each frame of fused video images and send it to the display terminal for display or playback.

[0106] In the above-mentioned billion-pixel fused video generation method, the image stitching model construction process takes into account the possibility that some areas in the target scene may lack feature information and cause inaccurate regional matching. Therefore, the initial matching area with an abnormal initial position is corrected to obtain a target matching area with a more accurate position, and then an image stitching model with a high degree of match with the target scene and a higher accuracy is constructed. When it is used to stitch and fuse local video images, the obtained fused video image has higher accuracy and higher imaging quality.

[0107] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0108] Based on the same inventive concept, the embodiment of the present application also provides an image stitching model construction device for implementing the image stitching model construction method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more image stitching device embodiments provided below can refer to the limitations of the image stitching model construction method above, and will not be repeated here.

[0109] In one embodiment, Figure 4 As shown, an image stitching model construction device 400 is provided, comprising: a matching module 401, an abnormality determination module 402, a correction module 403 and a registration module 404, wherein:

[0110] The matching module 401 is used to obtain multiple local sample images and global sample images taken by the array camera of the target scene, and perform region matching on each of the local sample images and the global sample image to obtain the initial position of the initial matching region corresponding to each of the local sample images in the global sample image.

[0111] The abnormality determination module 402 is used to form a candidate area pair from any two initial matching areas with different target row numbers and target column numbers, and to determine the sum of the initial position loss values ​​of the other initial matching areas based on the initial position of each candidate area pair, to determine the candidate area pair corresponding to the minimum sum as the reference area pair, and to determine the initial matching areas with abnormal initial positions and the initial matching areas with normal initial positions based on the initial position loss values ​​of the other initial matching areas based on the initial position of the reference area pair.

[0112] The correction module 403 is used to fit the correction position corresponding to the initial matching area with an abnormal initial position according to the initial positions of other initial matching areas in the same row or column as the initial matching area and with normal initial positions, determine the area of ​​the correction position in the global sample image as the target matching area, and determine the initial matching area with normal initial positions as the target matching area.

[0113] The registration module 404 is used to perform image registration on the regional images of each target matching region and the local sample image to construct an image stitching model of the target scene.

[0114] In one embodiment, the initial position includes the coordinate information of the pixel points included in the initial matching area. The correction module 403 is specifically used to: for each initial matching area with an abnormal initial position, according to the target number of rows and the target number of columns of each initial matching area, in each initial matching area with a normal initial position, determine the first reference area in the same row as the initial matching area and the second reference area in the same column as the initial matching area; when the number of the first reference area and the second reference area is greater than the preset threshold, according to the coordinate information of the pixel points at the first position and the second position in each first reference area, fit the first image fitting line and the second image fitting line, according to the coordinate information of the pixel points at the first position and the second position in each second reference area, fit the third image fitting line and the fourth image fitting line; use the coordinate information of the intersection of the first image fitting line and the third image fitting line, and the coordinate information of the second image fitting line and the fourth image fitting line as the coordinate information of the pixel points at the first position and the second position in the initial matching area, respectively, to obtain the correction position corresponding to the initial matching area.

[0115] In one embodiment, the correction module 403 is also used for: when the number of the first reference area or the second reference area is not greater than a preset threshold, taking the initial position of the reference area pair as a reference, calculating the position to be corrected corresponding to the initial matching area, and determining the coordinate information to be corrected of the pixel points of the first position and the second position in the initial matching area; fitting a first image fitting line and a second image fitting line according to the coordinate information of the pixel points of the first position and the second position in each of the first reference areas, and the coordinate information to be corrected, and fitting a third image fitting line and a fourth image fitting line according to the coordinate information of the pixel points of the first position and the second position in each of the second reference areas, and the coordinate information to be corrected.

[0116] In one embodiment, the abnormality determination module 402 is specifically used to: for each pair of the candidate area pairs, use any initial matching area in the candidate area pair as the candidate area benchmark, and calculate the unit spacing benchmark based on the spacing between the two initial matching areas in the candidate area pair and the number of row and column intervals between the two initial matching areas; calculate the comparison position of each other initial matching area based on the initial position of the candidate area benchmark, the unit spacing benchmark, and the number of row and column intervals between each other initial matching area and the candidate area benchmark, and use the difference between the comparison position and the initial position as the initial position loss value; calculate the sum of the initial position loss values ​​of each other initial matching area, determine the minimum sum, and determine the candidate area pair corresponding to the minimum sum as the benchmark area pair.

[0117] In one embodiment, the abnormality determination module 402 is specifically used to: determine the initial position loss values ​​of each of the other initial matching areas when the initial position of the reference area pair is used as a reference, and determine the initial matching area whose initial position loss value is greater than a preset threshold as an initial matching area with an abnormal initial position, and determine the initial matching area whose initial position loss value is not greater than the preset threshold and the two initial matching areas in the reference area pair as initial matching areas with normal initial positions.

[0118] In one embodiment, Figure 5 As shown, a device 500 for generating a billion-pixel fusion video is also provided, comprising an acquisition module 501 and a fusion module 502, wherein:

[0119] The acquisition module 501 is used to acquire multiple local video streams captured by the array camera on the target scene.

[0120] The fusion module 502 is used to use an image stitching model to perform image stitching and fusion on each group of local video images in multiple local video streams to obtain multiple fused video images, and generate a fused video stream based on the fused video images; wherein the image stitching model is constructed according to the image stitching model construction method provided in the above embodiment.

[0121] Each module in the above-mentioned image stitching model construction device and billion-pixel fusion video generation device can be implemented in whole or in part by software, hardware, and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0122] In one embodiment, a billion-pixel computational imaging system is also provided. The billion-pixel computational imaging system includes an array camera and a server, wherein the array camera includes a plurality of local cameras and a global camera, wherein:

[0123] The array camera is used to capture a local sample image of the target scene through each of the local cameras, capture a global sample image through the global camera, and send the local sample image and the global sample image to the server;

[0124] The server is used to perform region matching on each of the local sample images and the global sample image to obtain the initial position of the initial matching region corresponding to each of the local sample images in the global sample image; any two initial matching regions with different target row numbers and target column numbers are formed into a candidate region pair, and the sum of the initial position loss values ​​of the other initial matching regions is determined based on the initial position of each of the candidate region pairs, and the candidate region pair corresponding to the minimum sum is determined as the reference region pair, and the initial matching region with an abnormal initial position and the initial matching region with a normal initial position are determined based on the initial position loss values ​​of the other initial matching regions based on the initial position of the reference region pair; for the initial matching region with an abnormal initial position, the correction position corresponding to the initial matching region is obtained by fitting based on the initial positions of other initial matching regions in the same row or column as the initial matching region and with normal initial positions, and the region of the correction position in the global sample image is determined as the target matching region, and the initial matching region with a normal initial position is determined as the target matching region; the region images of the target matching regions are registered with the local sample image to construct an image stitching model of the target scene.

[0125] In one of the embodiments, the array camera is further used to capture multiple local video streams of the target scene through each of the local cameras, and send each of the local video streams to the server.

[0126] The server is also used to use the image stitching model to perform image stitching and fusion on each group of local video images in each of the local video streams to obtain multiple fused video images, and send the fused video stream generated based on the fused video images to the display terminal.

[0127] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data required or generated for executing the above-mentioned image stitching model construction method or billion-level pixel fusion video generation method. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image stitching model construction method or a billion-level pixel fusion video generation method is implemented.

[0128] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0129] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0130] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0131] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0132] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0133] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0134] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0135] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for constructing an image stitching model for a billion-pixel computational imaging system, characterized in that: The method comprises: Acquire multiple local sample images and a global sample image taken by the array camera of the target scene, and perform region matching on each of the local sample images and the global sample image to obtain an initial position of an initial matching region corresponding to each of the local sample images in the global sample image; Any two initial matching regions with different target row numbers and target column numbers are formed into candidate region pairs, and the initial position loss values ​​of the other initial matching regions are determined based on the initial position of each candidate region pair, and the candidate region pair corresponding to the minimum sum value is determined as the reference region pair, and the initial matching region with abnormal initial position and the initial matching region with normal initial position are determined based on the initial position loss values ​​of the other initial matching regions when the initial position of the reference region pair is used as the reference; For an initial matching region with an abnormal initial position, a correction position corresponding to the initial matching region is obtained by fitting according to the initial positions of other initial matching regions in the same row or column as the initial matching region and with normal initial positions, and the region of the correction position in the global sample image is determined as the target matching region, and the initial matching region with normal initial position is determined as the target matching region; The regional images of each target matching region and the local sample images are registered to construct an image stitching model of the target scene.

2. The method according to claim 1, characterized in that The initial position includes the coordinate information of the pixel points included in the initial matching area; the initial matching area with abnormal initial position is fitted to obtain the correction position corresponding to the initial matching area according to the initial positions of other initial matching areas in the same row or column as the initial matching area and with normal initial positions, including: For each initial matching region with an abnormal initial position, according to the target number of rows and the target number of columns of each initial matching region, in each initial matching region with a normal initial position, a first reference region in the same row as the initial matching region and a second reference region in the same column as the initial matching region are determined; When the number of the first reference areas and the number of the second reference areas are both greater than a preset threshold, a first image fitting line and a second image fitting line are obtained by fitting according to the coordinate information of the pixel points at the first position and the second position in each of the first reference areas, and a third image fitting line and a fourth image fitting line are obtained by fitting according to the coordinate information of the pixel points at the first position and the second position in each of the second reference areas; The coordinate information of the intersection of the first image fitting line and the third image fitting line, and the coordinate information of the second image fitting line and the fourth image fitting line are respectively used as the coordinate information of the pixel points of the first position and the second position in the initial matching area to obtain the corrected position corresponding to the initial matching area.

3. The method according to claim 2, characterized in that The method further comprises: When the number of the first reference area or the second reference area is not greater than a preset threshold, taking the initial position of the reference area pair as a reference, calculating the position to be corrected corresponding to the initial matching area, and determining the coordinate information to be corrected of the pixel points at the first position and the second position in the initial matching area; According to the coordinate information of the pixel points at the first position and the second position in each of the first reference areas and the coordinate information to be corrected, the first image fitting line and the second image fitting line are fitted, and according to the coordinate information of the pixel points at the first position and the second position in each of the second reference areas and the coordinate information to be corrected, the third image fitting line and the fourth image fitting line are fitted.

4. The method according to claim 1, characterized in that: The step of taking the initial positions of the candidate region pairs as references, determining the sum of the initial position loss values ​​of the other initial matching regions, and determining the candidate region pair corresponding to the minimum sum as the reference region pair, comprises: For each pair of candidate regions, any initial matching region in the candidate region pair is used as a candidate region benchmark, and a unit spacing benchmark is calculated according to a spacing between two initial matching regions in the candidate region pair and a number of row and column spacings between the two initial matching regions; Calculate the comparison positions of the other initial matching regions according to the initial position of the candidate region benchmark, the unit spacing benchmark, and the number of row and column intervals between the other initial matching regions and the candidate region benchmark, and use the difference between the comparison position and the initial position as the initial position loss value; The sum of the initial position loss values ​​of the other initial matching regions is calculated to determine the minimum sum, and the candidate region pair corresponding to the minimum sum is determined as the reference region pair.

5. The method according to claim 4, characterized in that The determining of the initial matching region with abnormal initial position and the initial matching region with normal initial position according to the initial position loss values ​​of the other initial matching regions when the initial position of the reference region pair is taken as the reference comprises: Determine the initial position loss values ​​of each of the other initial matching areas when the initial position of the reference area pair is used as a reference, and determine the initial matching area whose initial position loss value is greater than a preset threshold as an initial matching area with an abnormal initial position, and determine the initial matching area whose initial position loss value is not greater than the preset threshold and the two initial matching areas in the reference area pair as initial matching areas with normal initial positions.

6. A method for generating billion-pixel fusion video, characterized in that: The method comprises: Acquire multiple local video streams of the target scene captured by the array camera; Using an image stitching model to stitch and fuse each group of local video images in the multiple local video streams to obtain multiple fused video images, and generating a fused video stream based on the fused video images; Wherein, the image stitching model is constructed according to the image stitching model construction method according to any one of claims 1 to 5.

7. An image stitching model construction device, characterized in that: The device comprises: A matching module, used to obtain multiple local sample images and a global sample image taken by the array camera of the target scene, and perform region matching on each of the local sample images and the global sample image to obtain an initial position of an initial matching region corresponding to each of the local sample images in the global sample image; an abnormality determination module, used to form a candidate region pair from any two initial matching regions with different target row numbers and target column numbers, determine the sum of initial position loss values ​​of other initial matching regions based on the initial position of each candidate region pair, determine the candidate region pair corresponding to the minimum sum as the reference region pair, and determine the initial matching region with an abnormal initial position and the initial matching region with a normal initial position based on the initial position loss values ​​of other initial matching regions based on the initial position of the reference region pair; A correction module is used for fitting, for an initial matching region with an abnormal initial position, a correction position corresponding to the initial matching region according to the initial positions of other initial matching regions in the same row or column as the initial matching region and with normal initial positions, determining the region of the correction position in the global sample image as a target matching region, and determining the initial matching region with a normal initial position as a target matching region; The registration module is used to perform image registration on the regional images of each target matching area and the local sample image to construct an image stitching model of the target scene.

8. A billion-pixel computational imaging system, characterized in that: The billion-pixel computational imaging system includes an array camera and a server, wherein the array camera includes a plurality of local cameras and a global camera, wherein: The array camera is used to capture a local sample image of the target scene through each of the local cameras, capture a global sample image through the global camera, and send the local sample image and the global sample image to the server; The server is used to perform region matching on each of the local sample images and the global sample image to obtain the initial position of the initial matching region corresponding to each of the local sample images in the global sample image; any two initial matching regions with different target row numbers and target column numbers are formed into a candidate region pair, and the sum of the initial position loss values ​​of the other initial matching regions is determined based on the initial position of each of the candidate region pairs, and the candidate region pair corresponding to the minimum sum is determined as the reference region pair, and the initial matching region with an abnormal initial position and the initial matching region with a normal initial position are determined based on the initial position loss values ​​of the other initial matching regions based on the initial position of the reference region pair; for the initial matching region with an abnormal initial position, the correction position corresponding to the initial matching region is obtained by fitting based on the initial positions of other initial matching regions in the same row or column as the initial matching region and with normal initial positions, and the region of the correction position in the global sample image is determined as the target matching region, and the initial matching region with a normal initial position is determined as the target matching region; the region images of the target matching regions are registered with the local sample image to construct an image stitching model of the target scene.

9. The system according to claim 8, characterized in that The array camera is also used to shoot multiple local video streams of the target scene through each local camera, and send each local video stream to the server; The server is also used to use the image stitching model to perform image stitching and fusion on each group of local video images in each of the local video streams to obtain multiple fused video images, and send the fused video stream generated based on the fused video images to the display terminal.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 or 6 are implemented.

Citation Information

Patent Citations

  • An image splicing method based on Shi-Tomasi corner point matching and multi-resolution fusion

    CN109919886A

  • One billion-pixel virtual reality video collection device, system and method

    CN111343367A

  • Image splicing method and system based on hundred million pixel array camera

    CN114841862A

  • System and method for splicing images

    US20180018770A1

  • Local tile-based registration and global placement for stitching

    US20200043134A1