Image stitching model construction method and device for billion-pixel computational imaging system

Abstract

The present application relates to a method and device for constructing an image stitching model for a billion-pixel computational imaging system, a method and device for generating billion-pixel fused videos, a billion-pixel computational imaging system, a computer device, a storage medium, and a computer program product. The method comprises: performing regional matching on each local sample image and the global sample image captured by an array camera to obtain the initial position of each initial matching area; using the initial position of each candidate area pair as a reference, determining a reference area pair with the smallest initial position loss value and an initial matching area with an abnormal initial position; fitting the corrected position of the initial matching area with an abnormal position based on the initial positions of other initial matching areas in the same row or column and with normal initial positions; performing image registration on the regional image of the target matching area at the corrected position and the local sample image to construct an image stitching model. The use of this method can improve the adaptability and accuracy of the image stitching model.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for constructing an image stitching model for a billion-pixel computational imaging system, a method and device for generating a billion-pixel fusion video, a billion-pixel computational imaging system, a computer device, a storage medium, and a computer program product. Background Art

[0002] Video surveillance has been widely used in the security field due to its advantages such as high reliability, timelyness, and ease of viewing. When the monitoring area is large, to achieve ultra-high-definition video surveillance of large scenes, billion-pixel computational imaging systems based on array cameras have emerged. These systems use multiple local cameras in the array to capture multiple HD videos of the target scene. These videos are then stitched together using an image stitching model to create a large-field-of-view, ultra-high-definition fused video with resolution up to billion pixels. These systems are suitable for video capture and security monitoring of large scenes such as airports, highways, parks, sports stadiums, border crossings, and ocean surfaces.

[0003] In related technologies, an array camera typically includes multiple local cameras with long focal lengths and narrow fields of view, and a global camera with short focal lengths and wide fields of view. Before capturing a video of a target scene, an image stitching model can be constructed based on the local sample images captured by the local cameras and the global sample images captured by the global camera. This construction process primarily involves steps such as region matching, image registration, and establishing transformation relationships. Region matching involves matching the region images corresponding to each local sample image within the global sample image. The region images are then registered with the corresponding region images to determine the transformation relationships and fusion parameters corresponding to each local camera, ultimately resulting in an image stitching model.

[0004] However, when some areas in the target scene are pictures such as the sky and water surface that lack feature information, the local images taken by some local cameras will lack feature information. Therefore, in the process of establishing the image stitching model, when the local image is regionally matched with the global image, it is easy to match the regional image with a large position error. As a result, the adaptability and accuracy of the image stitching model established based on the erroneous regional image are low, resulting in poor fusion video imaging quality of the billion-pixel computational imaging system. Summary of the Invention

[0005] Based on this, it is necessary to provide an image stitching model construction method and device, a billion-pixel fusion video generation method and device, a billion-pixel computational imaging system, computer equipment, storage medium and computer program product that can improve the image stitching model accuracy and fusion video imaging quality of the billion-pixel computational imaging system in response to the above technical problems.

[0006] In a first aspect, the present application provides a method for constructing an image stitching model for a billion-pixel computational imaging system. The method comprises:

[0007] Acquire multiple local sample images and a global sample image captured by an array camera of a target scene, and perform region matching on each of the local sample images and the global sample image to obtain an initial position of an initial matching region corresponding to each of the local sample images in the global sample image;

[0008] Any two initial matching regions with different target row numbers and target column numbers are combined into a candidate region pair. The initial position of each candidate region pair is used as a reference to determine the sum of the initial position loss values ​​of the other initial matching regions. The candidate region pair corresponding to the minimum sum is determined as the reference region pair. Based on the initial position loss values ​​of the other initial matching regions when the initial position of the reference region pair is used as a reference, the initial matching regions with abnormal initial positions and the initial matching regions with normal initial positions are determined.

[0009] For an initial matching region with an abnormal initial position, a correction position corresponding to the initial matching region is obtained by fitting based on the initial positions of other initial matching regions in the same row or column as the initial matching region and with normal initial positions, and the region at the correction position in the global sample image is determined as the target matching region, and the initial matching region with normal initial position is determined as the target matching region;

[0010] Perform image registration on the regional images of each target matching region and the local sample images to construct an image stitching model of the target scene.

[0011] In one embodiment, the initial position includes coordinate information of pixels included in the initial matching area; and for an initial matching area with an abnormal initial position, fitting the corrected position corresponding to the initial matching area based on the initial positions of other initial matching areas in the same row or column as the initial matching area and with normal initial positions includes:

[0012] For each initial matching region with an abnormal initial position, determining, based on the target number of rows and the target number of columns of each initial matching region, a first reference region in the same row and a second reference region in the same column as the initial matching region in each initial matching region with a normal initial position;

[0013] When the number of the first reference areas and the number of the second reference areas are both greater than a preset threshold, fitting a first image fitting line and a second image fitting line based on the coordinate information of the pixel points at the first position and the second position in each of the first reference areas, and fitting a third image fitting line and a fourth image fitting line based on the coordinate information of the pixel points at the first position and the second position in each of the second reference areas;

[0014] The coordinate information of the intersection of the first image fitting line and the third image fitting line, and the coordinate information of the second image fitting line and the fourth image fitting line are respectively used as the coordinate information of the pixel points at the first position and the second position in the initial matching area to obtain the corrected position corresponding to the initial matching area.

[0015] In one embodiment, the method further comprises:

[0016] When the number of the first reference area or the second reference area is not greater than a preset threshold, calculating the position to be corrected corresponding to the initial matching area based on the initial position of the reference area pair, and determining the coordinate information to be corrected of the pixel points at the first position and the second position in the initial matching area;

[0017] Based on the coordinate information of the pixel points at the first position and the second position in each of the first reference areas and the coordinate information to be corrected, the first image fitting line and the second image fitting line are fitted, and based on the coordinate information of the pixel points at the first position and the second position in each of the second reference areas and the coordinate information to be corrected, the third image fitting line and the fourth image fitting line are fitted.

[0018] In one embodiment, the determining the sum of the initial position loss values ​​of the other initial matching regions based on the initial position of each candidate region pair, and determining the candidate region pair corresponding to the minimum sum as the reference region pair, includes:

[0019] For each candidate region pair, any initial matching region in the candidate region pair is used as a candidate region benchmark, and a unit spacing benchmark is calculated according to the spacing between the two initial matching regions in the candidate region pair and the number of row and column intervals between the two initial matching regions;

[0020] Calculating the comparison positions of each of the other initial matching regions based on the initial position of the candidate region benchmark, the unit spacing benchmark, and the number of row and column intervals between each of the other initial matching regions and the candidate region benchmark, and using the difference between the comparison position and the initial position as the initial position loss value;

[0021] The sum of the initial position loss values ​​of the other initial matching regions is calculated to determine a minimum sum, and the candidate region pair corresponding to the minimum sum is determined as the reference region pair.

[0022] In one embodiment, determining the initial matching region with abnormal initial position and the initial matching region with normal initial position based on the initial position loss values ​​of the other initial matching regions when the initial position of the reference region pair is used as a reference includes:

[0023] Determine the initial position loss values ​​of each of the other initial matching areas when the initial position of the reference area pair is used as a reference, and determine the initial matching area whose initial position loss value is greater than a preset threshold as an initial matching area with an abnormal initial position, and determine the initial matching area whose initial position loss value is not greater than the preset threshold and the two initial matching areas in the reference area pair as initial matching areas with normal initial positions.

[0024] In a second aspect, the present application also provides a method for generating billion-pixel fusion videos. The method comprises:

[0025] Obtain multiple local video streams captured by array cameras of the target scene;

[0026] Using an image stitching model to perform image stitching and fusion on each group of local video images in the multiple local video streams to obtain multiple fused video images, and generating a fused video stream based on the fused video images;

[0027] The image stitching model is constructed according to the image stitching model construction method described in the first aspect.

[0028] In a third aspect, the present application further provides an image stitching model construction device. The device comprises:

[0029] a matching module, configured to obtain a plurality of local sample images and a global sample image captured by the array camera of the target scene, and perform region matching on each of the local sample images and the global sample image to obtain an initial position of an initial matching region corresponding to each of the local sample images in the global sample image;

[0030] an abnormality determination module, configured to group any two initial matching regions having different numbers of target rows and columns into candidate region pairs, determine the sum of initial position loss values ​​of the other initial matching regions using the initial position of each candidate region pair as a reference, determine the candidate region pair corresponding to the minimum sum as the reference region pair, and determine initial matching regions with abnormal initial positions and initial matching regions with normal initial positions based on the initial position loss values ​​of the other initial matching regions using the initial position of the reference region pair as a reference;

[0031] a correction module for fitting, for an initial matching region with an abnormal initial position, a correction position corresponding to the initial matching region based on the initial positions of other initial matching regions in the same row or column as the initial matching region and with normal initial positions, determining the region at the correction position in the global sample image as a target matching region, and determining the initial matching region with normal initial positions as the target matching region;

[0032] The registration module is used to perform image registration on the regional images of each target matching area and the local sample images to construct an image stitching model of the target scene.

[0033] In a fourth aspect, the present application further provides a device for generating billion-pixel fusion videos. The device comprises:

[0034] An acquisition module is used to acquire multiple local video streams shot by the array camera of the target scene;

[0035] A fusion module is used to use an image stitching model to perform image stitching and fusion on each group of local video images in the multiple local video streams to obtain multiple fused video images, and generate a fused video stream based on the fused video images; wherein the image stitching model is constructed according to the image stitching model construction method described in the first aspect.

[0036] In a fifth aspect, the present application further provides a billion-pixel computational imaging system. The billion-pixel computational imaging system includes an array camera and a server, wherein the array camera includes multiple local cameras and a global camera, wherein:

[0037] The array camera is used to capture a local sample image of the target scene through each of the local cameras, capture a global sample image through the global camera, and send the local sample image and the global sample image to the server;

[0038] The server is used to perform region matching on each of the local sample images and the global sample image to obtain the initial position of the initial matching region corresponding to each of the local sample images in the global sample image; any two initial matching regions with different target row numbers and target column numbers are combined into a candidate region pair, and the sum of the initial position loss values ​​of the other initial matching regions is determined based on the initial position of each candidate region pair, and the candidate region pair corresponding to the minimum sum is determined as the reference region pair; and based on the initial position loss values ​​of the other initial matching regions when the initial position of the reference region pair is used as the reference, an initial matching region with an abnormal initial position and an initial matching region with a normal initial position are determined; for the initial matching region with an abnormal initial position, a correction position corresponding to the initial matching region is obtained by fitting based on the initial positions of other initial matching regions in the same row or column as the initial matching region and with normal initial positions, the region of the correction position in the global sample image is determined as the target matching region, and the initial matching region with a normal initial position is determined as the target matching region; the region images of the target matching regions are image registered with the local sample image to construct an image stitching model of the target scene.

[0039] In a sixth aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in the first aspect or the second aspect are implemented.

[0040] In a seventh aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect or the second aspect.

[0041] In an eighth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect or the second aspect.

[0042] The above-mentioned image stitching model construction method and device of the billion-pixel computational imaging system, billion-pixel fusion video generation method and device, billion-pixel computational imaging system, computer equipment, storage medium and computer program product, judge whether the initial position of each initial matching area is abnormal through the initial position of each initial matching area obtained by area matching. For the initial matching area with an abnormal initial position, the corrected position is obtained by fitting the initial position of the initial matching area with a normal initial position, and then the target matching area corresponding to the corrected position and the initial matching area with a normal initial position are aligned with each local sample image to determine the transformation relationship and fusion parameters to obtain the image stitching model.

[0043] Among them, to determine whether the initial position of each initial matching area is abnormal, the initial position loss value of each initial matching area is calculated based on the initial position of each pair of candidate areas, and then the candidate area pair with the smallest sum of the initial position loss values ​​is used as the reference area pair. The matching accuracy of the initial position of this reference area pair is the highest, so that according to the initial position loss values ​​of other initial matching areas calculated under this reference, it is possible to accurately determine whether the initial position of each initial matching area is abnormal. Then, based on the initial position of the normal initial matching area, the corrected position of the abnormal initial matching area is fitted to achieve position correction of the abnormal matching area. The matching accuracy of the target matching area at the corrected position is higher, thereby improving the matching degree and accuracy between the image stitching model and the target scene, which is beneficial to improving the fusion video imaging quality of the billion-pixel computational imaging system. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of an example of a billion-pixel computational imaging system;

[0045] Figure 2 A schematic diagram of a flow chart of a method for constructing an image stitching model in one embodiment;

[0046] Figure 3a is a schematic diagram of an array camera in an example;

[0047] Figure 3b A schematic diagram of the target matching area in a global sample image in an example;

[0048] Figure 3c A schematic diagram of the initial matching area in a global sample image in an example;

[0049] Figure 4 A structural block diagram of an image stitching model construction device in one embodiment;

[0050] Figure 5 This is a structural block diagram of a device for generating billion-pixel fusion video in one embodiment;

[0051] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0053] First, before specifically introducing the technical solutions of the embodiments of the present application, the technical background or technical evolution context on which the embodiments of the present application are based is introduced. In order to achieve ultra-high-definition video surveillance of large scenes, a billion-pixel computational imaging system based on array cameras has emerged. The array camera usually includes multiple local cameras with long focal lengths and narrow fields of view and a global camera with short focal lengths and wide fields of view. The field of view of the global camera can cover the field of view of each local camera. The local cameras capture local high-definition images rich in detail information of the target scene, and the global camera captures global images with lower resolution. Before capturing a video of the target scene, an image stitching model can be constructed based on the local sample images captured by the local cameras and the global sample images captured by the global camera. When capturing a video of the target scene, the image stitching model can be used to stitch and fuse the multiple local high-definition videos captured by the local cameras, and output a large or wide field of view ultra-high-definition fused video up to the billion-pixel level in real time. This is suitable for video capture and security monitoring of large scenes such as airports, highways, parks, sports stadiums, borders, and sea surfaces.

[0054] The process of establishing an image stitching model for a billion-pixel computational imaging system mainly includes steps such as regional matching, image registration, and determining transformation relationships and fusion parameters. Regional matching refers to matching the regional images corresponding to each local sample image in the global sample image, and then performing image registration on each local sample image and the corresponding regional image to determine the transformation relationship between the two images. This transformation relationship is used to geometrically transform the local sample image to obtain a transformed image that is consistent with the shooting angle of the regional image (i.e., the shooting angle of the global sample image), completing the unified coordinate transformation, and then determining the fusion parameters based on the unified coordinates of each transformed image. The fusion parameters are used to fuse the transformed images to obtain a large field of view, ultra-high resolution fused image. The transformation relationships and fusion parameters corresponding to each local camera can be constructed into an image stitching model to stitch and fuse the local videos shot by the array camera to output an ultra-high-definition fused video.

[0055] However, in some scenarios, such as when some areas in the target scene are the sky, water surface, etc. that lack feature information or contour information, the local images taken by some local cameras lack feature information. Therefore, in the process of establishing the image stitching model, when the local image is matched with the global image for region matching, it is easy to match the regional image with a large position error, and then the erroneous regional image is matched with the local image to determine the transformation relationship and fusion parameters. The adaptability and accuracy of the obtained image stitching model are low, resulting in poor imaging quality of the fused video image obtained by stitching and fusion, which affects the imaging quality of the billion-pixel computing imaging system.

[0056] Based on this background, the applicant, through long-term research and development and experimental verification, has proposed the image stitching model construction method of the billion-pixel computational imaging system of this application, which can improve the matching degree and accuracy of the image stitching model with the target scene, and is conducive to improving the fusion video imaging quality of the billion-pixel computational imaging system. In addition, it should be noted that the applicant has made a lot of creative efforts in discovering the technical problem of this application and the technical solutions introduced in the following embodiments.

[0057] The image stitching model construction method provided in the embodiment of the present application can be applied to Figure 1 The illustrated exascale pixel computational imaging system. Array camera 102 and display terminal 106 each communicate with server 104 via a network. Display terminal 106 can be, but is not limited to, various electronic devices with display units, such as personal computers, laptops, smartphones, tablets, smart displays, IoT devices, and portable wearable devices. Server 104 can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0058] In one embodiment, Figure 2 As shown, a method for constructing an image stitching model is provided, which can be applied to Figure 1 In this embodiment, the method includes the following steps:

[0059] Step 201: obtain multiple local sample images and global sample images captured by the array camera of the target scene, and perform region matching on each local sample image and the global sample image to obtain the initial position of the initial matching region corresponding to each local sample image in the global sample image.

[0060] In practice, the array camera can be installed in the environment of the target scene to capture a video of the target scene. Before the actual video capture of the target scene, the array camera can first collect sample images of the target scene to construct an image stitching model. This image stitching model can then be used to perform real-time image stitching and fusion during subsequent video capture, improving the efficiency of generating the fused video.

[0061] The array camera may include multiple local cameras distributed in multiple rows and columns and a global camera. Figure 3a The array camera shown includes 18 local cameras (or local lenses) arranged in 3 rows and 6 columns, and a global camera (or global lens) in the center. Each local camera can be identified by its row and column information. For example, looking from the back end of the camera toward the front end, the local camera in the first row and first column can be identified as A11. For ease of uniform representation, the local image captured by a local camera Aij (i represents the row number, j represents the column number) and the corresponding regional image in the global image can both be identified as Aij.

[0062] The shooting angles of each local camera are different. Usually, the lens optical axes of two local cameras in the same row have a certain angle in the horizontal direction, and the lens optical axes of two local cameras in the same column have a certain angle in the vertical direction (understandably, due to Figure 3a The center row of the array camera shown features a global camera. To maintain a compact overall structure, the lenses of the local cameras in this row are not aligned vertically with the lenses of the local cameras in the same column in other rows, but this does not affect the row and column relationship of the local cameras. Typically, the global camera's field of view is larger than that of the local cameras, and its focal length is smaller. The global camera's field of view can encompass the fields of view of each local camera, and the fields of view of adjacent local cameras can overlap to some extent.

[0063] The server can use template matching algorithm or other methods to perform regional matching between each local sample image (which can be scaled down to a certain ratio) and the global sample image, and can use the matched area with the greatest similarity as the initial matching area. The purpose of regional matching is to match the target matching area image of the same local scene (part of the target scene) captured by the global camera and the local camera. The target position of each local sample image in the target matching area in the global sample image A0 (if the match is correct) can be as follows: Figure 3b It is understandable that, due to the different parameters such as shooting angle, focal length, and resolution of the local camera and the global camera, the images taken of the same local scene are only similar images, not exactly the same.

[0064] In actual application, if some local sample images and the corresponding regional images in the global sample image lack feature information, the initial position of the matched initial matching area may have a large error. The initial matching area of ​​each local sample image in the global sample image can be as follows: Figure 3c As shown in this example, the initial position errors of the initial matching areas A13 and A15 are large and seriously deviate from the correct position (refer to Figure 3b The initial position of the initial matching area can be represented by the coordinates of the upper left and lower right pixel points of the area (x0, y0, x1, y1). It should be noted that in order to facilitate the stitching and fusion to obtain a seamless fusion image, the shooting fields of adjacent local cameras usually have overlapping areas, and the corresponding matching areas also have overlapping areas. In addition, due to factors such as the different distances between the photographed objects in each local scene of the target scene and the array camera, the sizes of the matching areas corresponding to each local sample image are not necessarily the same. In order to clearly show the row and column relationship of each matching area, Figure 3b and Figure 3c The overlapping area is not shown, nor is the size of each matching area distinguished. This is only for simple illustration.

[0065] In step 202, any two initial matching areas with different target row numbers and target column numbers are grouped into candidate area pairs. The initial position of each candidate area pair is used as a reference to determine the sum of the initial position loss values ​​of the other initial matching areas. The candidate area pair corresponding to the minimum sum value is determined as the reference area pair. Based on the initial position loss values ​​of the other initial matching areas when the initial position of the reference area pair is used as a reference, the initial matching areas with abnormal initial positions and the initial matching areas with normal initial positions are determined.

[0066] The target row and column numbers refer to the row and column numbers where each matching area is located when the area is correctly matched. Typically, once the installation angle (shooting angle) of each local camera in an array camera is determined, the row and column numbers of the matching area corresponding to the local image captured by each local camera can be determined, and the target row and column numbers corresponding to each local camera can be pre-stored. It should be noted that due to the potential for significant error in the initial positions of the initial matching areas, the actual row and column numbers of each initial matching area may differ from the target row and column numbers.

[0067] The target number of rows and target number of columns of the two initial matching regions in the candidate region pair are different. Figure 3c As shown, the initial matching area A11 (in order to clearly show the matching area with the wrong position, the identification of the matching area with the correct position is omitted in this figure. The identification of each matching area can be found in Figure 3b ) can form candidate area pairs with matching areas A22 to A26 and A32 to A36. For the 3-row 6-column array camera, 90 candidate area pairs can be formed.

[0068] The initial position loss value describes the difference between the initial position of the initial matching region and the reference position determined based on the initial position of the candidate region pair. The initial position of the candidate region pair includes the initial positions of the two initial matching regions, which can reflect the coordinate information of the two initial matching regions in the panoramic image and the distance (spacing) between the two initial matching regions.

[0069] In one implementation, the process of calculating the sum of the initial position loss values ​​based on each candidate region pair and determining the reference region pair specifically includes the following steps:

[0070] In step 2021 , for each candidate region pair, any initial matching region in the candidate region pair is used as a candidate region benchmark, and a unit spacing benchmark is calculated based on the spacing between the two initial matching regions in the candidate region pair and the number of row and column intervals between the two initial matching regions.

[0071] Among them, the spacing between the two initial matching areas includes row spacing and column spacing, which can be specifically calculated based on the coordinate information of the pixel point at the center position of the initial matching area. The unit spacing benchmark can include a unit row spacing benchmark and a unit column spacing benchmark. The unit row spacing benchmark can be obtained by dividing the row spacing by the number of row spacings, and the unit column spacing benchmark can be obtained by dividing the column spacing by the number of column spacings. The number of row and column spacings can be calculated based on the target number of rows and target number of columns of each initial matching area (or each local camera). For example, the center point coordinates of the initial matching area A11 are (x c11 ,y c11 ), the center coordinate of the initial matching area A23 is (x c23 ,y c23 ), the line spacing between the two is Lr=|x c11 -x c23 |, row spacing Nr = 1, column spacing Lc = |y c11 -y c23 |, the number of column intervals Nc = 2. The center point coordinates can be calculated based on the coordinates of the pixel points in the upper left corner and lower right corner of the region (x0, y0) and (x1, y1), or the coordinates of the pixel points in the lower left corner and upper right corner (x2, y2) and (x3, y3).

[0072] The candidate region benchmark can be any initial matching region in the candidate region pair, and can be randomly selected, or the initial matching region with the smaller number of rows in the region pair can be used as the candidate region benchmark when traversing the candidate region pairs corresponding to each initial matching region in sequence.

[0073] In step 2022, based on the initial position of the candidate region benchmark, the unit spacing benchmark, and the number of row and column intervals between the other initial matching regions and the candidate region benchmark, the comparison positions of the other initial matching regions are calculated, and the difference between the comparison position and the initial position is used as the initial position loss value.

[0074] Among them, the initial position loss value can be calculated based on the coordinates of one or more pixel points at the center point, four corner points (upper left, lower left, upper right, lower right) or other positions. In one example, since the initial position of the initial matching area can be represented by the coordinates of the upper left and lower right pixel points of the area (x0, y0, x1, y1), the initial position loss value can be calculated using the coordinates of the upper left and lower right corners. The difference can be obtained based on the coordinate difference of the pixel points at the same position (such as the upper left corner and the lower right corner).

[0075] In one example, the initial position of the candidate region reference in the candidate region pair can be expressed as (x 0_b ,y 0_b ,x 1_b ,y 1_b), the number of rows of the candidate region reference is br, the number of columns is bc, the unit row spacing reference is Sr, the unit column spacing reference is Sc, and the other initial matching regions A except the candidate region pair ij The initial position of (row number is i, column number is j) can be expressed as (x 0_Aij ,y 0_Aij ,x 1_Aij ,y 1_Aij ), the initial matching area A calculated based on the candidate area pair ij The control position can be expressed as The calculation formula is as follows:

[0076]

[0077]

[0078]

[0079]

[0080] Initial matching area A ij The calculation formula of the initial position loss value Lij is as follows:

[0081]

[0082] The server can calculate other initial matching areas A ij The sum of the loss values ​​Lij is used as the sum of the initial position loss values ​​corresponding to the candidate region pair.

[0083] Step 2023 , calculate the sum of the initial position loss values ​​of the other initial matching regions, determine the minimum sum, and determine the candidate region pair corresponding to the minimum sum as the reference region pair.

[0084] During implementation, the server may compare the sums of the initial position loss values ​​respectively calculated when each candidate area pair is used as a reference, determine the minimum sum, and determine the area pair corresponding to the minimum sum as the reference area pair.

[0085] The server can then compare the initial position loss values ​​of each of the other initial matching regions (excluding the two initial matching regions in the reference region pair) calculated using the reference region pair as a reference with a preset threshold. If the initial position loss value is greater than the preset threshold, it indicates that the comparison position calculated based on the position and spacing of the reference region pair with higher matching accuracy is significantly different from the initial position, and the initial position can be judged to be abnormal. If the initial position loss value is not greater than the preset threshold, the initial position is judged to be normal. In addition, the two initial matching regions in the reference region pair are also judged to have normal initial positions.

[0086] Step 203, for the initial matching area with an abnormal initial position, the correction position corresponding to the initial matching area is fitted according to the initial positions of other initial matching areas in the same row or column as the initial matching area and with normal initial positions, and the area of ​​the correction position in the global sample image is determined as the target matching area, and the initial matching area with normal initial position is determined as the target matching area.

[0087] In implementation, for the initial matching area with an abnormal initial position, the server can perform least squares fitting based on the coordinate information of the target position pixel points of the initial matching area with normal initial position in the same row (same target row number) as the initial matching area to obtain a first fitting relationship, and perform least squares fitting based on the coordinate information of the target position pixel points of the initial matching area with normal initial position in the same column (same target column number) as the initial matching area to obtain a second fitting relationship, and then based on the first fitting relationship and the second fitting relationship, determine the coordinate information of the corresponding position pixel points of the initial matching area as the correction position.

[0088] In one implementation, the process of determining the corrected position of the initial matching area whose initial position is abnormal includes the following steps:

[0089] Step 2031, for each initial matching area with an abnormal initial position, according to the target number of rows and target number of columns of each initial matching area, in each initial matching area with a normal initial position, determine a first reference area in the same row and a second reference area in the same column as the initial matching area.

[0090] For example, Figure 3c As shown, for the initial matching area A13 with an abnormal initial position, its first reference area includes A11, A12, A14, and A16, and its second reference area includes A23 and A33.

[0091] In step 2032, when the number of the first reference areas and the second reference areas is greater than a preset threshold, the first image fitting line and the second image fitting line are fitted according to the coordinate information of the pixel points at the first position and the second position in each first reference area, and the third image fitting line and the fourth image fitting line are fitted according to the coordinate information of the pixel points at the first position and the second position in each second reference area.

[0092] Among them, the preset threshold can be a directly set number, or it can be a preset ratio of the set initial position normal area to the initial position abnormal area or the total number of areas, and then the preset threshold is calculated based on the total number of matching areas in each row and column and the preset ratio.

[0093] If the number of both the first reference area and the second reference area is greater than a preset threshold, the abnormal area can be corrected based on the reference area with normal initial position. Specifically, the coordinate information of the pixel points at the first position and the second position in each first reference area is respectively subjected to least squares fitting to obtain the first image fitting line and the second image fitting line. The coordinate information of the pixel points at the first position and the second position in each second reference area is respectively subjected to least squares fitting to obtain the third image fitting line and the fourth image fitting line. The coordinate information of the pixel points at the first position and the second position can uniquely determine the position information of a rectangular area, for example, the upper left corner and the lower right corner, the upper right corner and the lower left corner, or any corner point and the center point of the rectangular area.

[0094] Step 2033: Use the coordinate information of the intersection of the first image fitting line and the third image fitting line, and the coordinate information of the second image fitting line and the fourth image fitting line as the coordinate information of the pixel points at the first position and the second position in the initial matching area, respectively, to obtain the corrected position corresponding to the initial matching area.

[0095] In implementation, after determining the first image fitting line (horizontally) and the third image fitting line (vertically) corresponding to the first position (e.g., the upper left corner), the coordinate information of the first intersection of the two fitting lines can be obtained. After determining the second image fitting line (horizontally) and the fourth image fitting line (vertically) corresponding to the second position (e.g., the lower right corner), the coordinate information of the second intersection of the two fitting lines can be obtained. The server can use the rectangular area determined by the two intersections as the target matching area after position correction.

[0096] In another example, if it is determined that the number of first reference areas or second reference areas is not greater than a preset threshold, that is, the number of initial matching areas with correct initial positions in the row or column is small, a large deviation will occur in the position fitting. Therefore, the server can use the initial position of the reference area pair as a reference to calculate the position to be corrected corresponding to the initial matching area and make preliminary adjustments to the position of the initial matching area. Specifically, the server can calculate the reference coordinates (coordinate information to be corrected) of the pixel points at the first position and the second position of the candidate area benchmark of the reference area pair, the unit spacing reference (row spacing reference and column spacing reference) of the reference area pair, and the number of row and column spacings between the initial matching area and the candidate area benchmark, and use the reference coordinates of the two pixel points to determine the position to be corrected of the initial matching area. Then, the server can fit the first image fitting line and the second image fitting line based on the coordinate information of the pixel points at the first position and the second position of each first reference area and the coordinate information to be corrected, and fit the third image fitting line and the fourth image fitting line based on the coordinate information of the pixel points at the first position and the second position of each second reference area and the coordinate information to be corrected. The corrected position of the initial matching area is then determined based on the intersection of each fitting line. That is, if the number of normal matching areas in the same row or column is small, the server can use the initial position of the abnormal matching area, after preliminary adjustment, to adjust the position to be corrected, for fitting, which can improve the correction accuracy when the number of normal matching areas is small.

[0097] Then, the server can determine the area with the corrected position in the global sample image as the target matching area, and determine the initial matching area with a normal initial position as the target matching area, thereby obtaining target matching areas with higher matching accuracy corresponding to each local sample image.

[0098] Step 204 : performing image registration on the regional image of each target matching region and the local sample image to construct an image stitching model of the target scene.

[0099] During implementation, the server can perform image registration on each local sample image and the regional image of its corresponding target matching area (which can be enlarged to the same size as the local sample image), determine the transformation relationship or image mapping relationship and corresponding fusion parameters between the two, and obtain the image stitching model corresponding to the target scene.

[0100] In the above-mentioned image stitching model construction method, the initial positions of the initial matching regions obtained by region matching are used to determine whether the initial positions of the initial matching regions are abnormal. For the initial matching regions with abnormal initial positions, the corrected positions are obtained based on the initial positions of the initial matching regions with normal initial positions. Then, the target matching regions corresponding to the corrected positions and the initial matching regions with normal initial positions are image-aligned with the local sample images to determine the transformation relationship and fusion parameters to obtain the image stitching model. In particular, to determine whether the initial positions of the initial matching regions are abnormal, the initial position loss values ​​of the initial matching regions are calculated based on the initial positions of each pair of candidate regions. Then, the candidate region pair with the smallest sum of the initial position loss values ​​is used as the reference region pair. The matching accuracy of the initial position of the reference region pair is the highest, so that the initial position loss values ​​of the other initial matching regions calculated under this reference can be used to accurately determine whether the initial positions of the initial matching regions are abnormal. Then, based on the initial position of the normal initial matching area, the corrected position of the abnormal initial matching area is fitted to realize the position correction of the abnormal matching area. The matching accuracy of the target matching area in the corrected position is higher, thereby improving the matching degree and accuracy between the image stitching model and the target scene, which is beneficial to improving the fused video imaging quality of the billion-pixel computational imaging system.

[0101] The present application also provides a method for generating billion-pixel fusion video. The method can be applied to Figure 1 In this embodiment, the method includes the following steps:

[0102] Step 1: Obtain multiple local video streams captured by array cameras of a target scene.

[0103] Step 2: Use an image stitching model to stitch and fuse each group of local video images in the multiple local video streams to obtain multiple fused video images, and generate a fused video stream based on the fused video images.

[0104] The image stitching model is constructed according to the image stitching model construction method provided in the above embodiments.

[0105] In implementation, before the server shoots a video of the target scene, it can first use the image stitching model construction method in the above embodiment to build an image stitching model corresponding to the target scene. Then, when officially shooting the video, the array camera can shoot local video streams respectively through multiple local cameras, and then send the local video streams to the server. The server can use the image stitching model to stitch and fuse a group of local video images shot by each local camera at the same time or similar time (there may be an acceptable time difference due to the inconsistent exposure time of each local camera), including performing image transformation on each local video image based on the transformation relationship corresponding to each local camera in the image stitching model, obtaining a transformed image with unified coordinates corresponding to each local video image, and then fusing each transformed image based on the fusion parameters in the image stitching model to obtain a fused video image of the current frame. The server can generate a fused video stream from each frame of fused video image and send it to the display terminal for display or playback.

[0106] In the above-mentioned billion-pixel fusion video generation method, the image stitching model construction process takes into account the possibility that some areas in the target scene may lack feature information, resulting in inaccurate regional matching. Therefore, the initial matching area with an abnormal initial position is corrected to obtain a target matching area with a more accurate position, and then an image stitching model with a high degree of matching and higher accuracy with the target scene is constructed. When it is used to stitch and fuse local video images, the obtained fused video image has higher accuracy and higher imaging quality.

[0107] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0108] Based on the same inventive concept, the present application also provides an image stitching model construction device for implementing the aforementioned image stitching model construction method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more of the following image stitching device embodiments can be found in the above-mentioned limitations of the image stitching model construction method and will not be repeated here.

[0109] In one embodiment, Figure 4 As shown, an image stitching model construction device 400 is provided, comprising: a matching module 401, an abnormality determination module 402, a correction module 403 and a registration module 404, wherein:

[0110] The matching module 401 is used to obtain multiple local sample images and global sample images taken by the array camera of the target scene, and perform regional matching on each of the local sample images and the global sample image to obtain the initial position of the initial matching area corresponding to each of the local sample images in the global sample image.

[0111] The abnormality determination module 402 is used to form a candidate area pair from any two initial matching areas with different target row numbers and target column numbers, and to determine the sum of the initial position loss values ​​of the other initial matching areas based on the initial position of each candidate area pair, and to determine the candidate area pair corresponding to the minimum sum value as the reference area pair, and to determine the initial matching areas with abnormal initial positions and the initial matching areas with normal initial positions based on the initial position loss values ​​of the other initial matching areas based on the initial position of the reference area pair.

[0112] The correction module 403 is used to fit the correction position corresponding to the initial matching area with an abnormal initial position based on the initial positions of other initial matching areas in the same row or column as the initial matching area and with normal initial positions, determine the area of ​​the correction position in the global sample image as the target matching area, and determine the initial matching area with normal initial positions as the target matching area.

[0113] The registration module 404 is configured to perform image registration on the regional images of each target matching region and the local sample images to construct an image stitching model of the target scene.

[0114] In one embodiment, the initial position includes the coordinate information of the pixel points included in the initial matching area. The correction module 403 is specifically used to: for each initial matching area with an abnormal initial position, according to the target number of rows and the target number of columns of each initial matching area, determine the first reference area in the same row as the initial matching area and the second reference area in the same column as the initial matching area in each initial matching area with a normal initial position; when the number of the first reference area and the second reference area is greater than a preset threshold, according to the coordinate information of the pixel points at the first position and the second position in each first reference area, fit the first image fitting line and the second image fitting line, and according to the coordinate information of the pixel points at the first position and the second position in each second reference area, fit the third image fitting line and the fourth image fitting line; use the coordinate information of the intersection of the first image fitting line and the third image fitting line, and the coordinate information of the second image fitting line and the fourth image fitting line as the coordinate information of the pixel points at the first position and the second position in the initial matching area, respectively, to obtain the correction position corresponding to the initial matching area.

[0115] In one embodiment, the correction module 403 is also used to: when the number of the first reference area or the second reference area is not greater than a preset threshold, calculate the position to be corrected corresponding to the initial matching area based on the initial position of the reference area pair, and determine the coordinate information to be corrected of the pixel points of the first position and the second position in the initial matching area; fit the first image fitting line and the second image fitting line according to the coordinate information of the pixel points of the first position and the second position in each of the first reference areas, and the coordinate information to be corrected; fit the third image fitting line and the fourth image fitting line according to the coordinate information of the pixel points of the first position and the second position in each of the second reference areas, and the coordinate information to be corrected.

[0116] In one embodiment, the abnormality determination module 402 is specifically used to: for each pair of candidate area pairs, take any initial matching area in the candidate area pair as the candidate area benchmark, and calculate the unit spacing benchmark based on the spacing between the two initial matching areas in the candidate area pair and the number of row and column intervals between the two initial matching areas; calculate the comparison position of each other initial matching area based on the initial position of the candidate area benchmark, the unit spacing benchmark, and the number of row and column intervals between each other initial matching area and the candidate area benchmark, and take the difference between the comparison position and the initial position as the initial position loss value; calculate the sum of the initial position loss values ​​of each other initial matching area, determine the minimum sum value, and determine the candidate area pair corresponding to the minimum sum value as the benchmark area pair.

[0117] In one embodiment, the abnormality determination module 402 is specifically used to: determine the initial position loss values ​​of each of the other initial matching areas when the initial position of the reference area pair is used as a reference, and determine the initial matching area whose initial position loss value is greater than a preset threshold as an initial matching area with an abnormal initial position, and determine the initial matching area whose initial position loss value is not greater than the preset threshold and the two initial matching areas in the reference area pair as initial matching areas with normal initial positions.

[0118] In one embodiment, Figure 5 As shown, a device 500 for generating a billion-pixel fusion video is also provided, comprising an acquisition module 501 and a fusion module 502, wherein:

[0119] The acquisition module 501 is used to acquire multiple local video streams captured by the array camera of the target scene.

[0120] The fusion module 502 is used to use an image stitching model to perform image stitching and fusion on each group of local video images in multiple local video streams to obtain multiple fused video images, and generate a fused video stream based on the fused video images; wherein the image stitching model is constructed according to the image stitching model construction method provided in the above embodiment.

[0121] Each module in the aforementioned image stitching model construction device and billion-pixel fusion video generation device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a computer device's memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0122] In one embodiment, a billion-pixel computational imaging system is provided. The billion-pixel computational imaging system includes an array camera and a server, wherein the array camera includes multiple local cameras and a global camera, wherein:

[0123] The array camera is used to capture a local sample image of the target scene through each of the local cameras, capture a global sample image through the global camera, and send the local sample image and the global sample image to the server;

[0124] The server is used to perform region matching on each of the local sample images and the global sample image to obtain the initial position of the initial matching region corresponding to each of the local sample images in the global sample image; any two initial matching regions with different target row numbers and target column numbers are combined into a candidate region pair, and the sum of the initial position loss values ​​of the other initial matching regions is determined based on the initial position of each candidate region pair, and the candidate region pair corresponding to the minimum sum is determined as the reference region pair; and based on the initial position loss values ​​of the other initial matching regions when the initial position of the reference region pair is used as the reference, an initial matching region with an abnormal initial position and an initial matching region with a normal initial position are determined; for the initial matching region with an abnormal initial position, a correction position corresponding to the initial matching region is obtained by fitting based on the initial positions of other initial matching regions in the same row or column as the initial matching region and with normal initial positions, the region of the correction position in the global sample image is determined as the target matching region, and the initial matching region with a normal initial position is determined as the target matching region; the region images of the target matching regions are image registered with the local sample image to construct an image stitching model of the target scene.

[0125] In one embodiment, the array camera is further configured to capture multiple local video streams of the target scene through each of the local cameras, and send each of the local video streams to the server.

[0126] The server is further configured to perform image stitching and fusion on each group of local video images in each of the local video streams using the image stitching model to obtain multiple fused video images, and to send a fused video stream generated based on the fused video images to a display terminal.

[0127] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data required or generated for executing the above-mentioned image stitching model construction method or billion-level pixel fusion video generation method. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an image stitching model construction method or a billion-level pixel fusion video generation method is implemented.

[0128] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0129] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0130] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0131] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0132] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0133] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0134] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0135] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for constructing an image stitching model for a billion-pixel computational imaging system, characterized in that: The method comprises: Acquire multiple local sample images and a global sample image captured by an array camera of a target scene, and perform region matching on each of the local sample images and the global sample image to obtain an initial position of an initial matching region corresponding to each of the local sample images in the global sample image; Any two initial matching regions with different target row numbers and target column numbers are combined into a candidate region pair. The initial position of each candidate region pair is used as a reference to determine the sum of the initial position loss values ​​of the other initial matching regions. The candidate region pair corresponding to the minimum sum is determined as the reference region pair. Based on the initial position loss values ​​of the other initial matching regions when the initial position of the reference region pair is used as a reference, the initial matching regions with abnormal initial positions and the initial matching regions with normal initial positions are determined. For an initial matching region with an abnormal initial position, a correction position corresponding to the initial matching region is obtained by fitting based on the initial positions of other initial matching regions in the same row or column as the initial matching region and with normal initial positions, and the region at the correction position in the global sample image is determined as the target matching region, and the initial matching region with normal initial position is determined as the target matching region; Performing image registration on the regional images of each target matching area and the local sample images to construct an image stitching model of the target scene; The step of determining the sum of the initial position loss values ​​of the other initial matching regions based on the initial position of each candidate region pair, and determining the candidate region pair corresponding to the minimum sum as the reference region pair, includes: For each candidate region pair, any initial matching region in the candidate region pair is used as a candidate region benchmark, and a unit spacing benchmark is calculated according to the spacing between the two initial matching regions in the candidate region pair and the number of row and column intervals between the two initial matching regions; Calculating the comparison positions of each of the other initial matching regions based on the initial position of the candidate region benchmark, the unit spacing benchmark, and the number of row and column intervals between each of the other initial matching regions and the candidate region benchmark, and using the difference between the comparison position and the initial position as the initial position loss value; The sum of the initial position loss values ​​of the other initial matching regions is calculated to determine a minimum sum, and the candidate region pair corresponding to the minimum sum is determined as the reference region pair.

2. The method according to claim 1, characterized in that The initial position includes coordinate information of the pixel points included in the initial matching area; for the initial matching area with an abnormal initial position, the correction position corresponding to the initial matching area is obtained by fitting the initial positions of other initial matching areas in the same row or column as the initial matching area and with normal initial positions, including: For each initial matching region with an abnormal initial position, determining, based on the target number of rows and the target number of columns of each initial matching region, a first reference region in the same row and a second reference region in the same column as the initial matching region in each initial matching region with a normal initial position; When the number of the first reference areas and the number of the second reference areas are both greater than a preset threshold, fitting a first image fitting line and a second image fitting line based on the coordinate information of the pixel points at the first position and the second position in each of the first reference areas, and fitting a third image fitting line and a fourth image fitting line based on the coordinate information of the pixel points at the first position and the second position in each of the second reference areas; The coordinate information of the intersection of the first image fitting line and the third image fitting line, and the coordinate information of the second image fitting line and the fourth image fitting line are respectively used as the coordinate information of the pixel points at the first position and the second position in the initial matching area to obtain the corrected position corresponding to the initial matching area.

3. The method according to claim 2, characterized in that The method further comprises: When the number of the first reference area or the second reference area is not greater than a preset threshold, calculating the position to be corrected corresponding to the initial matching area based on the initial position of the reference area pair, and determining the coordinate information to be corrected of the pixel points at the first position and the second position in the initial matching area; Based on the coordinate information of the pixel points at the first position and the second position in each of the first reference areas and the coordinate information to be corrected, the first image fitting line and the second image fitting line are fitted, and based on the coordinate information of the pixel points at the first position and the second position in each of the second reference areas and the coordinate information to be corrected, the third image fitting line and the fourth image fitting line are fitted.

4. The method according to claim 1, wherein The determining of the initial matching region with abnormal initial position and the initial matching region with normal initial position based on the initial position loss values ​​of the other initial matching regions when the initial position of the reference region pair is used as a reference includes: Determine the initial position loss values ​​of each of the other initial matching areas when the initial position of the reference area pair is used as a reference, and determine the initial matching area whose initial position loss value is greater than a preset threshold as an initial matching area with an abnormal initial position, and determine the initial matching area whose initial position loss value is not greater than the preset threshold and the two initial matching areas in the reference area pair as initial matching areas with normal initial positions.

5. A method for generating billion-pixel fusion video, characterized in that: The method comprises: Obtain multiple local video streams captured by array cameras of the target scene; Using an image stitching model to stitch and fuse each group of local video images in the multiple local video streams to obtain a plurality of fused video images, and generating a fused video stream based on the fused video images; The image stitching model is constructed according to the image stitching model construction method according to any one of claims 1 to 4.

6. An image stitching model construction device, characterized in that: The device comprises: a matching module, configured to obtain a plurality of local sample images and a global sample image captured by the array camera of the target scene, and perform region matching on each of the local sample images and the global sample image to obtain an initial position of an initial matching region corresponding to each of the local sample images in the global sample image; An abnormality determination module is used to form a candidate area pair from any two initial matching areas with different target row numbers and target column numbers, and to determine the sum of the initial position loss values ​​of the other initial matching areas based on the initial position of each candidate area pair, and to determine the candidate area pair corresponding to the minimum sum as the reference area pair, and to determine the initial matching area with an abnormal initial position and the initial matching area with a normal initial position based on the initial position loss values ​​of the other initial matching areas based on the initial position of the reference area pair; wherein, the initial position of each candidate area pair is used as a reference to determine the sum of the initial position loss values ​​of the other initial matching areas, and to determine the candidate area pair corresponding to the minimum sum as the reference area. The method comprises: for each pair of candidate area pairs, taking any initial matching area in the candidate area pair as a candidate area benchmark, and calculating a unit spacing benchmark based on the spacing between the two initial matching areas in the candidate area pair and the number of row and column intervals between the two initial matching areas; calculating a comparison position of each of the other initial matching areas based on the initial position of the candidate area benchmark, the unit spacing benchmark, and the number of row and column intervals between each of the other initial matching areas and the candidate area benchmark, and taking the difference between the comparison position and the initial position as an initial position loss value; calculating a sum of the initial position loss values ​​of the other initial matching areas, determining a minimum sum value, and determining the candidate area pair corresponding to the minimum sum value as a benchmark area pair; a correction module for fitting, for an initial matching region with an abnormal initial position, a correction position corresponding to the initial matching region based on the initial positions of other initial matching regions in the same row or column as the initial matching region and with normal initial positions, determining the region at the correction position in the global sample image as a target matching region, and determining the initial matching region with normal initial positions as the target matching region; The registration module is used to perform image registration on the regional images of each target matching area and the local sample images to construct an image stitching model of the target scene.

7. A billion-pixel computational imaging system, characterized in that: The billion-pixel computational imaging system includes an array camera and a server, wherein the array camera includes multiple local cameras and a global camera, wherein: The array camera is used to capture a local sample image of the target scene through each of the local cameras, capture a global sample image through the global camera, and send the local sample image and the global sample image to the server; The server is used to perform region matching on each of the local sample images and the global sample image to obtain the initial position of the initial matching region corresponding to each of the local sample images in the global sample image; any two initial matching regions with different target row numbers and target column numbers are formed into a candidate region pair, and the sum of the initial position loss values ​​of the other initial matching regions is determined based on the initial position of each candidate region pair, and the candidate region pair corresponding to the minimum sum is determined as the reference region pair, and based on the initial position loss values ​​of the other initial matching regions when the initial position of the reference region pair is used as the reference, an initial matching region with an abnormal initial position and an initial matching region with a normal initial position are determined; for the initial matching region with an abnormal initial position, a correction position corresponding to the initial matching region is obtained by fitting based on the initial positions of other initial matching regions in the same row or column as the initial matching region and with normal initial positions, and the region of the correction position in the global sample image is determined as the target matching region, and the initial matching region with a normal initial position is determined as the target matching region; the region images of the target matching regions are registered with the local sample image to construct an image stitching model of the target scene; The step of determining the sum of the initial position loss values ​​of the other initial matching regions based on the initial position of each candidate region pair, and determining the candidate region pair corresponding to the minimum sum as the reference region pair, includes: For each candidate region pair, any initial matching region in the candidate region pair is used as a candidate region benchmark, and a unit spacing benchmark is calculated according to the spacing between the two initial matching regions in the candidate region pair and the number of row and column intervals between the two initial matching regions; Calculating the comparison positions of each of the other initial matching regions based on the initial position of the candidate region benchmark, the unit spacing benchmark, and the number of row and column intervals between each of the other initial matching regions and the candidate region benchmark, and using the difference between the comparison position and the initial position as the initial position loss value; The sum of the initial position loss values ​​of the other initial matching regions is calculated to determine a minimum sum, and the candidate region pair corresponding to the minimum sum is determined as the reference region pair.

8. The system according to claim 7, characterized in that The array camera is further configured to capture multiple local video streams of the target scene through each of the local cameras, and send each of the local video streams to the server; The server is further configured to perform image stitching and fusion on each group of local video images in each of the local video streams using the image stitching model to obtain multiple fused video images, and to send a fused video stream generated based on the fused video images to a display terminal.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 or 5 are implemented.

Images