Image splicing method, device and system for hundred million-level pixel fusion video

Abstract

The invention relates to an image splicing method and device for a hundred-million-level pixel fusion video, a hundred-million-level pixel calculation imaging system, computer equipment and a storage medium. The method comprises the following steps: acquiring multiple paths of local videos shot by an array camera, and detecting whether an overlapping region of a current frame of local video image contains a target object or not; under the condition that the overlapping area of the local video image of the current frame contains the target object, according to the object distance ratio of the object distance of the target object to the background object distance of the overlapping area, the initial homography transformation matrix of the sub-camera is updated, and a target homography transformation matrix matched with the object distance of the target object is obtained; performing image transformation on the current frame local video image based on the target homography transformation matrix to obtain a current frame transformation image corresponding to the sub-camera; and splicing and fusing the current frame transformation images corresponding to the sub-cameras to obtain a current frame fused video image. By adopting the method, the splicing accuracy and the imaging quality of the fused video image can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image stitching method and device for billion-pixel fusion video, a billion-pixel computational imaging system, a computer device, and a storage medium. Background Art

[0002] The array camera contains multiple sub-cameras. The local videos taken by each sub-camera are stitched and fused through an image stitching model to obtain ultra-high-definition fused video with a large frame and a wide field of view. It is suitable for video shooting and security monitoring of larger scenes such as airports, highways, parks, sports stadiums, borders, and sea surfaces.

[0003] The image stitching model of the array camera is usually pre-established based on the background image of the shooting area, which mainly includes image transformation, stitching, fusion and other processing. However, during the shooting process, when the target object in the shooting area has a large object distance difference with the background image of the area, the image stitching model established based on the background image has low adaptability, resulting in large image stitching errors, which seriously affects the imaging quality of the fused video. Summary of the invention

[0004] Based on this, it is necessary to provide an image stitching method, device, billion-pixel computational imaging system, computer equipment, computer-readable storage medium and computer program product that can improve the image stitching accuracy and fusion video imaging quality in response to the above-mentioned technical problems.

[0005] In a first aspect, the present application provides an image stitching method. The method comprises:

[0006] Acquire multiple local videos captured by multiple sub-cameras included in the array camera, and for each of the local videos, detect whether an overlapping area of ​​a local video image of a current frame in the local video contains a target object;

[0007] When the overlapping area of ​​the local video image of the current frame includes a target object, updating the initial homography transformation matrix of the sub-camera according to the object distance ratio of the target object to the background object distance of the overlapping area to obtain a target homography transformation matrix matching the object distance of the target object;

[0008] Performing image transformation on the local video image of the current frame based on the target homography transformation matrix to obtain a transformed image of the current frame corresponding to the sub-camera;

[0009] The current frame transformed images corresponding to the sub-cameras are spliced ​​and fused to obtain a current frame fused video image.

[0010] In one embodiment, updating the initial homography transformation matrix of the sub-camera according to the object distance ratio of the target object and the background object distance of the overlapping area to obtain a target homography transformation matrix matching the object distance of the target object includes:

[0011] Calculating the ratio of the object distance of the target object to the background object distance of the overlapping area to obtain an object distance ratio;

[0012] Determine whether the object distance ratio satisfies an update condition, and if the update condition is satisfied, construct a first update matrix according to the object distance ratio, and construct a second update matrix according to the inverse of the object distance ratio;

[0013] The first update matrix, the initial homography transformation matrix corresponding to the sub-camera, and the second update matrix are multiplied in sequence to obtain a target homography transformation matrix that matches the object distance of the target object.

[0014] In one embodiment, the first update matrix is:

[0015]

[0016] The second update matrix is:

[0017]

[0018] Wherein, m is the background object distance, n is the object distance of the target object, n / m is the object distance ratio, and m / n is the reciprocal of the object distance ratio.

[0019] In one of the embodiments, performing image transformation on the local video image of the current frame based on the target homography transformation matrix to obtain the transformed image of the current frame corresponding to the sub-camera includes:

[0020] Using the target homography transformation matrix, performing image transformation on the target area image corresponding to the target object in the local video image of the current frame to obtain a target area transformation image;

[0021] Using the initial homography transformation matrix, performing image transformation on other area images except the target area image in the local video image of the current frame to obtain other area transformation images;

[0022] The target region transformation image and the other region transformation images are spliced ​​to obtain a current frame transformation image corresponding to the sub-camera.

[0023] In one embodiment, the method further comprises:

[0024] When the overlapping area of ​​the local video image of the current frame does not contain the target object, the local video image of the current frame is transformed based on the initial homography transformation matrix of the sub-camera to obtain a transformed image of the current frame corresponding to the sub-camera.

[0025] In one of the embodiments, the detecting whether the overlapping area of ​​the local video image of the current frame in the local video contains the target object includes:

[0026] Using a target detection algorithm to detect the objects of interest contained in the local video image of the current frame, and obtaining the local coordinates of the region of interest where each object of interest is located;

[0027] Performing coordinate transformation on the local coordinates of each of the regions of interest based on the initial homography transformation matrix of the sub-camera to obtain fused coordinates of each of the regions of interest;

[0028] According to the fusion coordinates of each of the regions of interest and the fusion coordinates of the initial overlapping region, it is determined whether there is a target region in each of the regions of interest that meets the intersection condition with the initial overlapping region; the initial overlapping region is an overlapping region between the sub-camera and the adjacent sub-camera determined based on the initial image stitching model of the array camera;

[0029] In the case where there is a target area that meets the intersection condition with the initial overlapping area, determining that the overlapping area of ​​the local video image of the current frame contains a target object, and the target object is an object of interest corresponding to the target area;

[0030] In the case that there is no target area that meets the intersection condition with the initial overlapping area, it is determined that the overlapping area of ​​the local video image of the current frame does not contain the target object.

[0031] In a second aspect, the present application also provides an image stitching device. The device comprises:

[0032] A detection module is used to obtain multiple local videos taken by multiple sub-cameras included in the array camera, and for each local video, detect whether an overlapping area of ​​a local video image of a current frame in the local video contains a target object;

[0033] a matrix updating module, configured to update the initial homography transformation matrix of the sub-camera according to the object distance ratio of the target object to the background object distance of the overlapping area when the overlapping area of ​​the local video image of the current frame contains the target object, so as to obtain a target homography transformation matrix matching the object distance of the target object;

[0034] An image transformation module, used for performing image transformation on the local video image of the current frame based on the target homography transformation matrix to obtain a transformed image of the current frame corresponding to the sub-camera;

[0035] The splicing module is used to splice and fuse the current frame transformed images corresponding to the sub-cameras to obtain a current frame fused video image.

[0036] In a third aspect, the present application also provides a billion-pixel computational imaging system. The billion-pixel computational imaging system includes an array camera, a server, and a display terminal, wherein the array camera includes multiple sub-cameras, wherein:

[0037] The array camera is used to shoot a video of a target scene through multiple sub-cameras, and send the shot multiple local videos to a server;

[0038] The server is used to obtain the multiple local videos, and for each of the local videos, detect whether the overlapping area of ​​the local video image of the current frame in the local video contains the target object; when the overlapping area of ​​the local video image of the current frame contains the target object, according to the object distance ratio of the target object and the background object distance of the overlapping area, update the initial homography transformation matrix of the sub-camera to obtain a target homography transformation matrix that matches the object distance of the target object; perform image transformation on the local video image of the current frame based on the target homography transformation matrix to obtain a current frame transformation image corresponding to the sub-camera; splice and fuse the current frame transformation images corresponding to each of the sub-cameras to obtain a current frame fused video image;

[0039] The server is further configured to generate a fused video stream based on the current frame fused video image, and send the fused video stream to the display terminal.

[0040] In a fourth aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in the first aspect are implemented.

[0041] In a fifth aspect, the present application further provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0042] In a sixth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0043] The above-mentioned image stitching method, device, billion-pixel computational imaging system, computer equipment, storage medium and computer program product, when detecting that the overlapping area of ​​the local video images taken by the sub-cameras contains the target object of interest, updates the initial homography transformation matrix of the sub-camera according to the object distance ratio of the target object and the background object distance of the overlapping area, thereby obtaining a target homography matrix that matches the object distance of the target object, and then performing image transformation based on the matched target homography matrix, which can improve the image stitching accuracy and the imaging quality of the fused video. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of a billion-pixel computational imaging system in an example;

[0045] Figure 2 A schematic diagram of a flow chart of an image stitching method for billion-pixel fusion video in an embodiment;

[0046] Figure 3 is a schematic diagram of a flow chart of an image transformation step in an embodiment;

[0047] Figure 4 It is a structural block diagram of an image stitching device for billion-pixel fusion video in one embodiment;

[0048] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] First of all, before specifically introducing the technical solution of the embodiment of the present application, the technical background or technical evolution context on which the embodiment of the present application is based is introduced. In order to achieve ultra-high-definition video surveillance with large scenes and wide fields of view, a billion-pixel computational imaging system based on an array camera has emerged. The array camera includes a plurality of sub-cameras with different shooting angles, which respectively shoot local high-definition videos (such as 4K resolution) of the target scene. Usually, there is an overlapping area in the field of view of adjacent sub-cameras. By splicing and fusing the local videos shot by each sub-camera, a large or wide field of view, ultra-high resolution fused video (up to billions or billions of pixels) can be obtained for display or playback on a display terminal. The splicing and fusion process can adopt a pre-established image splicing model, which mainly includes image transformation, splicing, fusion and other processing. The image transformation processing usually uses the homography transformation matrix corresponding to each sub-camera to transform the local video image, so that the shooting angle of view (observation angle) of each transformed image is the same, so as to facilitate subsequent splicing, fusion and other processing.

[0051] The image stitching model is usually pre-established based on the background image (the image of the fixed scene, relative to the moving foreground). When a moving target appears in the shooting area, if the object distance of the moving target relative to the camera is significantly different from the object distance of the background scene in the area, the image stitching model established based on the background image will no longer be applicable to stitching the video frame images containing such moving target objects. For example, in the scene of shooting the airport runway with an array camera, the background image shot by some sub-cameras is a mountain with a relatively long object distance. When the sub-camera shoots an airplane that has just taken off or is about to land over the runway, the object distance of the airplane is much smaller than the object distance of the mountain. For another example, if the background image used to establish the image stitching model contains objects such as branches, if the object is removed when the surveillance video is actually shot, there may be a situation where the object distance between the shooting target in the area and the background image is significantly different. In particular, when the target object is located in the overlapping area shot by adjacent sub-cameras, the image stitching error of the target object will be relatively large due to the incompatibility of the image stitching model, which will seriously affect the imaging quality of the fused video.

[0052] Based on this background, the applicant, through long-term research and development and experimental verification, proposed the video stitching method of the present application, which can adjust the homography transformation matrix in the initial image stitching model in real time according to the object distance of the target object in the overlapping area, thereby improving the imaging quality of the fused video. In addition, it should be noted that the applicant has made a lot of creative efforts in discovering the technical problem of the present application and the technical solutions introduced in the following embodiments.

[0053] The image stitching method of the billion-pixel fusion video provided in the embodiment of the present application can be applied to Figure 1The billion-pixel computational imaging system shown in FIG. 1 . The array camera 102 and the display terminal 106 communicate with the server 104 through a network respectively. The display terminal 106 can be, but is not limited to, various electronic devices with display units such as personal computers, laptops, smart phones, tablet computers, smart displays, Internet of Things devices, portable wearable devices, etc. The server 104 can be implemented as an independent server or a server cluster consisting of multiple servers.

[0054] In one embodiment, Figure 2 As shown in the figure, a method for image stitching of billion-pixel fusion videos is provided, which can be applied to Figure 1 In this embodiment, the method includes the following steps:

[0055] Step 201 , obtaining multiple local videos shot by multiple sub-cameras included in the array camera, and for each local video, detecting whether an overlapping area of ​​a local video image of a current frame in the local video contains a target object.

[0056] In implementation, the array camera can send the captured local video stream to the server, so that the server can splice and fuse the local video images captured by each sub-camera at the same time, obtain the fused video image at that moment, and generate a fused video stream based on the fused video images at multiple moments for terminal playback. The target object can be a preset type of target, for example, an aircraft, a drone, or other target types of interest. For each frame of local video image, a target detection algorithm (such as yolo, rcnn, fast-rcnn, etc.) can be used to detect whether it contains a target object, and determine whether the detected target object is located in an overlapping area. The overlapping area refers to an area that can be captured by at least two sub-cameras. For an array camera with multiple rows and columns, the current sub-camera usually has an overlapping area with an adjacent sub-camera (4 neighborhoods or 8 neighborhoods), that is, there can be multiple overlapping areas, such as the image captured by the second row and second column camera A22, can have an overlapping area with the first row and second column camera A12, the second row and first column camera A21, the second row and third column camera A23, and the third row and second column camera A32 (if any). Based on the pre-established image stitching model, the initial overlapping area of ​​each sub-camera (i.e., the overlapping area of ​​each sub-camera when shooting the background image) can be determined. If the detection frame of the target of interest in the local video image of the current frame intersects with the initial overlapping area, it can be determined that the overlapping area of ​​the local video image of the current frame contains the target object.

[0057] Step 202, when the overlapping area of ​​the local video image of the current frame contains the target object, the initial homography transformation matrix of the sub-camera is updated according to the object distance ratio of the target object and the background object distance of the overlapping area to obtain a target homography transformation matrix matching the object distance of the target object.

[0058] In implementation, for a local video image in which an overlapping area is detected to contain a target object, the server can update the initial homography transformation matrix corresponding to the local video image. The object distance of the target object (which can be recorded as n) can be obtained by image ranging algorithm, radar ranging, etc., and the background object distance (which can be recorded as m) can be calibrated in advance. For example, after the initial image stitching model is established, the object distance of the background scene in each overlapping area can be calibrated and stored, so that the server can calculate the object distance ratio (which can be recorded as n / m) based on the object distance n of the target object and the background object distance m of the area. Then, the server can update the initial homography transformation matrix (which can be recorded as H) based on the object distance ratio n / m to obtain the target homography transformation matrix (which can be recorded as H) that matches the target object. n ). The initial homography transformation matrix refers to the image transformation matrix in the initial image stitching model established based on the background image. The update strategy of the initial homography transformation matrix H is related to the size of the overlapping area of ​​adjacent sub-cameras. When the object distance of the imaging screen changes from far to near, the overlapping area will change from large to small. Specifically, for two adjacent sub-cameras on the left and right, if the angle between the right boundary of the left camera field of view and the left boundary of the right camera field of view is α, the calculation formula for the size of the overlapping area of ​​the imaging screen (which can be recorded as w) is:

[0059]

[0060] Where l is the object distance of the camera imaging screen. From the above formula, it can be seen that the size of the overlapping area is proportional to the object distance of the camera imaging screen. Therefore, the update strategy of the homography transformation matrix can be derived based on this rule.

[0061] In an example, the initial homography transformation matrix H may be updated using the following formula:

[0062]

[0063] It can be understood that if the difference between the target object's object distance n and the background object distance m is small, the initial homography transformation matrix can be directly used as the target homography transformation matrix, that is, no updating is required.

[0064] In one implementation, the server may calculate the ratio of the object distance n of the target object and the background object distance m of the overlapping area to obtain the object distance ratio n / m, and then determine whether the object distance ratio n / m satisfies the update condition. The object distance ratio that satisfies the update condition refers to the case where the object distance n of the target object and the scene distance m are significantly different and it is necessary to update the homography transformation matrix. The update condition may be that the object distance ratio n / m is outside a preset interval, such as the preset interval may be 0.99 to 1.01. When the object distance ratio is not within the interval, the update condition is satisfied. When the object distance ratio satisfies the update condition, a first update matrix (the 3*3 matrix on the left side of H in the above formula) is constructed according to the object distance ratio n / m, and a second update matrix (the 3*3 matrix on the right side of H in the above formula) is constructed according to the inverse of the object distance ratio m / n. Then, according to the above formula, the first update matrix, the initial homography transformation matrix H, and the second update matrix are multiplied in sequence (cross multiplication, corresponding to the above formula) to obtain a target homography transformation matrix H that matches the object distance n of the target object. n When the object distance ratio does not meet the update condition, the initial homography transformation matrix is ​​directly used as the target homography transformation matrix to transform the local video image of the current frame, thereby realizing image splicing and fusion to obtain a fused video image.

[0065] In one example, the process of establishing the initial homography transformation matrix corresponding to each sub-camera includes: shooting a panoramic video of the background scene of the target area through the panoramic camera included in the array camera, the focal length of the panoramic camera is smaller than the focal length of the sub-camera, and the panoramic video image can completely cover the shooting field of view of each local video image. Then, the local video images of the background scene of the target area shot by each sub-camera can be image-aligned with the corresponding areas in the panoramic video image, and the initial homography transformation matrix of each local video image relative to the panoramic video image is established, that is, the geometric correspondence between the local video image and the panoramic video image is established, so that each local video image is aligned with the corresponding area in the panoramic video image. As a result, the images shot by each sub-camera are transformed through their corresponding homography transformation matrices, and are converted into images consistent with the perspective of the panoramic video image, so as to facilitate splicing and fusion.

[0066] Step 203 , performing image transformation on the local video image of the current frame based on the target homography transformation matrix to obtain a transformed image of the current frame corresponding to the sub-camera.

[0067] In implementation, the updated target homography transformation matrix can be used to transform the entire local video image, or the target homography transformation matrix can be used only for the area where the target object is located, and the initial homography transformation matrix can be used for other areas.

[0068] It can be understood that the local video images of the current frame taken by each sub-camera need to be transformed so that each local video image is transformed to the same coordinate system. For the local video images whose overlapping areas contain the target object, step 202 and step 203 are executed. For the local video images whose overlapping areas do not contain the target object, the initial homography transformation matrix in the initial image stitching model can be directly used to transform the local video images.

[0069] Step 204 , stitching and fusing the current frame transformed images corresponding to the sub-cameras to obtain a current frame fused video image.

[0070] In implementation, after obtaining the current frame transformation images corresponding to each sub-camera, the current frame transformation images can be spliced ​​and fused, such as splicing, fusion, cropping, color correction, etc., to obtain the current frame fused video image. A fused video stream can be generated based on the fused video images at multiple moments for terminal playback.

[0071] The above-mentioned image stitching method, when detecting that the overlapping area of ​​the local video images taken by the sub-cameras contains the target object of interest, updates the initial homography transformation matrix of the sub-camera according to the object distance ratio of the target object and the background object distance of the overlapping area, thereby obtaining a target homography matrix that matches the object distance of the target object, and then performing image transformation based on the matched target homography matrix, which can improve the image stitching accuracy and the imaging quality of the fused video.

[0072] In one embodiment, Figure 3 As shown, the image transformation process in the above step 203 specifically includes the following steps:

[0073] Step 301 , using a target homography transformation matrix, performs image transformation on a target region image corresponding to a target object in a local video image of a current frame to obtain a target region transformed image.

[0074] Step 302: using the initial homography transformation matrix, perform image transformation on other area images except the target area image in the local video image of the current frame to obtain other area transformation images.

[0075] Step 303: splice the target region transformation image and other region transformation images to obtain the current frame transformation image corresponding to the sub-camera.

[0076] In implementation, since the local video image may contain multiple overlapping areas, if the target object is detected in each overlapping area, the target homography transformation matrix matching the object distance of each target object can be calculated, so that the image of the target area where the target object is located (such as the minimum circumscribed rectangular area containing the target object) can be transformed using the matching target homography transformation matrix. For other areas except the target area where the target object is located, the initial homography transformation matrix can be used for image transformation. Thus, the current frame transformation image corresponding to each sub-camera can be obtained, and the splicing and fusion processing can be performed based on the current frame change image, so as to obtain a fused video with higher splicing accuracy and better imaging quality.

[0077] In one embodiment, the process of detecting whether the overlapping area contains the target object in the above step 201 specifically includes the following steps: using a target detection algorithm to detect the objects of interest contained in the local video image of the current frame, and obtaining the local coordinates of the regions of interest where each object of interest is located; performing coordinate transformation on the local coordinates of each region of interest based on the initial homography transformation matrix of the sub-camera to obtain the fusion coordinates of each region of interest; judging whether there is a target area in each region of interest that meets the intersection condition with the initial overlapping area according to the fusion coordinates of each region of interest and the fusion coordinates of the initial overlapping area; the initial overlapping area is the overlapping area between the sub-camera and the adjacent sub-camera determined based on the initial image stitching model of the array camera; in the case that there is a target area that meets the intersection condition with the initial overlapping area, determining that the overlapping area of ​​the local video image of the current frame contains the target object, and the target object is the object of interest corresponding to the target area; in the case that there is no target area that meets the intersection condition with the initial overlapping area, determining that the overlapping area of ​​the local video image of the current frame does not contain the target object.

[0078] In implementation, the detection module of the server can use the target detection algorithm to detect the objects of interest contained in each local video image, and obtain the local coordinates of the area where each object of interest is located (which can be the minimum circumscribed rectangular area or detection frame containing the object of interest) in the local video image. Then, the server can perform coordinate transformation on the local coordinates of each region of interest based on the initial homography transformation matrix of the sub-camera to obtain the fused coordinates of each region of interest in the fused image, and then determine whether each region of interest meets the intersection condition with the initial overlapping area (such as the area intersection ratio of the two areas is greater than a preset threshold) according to the fused coordinates of each region of interest and the fused coordinates of the initial overlapping area. If there is a region of interest that meets the intersection condition, the object of interest in the region of interest is determined as the target object, and it is determined that there is a target object in the overlapping area.

[0079] Since the size of the overlapping area of ​​the sub-cameras is related to the object distance of the imaging screen, that is, the position of the overlapping area is not fixed, the area where each target of interest is located can be transformed based on the initial homography transformation matrix. If the transformed area meets the initial overlapping area in the initial image stitching model, it can be determined that there is a target object in the overlapping area. Therefore, it can be quickly determined whether the homography transformation matrix needs to be updated according to the object distance ratio, thereby taking into account both image stitching accuracy and stitching efficiency.

[0080] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0081] Based on the same inventive concept, the embodiment of the present application also provides an image stitching device for implementing the above-mentioned image stitching method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more image stitching device embodiments provided below can refer to the limitations on the image stitching method above, and will not be repeated here.

[0082] In one embodiment, Figure 4 As shown, an image stitching device 400 is provided, comprising: a detection module 401, a matrix update module 402, a first image transformation module 403 and a stitching module 404, wherein:

[0083] The detection module 401 is used to obtain multiple local videos shot by multiple sub-cameras included in the array camera, and for each of the local videos, detect whether the overlapping area of ​​the local video image of the current frame in the local video contains the target object.

[0084] The matrix updating module 402 is used to update the initial homography transformation matrix of the sub-camera according to the object distance ratio of the target object and the background object distance of the overlapping area when the overlapping area of ​​the local video image of the current frame contains the target object, so as to obtain a target homography transformation matrix matching the object distance of the target object.

[0085] The first image transformation module 403 is used to perform image transformation on the local video image of the current frame based on the target homography transformation matrix to obtain a transformed image of the current frame corresponding to the sub-camera.

[0086] The stitching module 404 is used to stitch and fuse the current frame transformed images corresponding to the sub-cameras to obtain a current frame fused video image.

[0087] In one embodiment, the matrix update module 402 is specifically used to: calculate the ratio of the object distance of the target object and the background object distance of the overlapping area to obtain the object distance ratio; determine whether the object distance ratio satisfies an update condition, and if the update condition is satisfied, construct a first update matrix according to the object distance ratio, and construct a second update matrix according to the inverse of the object distance ratio; multiply the first update matrix, the initial homography transformation matrix corresponding to the sub-camera, and the second update matrix in sequence to obtain a target homography transformation matrix that matches the object distance of the target object.

[0088] In one embodiment, the first update matrix is:

[0089]

[0090] The second update matrix is:

[0091]

[0092] Wherein, m is the background object distance, n is the object distance of the target object, n / m is the object distance ratio, and m / n is the reciprocal of the object distance ratio.

[0093] In one of the embodiments, the first image transformation module 403 is specifically used to: use the target homography transformation matrix to perform image transformation on the target area image corresponding to the target object in the local video image of the current frame to obtain a target area transformation image; use the initial homography transformation matrix to perform image transformation on other area images except the target area image in the local video image of the current frame to obtain other area transformation images; splice the target area transformation image and the other area transformation images to obtain the current frame transformation image corresponding to the sub-camera.

[0094] In one of the embodiments, the device further includes a second image transformation module, which is used to perform image transformation on the local video image of the current frame based on the initial homography transformation matrix of the sub-camera when the overlapping area of ​​the local video image of the current frame does not contain the target object, so as to obtain the current frame transformed image corresponding to the sub-camera.

[0095] In one embodiment, the detection module 401 is specifically used to: use a target detection algorithm to detect the objects of interest contained in the local video image of the current frame, and obtain the local coordinates of the region of interest where each object of interest is located; perform coordinate transformation on the local coordinates of each region of interest based on the initial homography transformation matrix of the sub-camera to obtain the fusion coordinates of each region of interest; according to the fusion coordinates of each region of interest and the fusion coordinates of the initial overlapping region, determine whether there is a target region in each region of interest that meets the intersection condition with the initial overlapping region; the initial overlapping region is the overlapping region of the sub-camera and the adjacent sub-camera determined based on the initial image stitching model of the array camera; in the case that there is a target region that meets the intersection condition with the initial overlapping region, determine that the overlapping region of the local video image of the current frame contains the target object, and the target object is the object of interest corresponding to the target region; in the case that there is no target region that meets the intersection condition with the initial overlapping region, determine that the overlapping region of the local video image of the current frame does not contain the target object.

[0096] Each module in the above-mentioned image stitching device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0097] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data required or generated for executing the above-mentioned image stitching method. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image stitching method is implemented.

[0098] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0099] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0100] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0101] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0102] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0103] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0104] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for image stitching of billion-pixel fusion videos, characterized in that: The method comprises: Acquire multiple local videos captured by multiple sub-cameras included in the array camera, and for each of the local videos, detect whether an overlapping area of ​​a local video image of a current frame in the local video contains a target object; When the overlapping area of ​​the local video image of the current frame includes a target object, updating the initial homography transformation matrix of the sub-camera according to the object distance ratio of the target object to the background object distance of the overlapping area to obtain a target homography transformation matrix matching the object distance of the target object; Performing image transformation on the local video image of the current frame based on the target homography transformation matrix to obtain a transformed image of the current frame corresponding to the sub-camera; The current frame transformed images corresponding to the sub-cameras are spliced ​​and fused to obtain a current frame fused video image.

2. The method according to claim 1, characterized in that: The updating of the initial homography transformation matrix of the sub-camera according to the object distance ratio of the target object and the background object distance of the overlapping area to obtain a target homography transformation matrix matching the object distance of the target object includes: Calculating the ratio of the object distance of the target object to the background object distance of the overlapping area to obtain an object distance ratio; Determine whether the object distance ratio satisfies an update condition, and if the update condition is satisfied, construct a first update matrix according to the object distance ratio, and construct a second update matrix according to the inverse of the object distance ratio; The first update matrix, the initial homography transformation matrix corresponding to the sub-camera, and the second update matrix are multiplied in sequence to obtain a target homography transformation matrix that matches the object distance of the target object.

3. The method according to claim 2, characterized in that The first update matrix is: The second update matrix is: Wherein, m is the background object distance, n is the object distance of the target object, n / m is the object distance ratio, and m / n is the reciprocal of the object distance ratio.

4. The method according to claim 1, characterized in that The performing image transformation on the local video image of the current frame based on the target homography transformation matrix to obtain the transformed image of the current frame corresponding to the sub-camera includes: Using the target homography transformation matrix, performing image transformation on the target area image corresponding to the target object in the local video image of the current frame to obtain a target area transformation image; Using the initial homography transformation matrix, performing image transformation on other area images except the target area image in the local video image of the current frame to obtain other area transformation images; The target region transformation image and the other region transformation images are spliced ​​to obtain a current frame transformation image corresponding to the sub-camera.

5. The method according to claim 1, characterized in that The method further comprises: When the overlapping area of ​​the local video image of the current frame does not contain the target object, the local video image of the current frame is transformed based on the initial homography transformation matrix of the sub-camera to obtain a transformed image of the current frame corresponding to the sub-camera.

6. The method according to claim 1, characterized in that The detecting whether the overlapping area of ​​the local video image of the current frame in the local video contains the target object includes: Using a target detection algorithm to detect the objects of interest contained in the local video image of the current frame, and obtaining the local coordinates of the region of interest where each object of interest is located; Performing coordinate transformation on the local coordinates of each of the regions of interest based on the initial homography transformation matrix of the sub-camera to obtain fused coordinates of each of the regions of interest; According to the fusion coordinates of each of the regions of interest and the fusion coordinates of the initial overlapping region, it is determined whether there is a target region in each of the regions of interest that meets the intersection condition with the initial overlapping region; the initial overlapping region is an overlapping region between the sub-camera and the adjacent sub-camera determined based on the initial image stitching model of the array camera; In the case where there is a target area that meets the intersection condition with the initial overlapping area, determining that the overlapping area of ​​the local video image of the current frame contains a target object, and the target object is an object of interest corresponding to the target area; In the case that there is no target area that meets the intersection condition with the initial overlapping area, it is determined that the overlapping area of ​​the local video image of the current frame does not contain the target object.

7. An image stitching device for billion-pixel fusion video, characterized in that: The device comprises: A detection module is used to obtain multiple local videos taken by multiple sub-cameras included in the array camera, and for each local video, detect whether an overlapping area of ​​a local video image of a current frame in the local video contains a target object; a matrix updating module, configured to update the initial homography transformation matrix of the sub-camera according to the object distance ratio of the target object to the background object distance of the overlapping area when the overlapping area of ​​the local video image of the current frame contains the target object, so as to obtain a target homography transformation matrix matching the object distance of the target object; A first image transformation module, configured to perform image transformation on the local video image of the current frame based on the target homography transformation matrix to obtain a transformed image of the current frame corresponding to the sub-camera; The splicing module is used to splice and fuse the current frame transformed images corresponding to the sub-cameras to obtain a current frame fused video image.

8. A billion-pixel computational imaging system, characterized in that: The billion-pixel computational imaging system comprises an array camera, a server and a display terminal, wherein the array camera comprises a plurality of sub-cameras, wherein: The array camera is used to shoot a video of a target scene through multiple sub-cameras, and send the shot multiple local videos to a server; The server is used to obtain the multiple local videos, and for each of the local videos, detect whether the overlapping area of ​​the local video image of the current frame in the local video contains the target object; when the overlapping area of ​​the local video image of the current frame contains the target object, according to the object distance ratio of the target object and the background object distance of the overlapping area, update the initial homography transformation matrix of the sub-camera to obtain a target homography transformation matrix that matches the object distance of the target object; perform image transformation on the local video image of the current frame based on the target homography transformation matrix to obtain a current frame transformation image corresponding to the sub-camera; splice and fuse the current frame transformation images corresponding to each of the sub-cameras to obtain a current frame fused video image; The server is further configured to generate a fused video stream based on the current frame fused video image, and send the fused video stream to the display terminal.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images