Image processing method, device and system

Abstract

The invention discloses an image processing method, device and system, and relates to the technical field of image processing. The method comprises: obtaining a to-be-stitched image comprising a first image frame in a first video and a second image frame in a second video, the to-be-stitched image comprising a first sub-image and a second sub-image, the first sub-image being an image of an overlapping region in the first image frame, and the second sub-image being an image of an overlapping region in the second image frame; when the jitter amplitude of the to-be-spliced image is smaller than or equal to a preset threshold value, a first splicing seam position is determined, the first splicing seam position is located on the same side of a target object in a reference sub-image and a target object in a historical sub-image, the reference sub-image is a first sub-image or a second sub-image, and the historical sub-image is an image of an overlapping area in a historical spliced image. And splicing the first image frame and the second image frame according to the first splicing seam position to obtain a target spliced image. Through the method, the stability of the spliced image in the time domain can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device and system. Background Art

[0002] The camera field of view of existing video conferencing terminals is relatively small, and the horizontal field of view of most products is within 90°. For large and medium-sized conference rooms or when participants are in the blind spot of the camera's field of view, the camera field of view of the video conferencing terminal cannot fully cover the participants and personnel. To solve this problem, images taken by multiple cameras with different fields of view can be stitched together. However, due to the binocular parallax between different cameras, as well as the influence of factors such as stitching algorithm errors and limited computing power, the stitched images taken by multiple cameras with different fields of view usually have obvious stitching seams.

[0003] To prevent a face (or body, object, etc.) located near a stitching seam of a stitched image from being displayed incompletely due to the stitching seam, the stitching seam position in the stitching image may be adjusted in real time, which may result in poor stability of the stitching image in the time domain. Summary of the invention

[0004] The present application provides an image processing method, device and system, which can improve the stability of spliced ​​video in the time domain.

[0005] To achieve the above objectives, this application provides the following technical solutions:

[0006] In a first aspect, the present application provides an image processing method, the method comprising: obtaining an image to be spliced ​​including a first image frame in a first video and a second image frame in a second video. There is an overlapping area between the shooting field of view of the first video and the shooting field of view of the second video, the image to be spliced ​​includes a first sub-image and a second sub-image, the first sub-image is an image of the overlapping area in the first image frame, the second sub-image is an image of the overlapping area in the second image frame, and the first sub-image and the second sub-image include a target object. When the jitter amplitude of the image to be spliced ​​is less than or equal to a preset threshold, a first stitching position is determined, the first stitching position is located on the same side of the target object in the reference sub-image and the target object in the historical sub-image, the reference sub-image is the first sub-image or the second sub-image, the historical sub-image is an image of the overlapping area in the historical stitching image, and the historical stitching image is obtained by stitching the historical frame in the first video before the first image frame and the historical frame in the second video before the second image frame. The first image frame and the second image frame are spliced ​​according to the first stitching position to obtain a target stitching image.

[0007] The method provided by the present application can achieve that when the jitter amplitude of the first image frame and the second image frame is less than or equal to a preset threshold value, the target spliced ​​image obtained after the first image frame and the second image frame are spliced ​​will not have screen jitter in the time domain due to the change of the splicing position. Therefore, the method provided by the embodiment of the present application can improve the stability of the target spliced ​​video obtained after the first video and the second video are spliced ​​in the time domain.

[0008] In a possible design, the first sub-image and the second sub-image also include a background, and the background refers to the area in the image other than the target object. The method also includes: when the jitter amplitude of the image to be spliced ​​is less than or equal to a preset threshold, determining a second stitching position, the second stitching position is the same position in the reference sub-image as the stitching position of the target area in the historical sub-image, and the target area is the same area in the historical sub-image as the background position in the reference sub-image. The stitching of the first image frame and the second image frame according to the first stitching position includes: stitching the first image frame and the second image frame according to the first stitching position and the second stitching position.

[0009] Through this possible design, for the background in the reference sub-image (such as the first sub-image or the second sub-image), the seam at the same position as the background position in the historical sub-image can be used. In other words, the seam position in the background of the reference sub-image is the position of the seam at the same position as the background position in the historical sub-image. In this way, in a scene where the target object in the target scene has a small range of activity and the background in the target scene is static, the seam position of the background area in the target spliced ​​video is basically unchanged in the time domain, which can ensure the stability of the target spliced ​​video in the time domain.

[0010] In another possible design, the method further includes: displaying the target spliced ​​image.

[0011] In another possible design, the historical frames located before the first image frame include a preset number of image frames before the first image frame in the first video, and the historical frames located before the second image frame include a preset number of image frames before the second image frame in the second video, and the preset number is greater than or equal to 1.

[0012] In this possible design, when the preset number is greater than 1, the image processing device may set the seam that bypasses the target object outside the motion range of the target object in the continuous multiple image frames. In this way, when the seam that bypasses the target object is subsequently determined, if the subsequent motion range of the target object does not exceed the aforementioned motion range, the position of the seam that bypasses the target object can be kept unchanged, which can improve the stability of the spliced ​​video screen.

[0013] In another possible design, the method further includes: determining the jitter amplitude of the first image frame and the jitter amplitude of the second image frame. When the jitter amplitude of the first image frame is less than or equal to the first threshold, or the jitter amplitude of the second image frame is less than or equal to the first threshold, or the sum of the jitter amplitude of the first image frame and the jitter amplitude of the second image frame is less than or equal to the second threshold, it is determined that the jitter amplitude of the image to be spliced ​​is less than or equal to the preset threshold. Through this possible design, the jitter amplitude of the first image frame and the second image frame can be determined.

[0014] In another possible design, the method further includes: determining the target object and the background in the first sub-image and the second sub-image.

[0015] In another possible design, the method further includes: determining a geometric transformation relationship between the first image frame and the second image frame. Reprojecting the first image frame and the second image frame according to the geometric transformation relationship. The stitching of the first image frame and the second image frame according to the first seam position includes: stitching the reprojected first image frame and the reprojected second image frame according to the first seam position.

[0016] In another possible design, the geometric transformation relationship is represented by a target homography matrix, and the method further includes: performing temporal smoothing on the target homography matrix according to the first weight and the homography matrix of the historical frame, wherein the historical homography matrix is ​​a homography matrix for reprojecting the historical frame when the historical spliced ​​image is obtained. The reprojecting of the first image frame and the second image frame according to the geometric transformation relationship includes: reprojecting the first image frame and the second image frame based on the target homography matrix that has been smoothed in the temporal domain.

[0017] Through this possible design method, it is possible to implement, in the image stitching process, the target homography matrix used to reproject the current image to be stitched, through the historical homography matrix corresponding to the historical stitching image, to constrain the target homography matrix, and to achieve time-domain smoothing of the target homography matrix. This can improve the stability of the transition from the historical stitching image to the target stitching image obtained by image stitching based on the reprojection of the target homography matrix, thereby improving the stability of the target stitching video.

[0018] In another possible design, the geometric transformation relationship is represented by a target homography matrix, and the target homography matrix includes a global homography matrix and multiple local homography matrices. Wherein, the global homography matrix is ​​calculated based on the entire image of the first image frame and the entire image of the second image frame, and multiple local homography matrices correspond to multiple sub-regions divided by the overlapping area, and a local homography matrix is ​​calculated based on the image of a sub-region in the overlapping area. The above method also includes: according to the second weight and the global homography matrix, the local homography matrix is ​​spatially smoothed, or, according to the difference between adjacent homography matrices in multiple local homography matrices, the local homography matrix is ​​spatially smoothed, wherein the sub-regions corresponding to adjacent homography matrices are adjacent sub-regions in the overlapping area. The above reprojection of the first image frame and the second image frame according to the geometric transformation relationship includes: reprojecting the images of the first image frame and the second image frame except the overlapping area based on the global homography matrix, and reprojecting the images of multiple sub-regions based on multiple local homography matrices that have been spatially smoothed.

[0019] Through this possible design, it is possible to achieve, in the image stitching process, targeted reprojection of images in different areas of the first image frame and the second image frame using the corresponding homography matrix that has been smoothed in the spatial domain, thereby improving the reprojection accuracy of the first image frame and the second image frame, and further improving the image quality and stitching quality of the target stitched image.

[0020] In another possible design, the first video and the second video are videos shot by adjacent lenses in a panoramic camera. Alternatively, the first video and the second video are videos shot by two cameras with overlapping shooting fields in the target scene.

[0021] In another possible design, the jitter of the above-mentioned images to be stitched includes at least one of the following: image jitter caused by the movement of the camera / lens used to shoot the images to be stitched, image jitter caused by the movement / movement of the target object when shooting the images to be stitched, and image jitter caused by changes in ambient lighting where the camera / lens is located when shooting the images to be stitched.

[0022] In a second aspect, the present application provides an image processing device.

[0023] In a possible design, the image processing device is used to execute any one of the methods provided in the first aspect above. The present application may divide the image processing device into functional modules according to any one of the methods provided in the first aspect above. For example, each functional module may be divided according to each function, or two or more functions may be integrated into one processing module. Exemplarily, the present application may divide the training device into an acquisition unit, a determination unit, and a splicing unit, etc. according to the function. The description of the possible technical solutions and beneficial effects executed by the above-mentioned divided functional modules can refer to the technical solutions provided by the first aspect or its corresponding possible design, and will not be repeated here.

[0024] In another possible design, the image processing device includes: one or more processors and a memory, and the one or more processors are configured to call program instructions stored in the memory so that the image processing device executes any method provided by the first aspect and any possible design thereof.

[0025] In a third aspect, the present application provides an image processing system, which includes an image processing device, a first image acquisition device, and a second image acquisition device. The first image acquisition device is used to acquire a first video, and the second image acquisition device is used to acquire a second video, and there is an overlapping area between the shooting field of view of the first video and the shooting field of view of the second video. The image processing device is used to execute any method provided in the first aspect and any possible design thereof to realize the splicing of image frames in the first video and the second video.

[0026] In a fourth aspect, the present application provides a computer-readable storage medium, which includes program instructions. When the program instructions are executed on a computer or a processor, the computer or the processor executes any method provided by any possible implementation of the first aspect.

[0027] In a fifth aspect, the present application provides a computer program product, which, when executed on an image processing device, enables any method provided by any possible implementation in the first aspect to be executed.

[0028] It can be understood that any of the above-mentioned devices, systems, computer storage media or computer program products can be applied to the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods and will not be repeated here.

[0029] In this application, the names of the above-mentioned image processing devices and image processing systems do not limit the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear with other names. As long as the functions of each device or functional module are similar to those of this application, they fall within the scope of the claims of this application and their equivalent technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of a spliced ​​video causing image jitter in the time domain due to a change in the position of a splicing seam provided in an embodiment of the present application;

[0031] Figure 2 A schematic diagram of the architecture of an image processing system provided in an embodiment of the present application;

[0032] Figure 3 A schematic diagram of the architecture of another image processing system provided in an embodiment of the present application;

[0033] Figure 4 A schematic diagram of deploying multiple image acquisition devices provided in an embodiment of the present application;

[0034] Figure 5 A schematic diagram of another arrangement of multiple image acquisition devices provided in an embodiment of the present application;

[0035] Figure 6 A schematic diagram of the hardware structure of a computing device provided in an embodiment of the present application;

[0036] Figure 7 A flowchart of an image processing method provided in an embodiment of the present application;

[0037] Figure 8 A schematic diagram of a first image frame, a second image frame, and an overlapping area image provided in an embodiment of the present application;

[0038] Fig. 9 A schematic diagram of a seam position change in the time domain provided in an embodiment of the present application;

[0039] Fig.10 A schematic diagram of dividing a reference sub-image into multiple sub-areas provided in an embodiment of the present application;

[0040] Fig.11 A schematic diagram of a first seam and a second seam provided in an embodiment of the present application;

[0041] Fig.12 A flowchart of another image processing method provided in an embodiment of the present application;

[0042] Fig.13 A flowchart of another image processing method provided in an embodiment of the present application;

[0043] Fig.14 A flowchart of another image processing method provided in an embodiment of the present application;

[0044] Fig.15 A flowchart of another image processing method provided in an embodiment of the present application;

[0045] Fig.16 A schematic diagram of dividing an overlapping area of ​​a first image frame and a second image frame into a plurality of sub-areas provided in an embodiment of the present application;

[0046] Fig.17 Another schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application;

[0047] Fig.18 A schematic diagram of another process flow of the image processing method provided in the embodiment of the present application;

[0048] Fig.19 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to more clearly understand the embodiments of the present application, some terms or technologies involved in the embodiments of the present application are explained below:

[0050] 1) Object detection, face detection

[0051] Object detection refers to finding the object of interest in an image and determining the category and location of the object in the image. The object can be a person, an animal, an object, etc., but is not limited to these.

[0052] Face detection refers to finding the location of faces in an image.

[0053] It should be understood that the position of the target or face in the image can be represented by a position frame including the target or face in the image, or by a position frame having the outline of the target or face, without limitation.

[0054] 2) Screen shaking

[0055] When the content, color or brightness of a certain image frame in a video changes relative to the image frame before the image frame, it can be said that the image of the video jitters.

[0056] There are many reasons that cause the video image to shake, such as the displacement and shaking of the camera / lens used to shoot the video, or changes in the ambient lighting during the video shooting, which can cause the video image to shake.

[0057] Currently, the shaking of video images can be reduced or even prevented by adopting an optical image stabilization (OIS) system or an electronic image stabilization (EIS) system.

[0058] Among them, OIS compensates for the displacement and shaking of the camera / lens during shooting by moving the position of the lens group or photosensitive chip in the camera / lens, thereby achieving the function of shock absorption and anti-shake, which can make the captured picture clear and stable.

[0059] EIS can detect the amplitude of camera / lens shake through acceleration sensor and / or gyroscope module, or estimate the camera motion through software algorithm, so as to dynamically adjust ISO sensitivity, shutter and imaging algorithm according to the amplitude of camera / lens shake detected or estimated camera motion, so as to achieve the purpose of blur correction of the shaken picture. Here, ISO refers to the International Standards Organization.

[0060] Among them, OIS anti-shake effect is better than EIS, but it is costly and power-consuming.

[0061] 3) Camera calibration

[0062] The principle of camera imaging is to map the three-dimensional physical world (i.e., the world coordinate system) into a two-dimensional image (i.e., the pixel coordinate system) through a complex imaging process, and camera calibration is to solve the conversion relationship between the world coordinate system and the pixel coordinate system through a series of algorithms.

[0063] 4) Homography matrix

[0064] The homography matrix refers to the projection matrix from one plane to another plane, so the homography matrix can represent the geometric transformation relationship from one plane to another plane.

[0065] In the image field, the homography matrix can represent the geometric transformation relationship between two images and can be applied to image correction, image stitching, camera pose estimation, visual simultaneous localization and mapping (SLAM) and other fields.

[0066] Taking the field of image stitching as an example, since the homography matrix can represent the geometric transformation relationship between two images, the homography matrix can be used to transform two images at different viewing angles to the same viewing angle, thereby facilitating image stitching. Specifically, the homography matrix can be used to project two images at different viewing angles to the same plane so that the two images are at the same viewing angle, thereby facilitating the stitching of the two images. Among them, the process of projecting two images at different viewing angles to the same plane through the homography matrix is ​​also called image reprojection.

[0067] The homography matrix can be calculated by the feature points with matching relationship in two images under different viewing angles. Among them, the feature points with matching relationship in the two images refer to the pixel points / blocks corresponding to the same physical object (such as the same object / person / animal / position, etc.) in the two images. For example, for image 1 and image 2, the pixel points / blocks corresponding to physical object 1 (such as table corners, wall corners, etc.) in image 1 and the pixel points / blocks of physical object 1 in image 2 are called a pair of feature points with matching relationship in image 1 and image 2. It can be seen that only two images of the same physical object in the physical space can solve the homography matrix that represents the geometric transformation relationship between the two images.

[0068] Specifically, when the feature points with matching relationship in two images are called feature point pairs, the coordinate mapping relationship equation established based on the feature point pairs can be solved to obtain the homography matrix representing the geometric transformation relationship between the two images. For example, for the feature point pairs a1 and a2 in image 1 and image 2, the coordinate mapping relationship equation a1=H*a2 can be established based on a1 and a2. By solving the coordinate mapping relationship equation established based on all the feature point pairs in image 1 and image 2, the homography matrix H representing the geometric transformation relationship between image 1 and image 2 can be obtained.

[0069] Where H is a 3x3 matrix, including 8 degrees of freedom (i.e., 8 unknowns). Since the feature points in a two-dimensional image are represented by the horizontal coordinate x and the vertical coordinate y in the two-dimensional image, two coordinate mapping relationship equations can be constructed based on a pair of feature points. For example, for the feature point pair a1 and a2, the coordinate mapping relationship equations established based on a1 and a2 include: a1 x =H*a2 x , and a1 y =H*a2 y . Furthermore, at least 4 point pairs are needed to find the unique solution H.

[0070] 5) Shooting field of view

[0071] The shooting field of view of a video (or image) refers to the field of view of the camera / lens that shoots the video (or image). The range of the field of view of the camera / lens corresponds to the field of view angle of the camera / lens.

[0072] 6) Other terms

[0073] In the embodiments of the present application, the terms "first" and "second" do not indicate an order relationship, but are used to distinguish different objects. The first, second, etc. mentioned in the following documents are also used to distinguish different messages, etc., and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0074] It should also be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0075] In the embodiment of the present application, stitching videos means stitching multiple videos with different shooting fields of view. The shooting fields of the multiple videos have overlapping areas, and the shooting fields of the multiple videos can cover the area of ​​interest to the user. Therefore, after stitching the multiple videos based on the overlapping shooting fields of view, the shooting field of the stitched video obtained can cover the area of ​​interest to the user. In this way, the stitched video is a panoramic video of the area of ​​interest to the user.

[0076] Specifically, stitching multiple videos with different shooting fields of view and overlapping shooting fields of view means stitching image frames with the same timestamp and overlapping shooting fields of view in the multiple videos, and the stitched image is a frame of image in the stitched video, and the timestamp of the frame of image in the stitched video is the same as the aforementioned timestamp. The timestamp of the image frame in the video can be the time when the image frame is actually shot. Alternatively, the timestamp of the image frame in the video represents the offset of the image frame relative to the initial image frame in the video, which is not limited.

[0077] As an example, for video 1, video 2, and video 3 shot synchronously and having the same frame rate, video 1, video 2, and video 3 have different shooting fields of view, and the shooting fields of view have overlapping areas. Therefore, when video 1, video 2, and video 3 are stitched together based on the overlapping shooting fields of view, the specific process is as follows: stitching together the first frame image of video 1, the first frame image of video 2, and the first frame image of video 3 based on the overlapping shooting fields of view to obtain stitched image 1; stitching together the second frame image of video 1, the second frame image of video 2, and the second frame image of video 3 based on the overlapping shooting fields of view to obtain stitched image 2, ..., and stitching together the mth frame image of video 1, the mth frame image of video 2, and the mth frame image of video 3 based on the overlapping shooting fields of view to obtain stitched image m, where m is a positive integer greater than 2. In this way, the video composed of stitched image 1, stitched image 2, ..., and stitched image m is the stitched video obtained after stitching together video 1, video 2, and video 3.

[0078] In the scenario where image frames from multiple videos are stitched together, in order to prevent objects such as faces (or people, objects, etc.) located near the stitching seam in the stitched image from being incompletely displayed due to the stitching seam, the stitching seam position in the stitched image can be adjusted in real time so that objects such as faces (or people, objects, etc.) located near the stitching seam position are fully displayed. However, this will cause the image frames in the stitched video to jitter in the time domain, that is, the spliced ​​video will have poor stability in the time domain. Among them, the detailed description of the image jitter can be referred to the above description and will not be repeated.

[0079] In an example, it is assumed that the field of view of camera 1 and the field of view of camera 2 overlap, and camera 1 and camera 2 shoot videos synchronously. Figure 1 , Figure 1 A schematic diagram showing a video spliced ​​with jitter in the time domain due to a change in the position of a seam is shown.

[0080] like Figure 1 As shown in (a), image 1 is the mth frame image in video 1 taken by camera 1, and image 2 is the mth frame image in video 2 taken by camera 2. Image 1 and image 2 have the same timestamp, both image 1 and image 2 include images of overlapping areas, and object 1 in image 1 and object 1 in image 2 are the same target object in the overlapping area. It should be understood that due to the binocular parallax between camera 1 and camera 2, the display position of the target object in image 1 and image 2 is different.

[0081] like Figure 1 As shown in (b) in the figure, when image 1 and image 2 are stitched together, the stitching seam determined by the stitching algorithm is Figure 1The stitching 1 shown in (b) in the figure is used to stitch image 1 and image 2 according to the stitching 1, and we can get Figure 1 The m-th frame stitched image shown in (b) of FIG. 7 . The target object in the m-th frame stitched image is object 1 in image 1 captured by camera 1 .

[0082] like Figure 1 As shown in (c) in FIG. 1 , when the next frame image of image 1 in video 1 and the next frame image of image 2 in video 2 are spliced, the seam determined by the splicing algorithm is Figure 1 The stitching 2 shown in (c) in FIG. 1 is used to stitch the next frame of image 1 and the next frame of image 2 according to the stitching 2, and we can get Figure 1 The m+1th frame stitched image shown in (c) in FIG. The target object in the m+1th frame stitched image is object 1 in image 3 taken by camera 2. Image 3 is the next frame image of image 2 in video 2.

[0083] It can be seen that the target object is Figure 1 The display positions in the n+1th frame stitched image and the nth frame stitched image are different. In other words, in the stitched video obtained by stitching video 1 and video 2, the display position of the same target object in two consecutive frames jumps, that is, the stitched video has screen jitter.

[0084] Based on this, an embodiment of the present application provides an image processing method, which can achieve that when the jitter amplitude of the first image frame and the second image frame is less than or equal to a preset threshold, the target spliced ​​image obtained after the first image frame and the second image frame are spliced ​​will not have screen jitter in the time domain due to the change of the splicing position. Therefore, the method provided in the embodiment of the present application can improve the stability of the target spliced ​​video obtained after the first video and the second video are spliced ​​in the time domain.

[0085] Optionally, the jitter of the first image frame and the second image frame includes at least one of the following: image jitter caused by the movement of the camera / lens used to shoot the first image frame and the second image frame, image jitter caused by the movement / movement of the target object when shooting the first image frame and the second image frame, image jitter caused by the change of the ambient light in which the camera / lens is located when shooting the first image frame and the second image frame, and the like. Therefore, when the jitter amplitude of the first image frame and the second image frame is less than or equal to the preset threshold, it indicates that the camera / lens used to shoot the first image frame and the second image frame may have slightly moved, or the target object may have slightly moved when shooting the first image frame and the second image frame, or the ambient light in which the camera / lens is located may have slightly changed when shooting the first image frame and the second image frame, and the like. By using the method provided in the embodiment of the present application, it is possible to avoid jittering of the target spliced ​​video in the time domain when these situations occur, and thus, the method can improve the stability of the target spliced ​​video in the time domain.

[0086] An embodiment of the present application also provides an image processing system, which includes an image processing device and multiple image acquisition devices.

[0087] Wherein, a plurality of image acquisition devices are used to synchronously acquire video of a target scene. In the target scene, each of the plurality of image acquisition devices has a different shooting field of view, and the shooting fields of view of the image acquisition devices arranged adjacently in the plurality of image acquisition devices overlap. In this way, the shooting fields of view of the plurality of image acquisition devices can cover the target scene. Wherein, the target scene can be a conference room, a live video broadcast, a monitoring place with legal authorization, etc., but is not limited thereto.

[0088] The image processing device is used to stitch the videos collected by the above-mentioned multiple image acquisition devices online or offline to obtain a stitched video, which is a panoramic video of the target scene. Among them, the video stitching described in the embodiment of the present application refers to stitching image frames with the same timestamp in the video to be stitched. The detailed stitching process of stitching image frames with the same timestamp can be referred to the method description below, which will not be repeated here.

[0089] refer to Figure 2 , Figure 2 FIG. 1 is a schematic diagram showing the architecture of an image processing system provided by an embodiment of the present application. Figure 2As shown, the image processing system 20 includes three image acquisition devices arranged in the target scene and whose shooting field of view covers the target scene, which are respectively image acquisition device 211, image acquisition device 212, and image acquisition device 213. Among them, the image acquisition device 211, the image acquisition device 212, and the image acquisition device 213 can synchronously acquire the video of the target scene, so as to obtain the video 1 acquired by the image acquisition device 211, the video 2 acquired by the image acquisition device 212, and the video 3 acquired by the image acquisition device 213. The image processing system 20 also includes an image processing device 22. The image processing device 22 is used to splice the video 1, video 2, and video 3 acquired by the above three image acquisition devices online or offline to obtain a spliced ​​video including a panoramic view of the target scene.

[0090] refer to Figure 3 , Figure 3 FIG. 2 shows a schematic diagram of the architecture of another image processing system provided by an embodiment of the present application. Figure 3 As shown, the image processing system 30 includes q image acquisition devices 21 (such as Figure 2 In addition to the image acquisition device 211, image acquisition device 212, and image acquisition device 213 shown in the figure and the image processing device 22, the image processing system 30 optionally further includes a display module 33, an audio module 34, and a network module 35. The display module 33 is used to display the image of the video spliced ​​by the image processing device 22, the audio module 34 is used to output the audio of the video spliced ​​by the image processing device 22, and the network module 35 is used to send the video spliced ​​by the image processing device 22 to a remote device.

[0091] It is understandable that the display module 33 and the image processing device 22 can be integrated into one device (such as a computing device) or into different devices, such as the display module 33 is a display device connected to the image processing device 22, but is not limited thereto. The audio module 34 can be integrated into one device (such as a computing device) and the image processing device 22, or into different devices, such as the audio module 34 is an audio device of an independent device connected to the image processing device 22, but is not limited thereto.

[0092] In addition, in some possible cases, the above-mentioned multiple image acquisition devices are integrated into one device. As an example, the device can be a panoramic camera including multiple lenses. It should be understood that the multiple lenses in the panoramic camera are generally controlled by one processing module. In this case, when the panoramic camera starts shooting, the multiple lenses in the panoramic camera shoot synchronously.

[0093] In this case, the image processing device may be an independent computing device, or the image processing device may be integrated with multiple image acquisition devices in one device, without limitation. Optionally, the computing device may be a terminal device such as a general-purpose computer or a laptop computer, or the computing device may be a network computing device such as a server, without limitation.

[0094] In some other possible situations, the above-mentioned multiple image acquisition devices are integrated into multiple devices respectively. Exemplarily, each of the multiple image acquisition devices is an independent electronic device with a shooting function, such as a digital camera, a mobile phone, a car computer, a camera, etc. Another exemplary embodiment, some of the multiple image acquisition devices are integrated into one device (such as panoramic camera 1), and another part of the image acquisition devices are integrated into another device (such as panoramic camera 2), but it is not limited thereto. Among them, panoramic camera 1 and panoramic camera 2 are two different panoramic cameras.

[0095] In this case, the image processing device may be an independent computing device, or the image processing device may be integrated with any one of the above-mentioned multiple image acquisition devices, without limitation. Optionally, the computing device may be a terminal device such as a general-purpose computer or a laptop computer, or the computing device may be a network computing device such as a server. Not limited thereto.

[0096] It should be noted that when the above-mentioned multiple image acquisition devices are respectively integrated in multiple devices, it is necessary to control the multiple image acquisition devices through a control device to perform synchronous shooting. Optionally, the control device can be integrated with the image processing device, or the control device is integrated with any one of the above-mentioned multiple image acquisition devices, or the control device is a control device independent of the image processing device and the image acquisition device, but is not limited thereto.

[0097] The following is an exemplary description of the layout positions of the above-mentioned multiple image acquisition devices.

[0098] In one example, in a smaller indoor scene, such as a conference room scene, in order to make the shooting fields of view of multiple image acquisition devices completely cover all target objects in the conference room, reference Figure 4 , Figure 4 The schematic diagram of deploying multiple image acquisition devices provided by the embodiment of the present application is shown. The target objects in the conference room include but are not limited to participants, electronic whiteboards, projection equipment, large screens, etc.

[0099] like Figure 4As shown, a panoramic camera including lens 1, lens 2 and lens 3 is arranged at the center of the conference room to achieve the purpose of covering all target objects in the conference room with the shooting field of view. Among them, the shooting fields of view of any two adjacent lenses in the panoramic camera have overlapping areas. Among them, the shooting field of view of lens 1 in the panoramic camera is range ad, and the corresponding field of view angle is α. The shooting field of view of lens 2 in the panoramic camera is range cf, and the corresponding field of view angle is β. And, the shooting field of view of lens 3 in the panoramic camera is range eb, and the corresponding field of view angle is γ.

[0100] Among them, the range ad and the range cf have an overlapping range cd, and the overlapping range cd is the overlapping area of ​​the field of view of lens 1 and the field of view of lens 2. The range cf and the range eb have an overlapping range ef, and the overlapping range ef is the overlapping area of ​​the field of view of lens 2 and the field of view of lens 3. And, the range eb and the range ad have an overlapping range ab, and the overlapping range ab is the overlapping area of ​​the field of view of lens 3 and the field of view of lens 1.

[0101] In another example, in a large open space, in order to make the shooting fields of view of multiple image acquisition devices cover most areas of the open space, refer to Figure 5 , Figure 5 A schematic diagram of another arrangement of multiple image acquisition devices provided in an embodiment of the present application is shown.

[0102] like Figure 5 As shown, cameras 51, 52, 53 and 54 are arranged at the four edge vertices of the place 50. The shooting fields of cameras 51, 52, 53 and 54 are as follows: Figure 5 The range indicated by the arrow in the figure can cover most of the area in the place 50. In addition, the shooting fields of the cameras 51, 52, 53 and 54 that are arranged adjacently have overlapping areas. For example, the overlapping area of ​​the shooting field of view of the camera 51 and the shooting field of view of the camera 52 is Figure 5 For another example, the overlapping area of ​​the camera 51's field of view and the camera 53's field of view is Figure 5 The elliptical area shown in shaded area, and so on.

[0103] The embodiment of the present application also provides an image processing device, which is the image processing device in the above-mentioned image processing system, and is used to execute the image processing method described in the embodiment of the present application to realize the splicing of the videos collected by the above-mentioned multiple image collection devices. Optionally, the image processing device provided in the embodiment of the present application can be any computing device with computing processing capabilities, or a functional module in a computing device, etc., without limitation.

[0104] Taking the image processing device as a computing device as an example, refer to Figure 6 , Figure 6 FIG. 1 is a schematic diagram showing a hardware structure of a computing device provided in an embodiment of the present application. Figure 6 As shown, the computing device 60 includes a processor 601, a memory 602, a communication interface 603, and a bus 604. The processor 601, the memory 602, and the communication interface 603 are connected via the bus 604. Optionally, the computing device 60 also includes an input / output interface 605, and the input / output interface 605 communicates with the processor 601, the memory 602, the communication interface 603, etc. via the bus 604.

[0105] Processor 601 is the control center of computing device 60, which may be a general-purpose central processing unit (CPU). Processor 601 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, graphics processing units (GPU), neural processing units (NPU), tensor processing units (TPU) or artificial intelligence (artificial intelligent) chips, data processors (DPU), etc.

[0106] As an example, the processor 601 includes one or more CPUs, such as Figure 6 In addition, the present application does not limit the number of processor cores in each processor.

[0107] The memory 602 is used to store program instructions or data to be accessed by the application process. The processor 601 can implement the image processing method provided in the embodiment of the present application by executing the program instructions in the memory 602.

[0108] The memory 602 includes a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the volatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM) and direct rambus RAM (DR RAM). Non-volatile memory can be storage class memory (SCM), solid state drive (SSD), hard disk drive (HDD), etc. Among them, storage class memory can be, for example, non-volatile memory (NVM), phase-change memory (PCM), persistent memory, etc.

[0109] In a possible implementation, the memory 602 exists independently of the processor 601. The memory 602 is connected to the processor 601 via a bus 604 and is used to store data, instructions or program codes. When the processor 601 calls and executes the instructions or program codes stored in the memory 602, the image processing method provided in the embodiment of the present application can be implemented.

[0110] In another possible implementation, the memory 602 and the processor 601 are integrated together.

[0111] The communication interface 603 is used for the computing device 60 to connect with other devices (such as image acquisition devices, etc.) through a communication network, and the communication network can be Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. Alternatively, the communication interface 603 is used for the computing device 60 to communicate with other devices (such as image acquisition devices, etc.) through a cable connection. The communication interface 603 includes a receiving unit for receiving data / messages, and a sending unit for sending data / messages.

[0112] The bus 604 may be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, a high-speed serial computer expansion bus (PCIe), a compute express link (CXL) or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0113] The input and output interface 605 is used to implement human-computer interaction between the user and the computing device 60, such as displaying a spliced ​​video to the user.

[0114] The input / output interface 605 includes an input interface for enabling a user to input information to the computing device 60 , and an output interface for enabling the computing device 60 to output information to the user.

[0115] As an example, the input interface includes but is not limited to a touch screen, a keyboard, a mouse or a microphone, etc., and the output interface includes but is not limited to a display screen, a speaker, etc. Among them, the touch screen, keyboard or mouse is used to input text / image information, the microphone is used to input voice information, the display screen is used to output text / image information, and the speaker is used to output voice information. It can be understood that, in combination with Figure 3 , when the output interface is a display screen, the output interface corresponds to Figure 3 The display module 33 shown. When the output interface is a speaker, the output interface corresponds to Figure 3 The audio module 34 is shown.

[0116] It should be pointed out that Figure 6 The structure shown in the figure does not constitute a limitation on the computing device 60, except Figure 6 In addition to the components shown, the computing device 60 includes Figure 6 More or fewer components may be shown, or certain components may be combined, or the components may be arranged differently.

[0117] The image processing method provided in the embodiment of the present application is described below with reference to the accompanying drawings.

[0118] It should be understood that, since multiple videos are stitched together, image frames in videos with the same timestamp and overlapping shooting fields are actually stitched together in pairs, thereby obtaining image frames in the stitched video.

[0119] As an example, in the process of stitching three videos with different shooting fields of view and overlapping shooting fields of view, when stitching image frame 1, image frame 2 and image frame 3 with the same timestamp in these three videos, assuming that the shooting field of view of image frame 1 and the shooting field of view of image frame 3 have overlapping areas, and the shooting field of view of image frame 2 and the shooting field of view of image frame 3 have overlapping areas, image frame 1 and image frame 3 can be stitched first to obtain stitched image 1, and then stitched image 1 and image frame 2 are stitched to obtain stitched image 2 (or, image frame 2 and image frame 3 can be stitched first to obtain stitched image 1, and then stitched image 1 and image frame 1 are stitched to obtain stitched image 2). In this way, stitched image 2 is the image frame obtained after stitching image frame 1, image frame 2 and image frame 3. In other words, stitched image 2 is an image frame in the video after stitching video 1, video 2 and video 3. Among them, the shooting field of view of image frame 1 is the shooting field of view of video 1. Similarly, the shooting field of view of image frame 2 is the shooting field of view of video 2, and the shooting field of view of image frame 3 is the shooting field of view of video 3.

[0120] Therefore, for the sake of simplicity, the embodiments of the present application are introduced below by taking the stitching of two videos (such as the first video and the second video) with overlapping shooting fields as an example.

[0121] refer to Figure 7 , Figure 7 The figure shows a flow chart of an image processing method provided by an embodiment of the present application. Optionally, the method can be applied to Figure 2 or Figure 3 The image processing system shown in FIG. Figure 6 The method is performed by an image processing device with the hardware structure shown in FIG.

[0122] S101 . Obtain images to be stitched, where the images to be stitched include a first image frame in a first video and a second image frame in a second video.

[0123] The shooting field of view of the first video and the shooting field of view of the second video have an overlapping area.

[0124] In some examples, the first video may be Figure 4 The video captured by lens 1 in the panoramic camera shown in the figure, the second video can be Figure 4 The video captured by lens 2 adjacent to lens 1 in the panoramic camera shown is not limited thereto.

[0125] In other examples, the first video may be Figure 5 The video captured by the camera 51 shown in the figure, the second video can be Figure 5 The video captured by the camera 52 arranged adjacent to the camera 51 is shown, but not limited to this.

[0126] Since the shooting field of view of the first video and the shooting field of view of the second video have an overlapping area, and the target object is located in the overlapping area. Therefore, the above-mentioned image to be stitched includes a first sub-image and a second sub-image. The first sub-image is an image of the overlapping area in the first image frame, and the second sub-image is an image of the overlapping area in the second image frame. Moreover, the first sub-image and the second sub-image both include the target object and the background. Among them, the target object in the image refers to the area in the image where the target object is displayed, and the background in the image refers to the area in the image other than the target object.

[0127] It should be understood that the target object can be any object that the user is interested in. For example, the target object can be a person, a face, an animal, an object, etc., but is not limited thereto. The detailed description of the image processing device identifying the target object and the background in the first sub-image and the second sub-image can be referred to the detailed description of S201 below, which will not be repeated here.

[0128] As an example, when the images to be stitched are image frames in a video recording a meeting scene, the target object is the participants in the conference room, and the target object in the images to be stitched is the participants displayed in the images to be stitched. Similarly, the target object in the first sub-image is the participants displayed in the first sub-image, and the target object in the second sub-image is the participants displayed in the second sub-image.

[0129] refer to Figure 8 , Figure 8 A schematic diagram of a first image frame, a second image frame and an overlapping area image is shown. Figure 8As shown, the image of the overlapping area in the first image frame is the first sub-image, and the image of the overlapping area in the second image frame is the second sub-image. Among them, the target object is displayed as object 81 in the first sub-image, and the area except object 81 in the first sub-image is the background of the first sub-image. Due to the binocular time difference between the camera / lens that shoots the first image frame and the second image frame, the target object is displayed as object 82 in the second sub-image, and the area except object 82 in the second sub-image is the background area of ​​the second sub-image.

[0130] In some embodiments, the target object in the image may also be referred to as the foreground in the image, and the area where the foreground in the image is located may be referred to as the foreground area. Optionally, the foreground area in the image may be an area surrounded by the outline of the target object, or the foreground area in the image may be an area represented by a position box including the target object, which is not limited.

[0131] In addition, the first video and the second video are videos synchronously captured by two image capture devices, and the first video and the second video have the same frame rate. In this way, the first image frame and the second image frame in the above-mentioned image to be stitched have the same timestamp. For the sake of simplicity, the embodiments of the present application are described below by taking the timestamp of the first image frame and the timestamp of the second image frame as the first timestamp as an example.

[0132] Optionally, assuming that the first video is captured by a first image acquisition device and the second video is captured by a second image acquisition device, the image processing device can extract the first image frame currently captured by the first image acquisition device and the second image frame currently captured by the second image acquisition device during the process in which the first image acquisition device captures the first video in real time and the second image acquisition device captures the second video in real time, thereby obtaining the first image frame and the second image frame to be spliced. The first image acquisition device and the second acquisition device respectively capture videos synchronously at the same frame rate.

[0133] Optionally, the image processing device can extract the first image frame from the first video and the second image frame from the second video based on the timestamps of the image frames in the first video and the second video in advance, thereby obtaining the first image frame and the second image frame to be spliced.

[0134] Furthermore, after acquiring the first image frame and the second image frame, the image processing device determines to determine the first sub-image in the first image frame and to determine the second sub-image in the second image frame. In other words, after acquiring the first image frame and the second image frame, the image processing device determines the image of the overlapping area (e.g. Figure 8 Image of the overlapping area shown).

[0135] In a possible implementation, when the image processing device for capturing the first video and the second video is a panoramic camera with multiple fixed lenses, the field of view of each lens in the panoramic camera is preset, and the overlapping field of view of adjacent lenses is also preset. Therefore, in this case, the image processing device can determine the overlapping field of view of two adjacent lenses according to the field of view angle parameters of the panoramic camera, and based on the determined overlapping field of view, can determine the image of the overlapping area in the first image frame and the second image frame.

[0136] In another possible implementation, assuming that the first video is captured by the first image acquisition device and the second video is captured by the second image acquisition device, the user can manually measure and calibrate the deployed first image acquisition device and the second image acquisition device in advance to determine the feature points with matching relationships in the image captured by the first image acquisition device and the image captured by the second image acquisition device, and determine the image area in the first image frame that includes all the feature points with matching relationships as the image of the overlapping area in the first image frame, and determine the image area in the second image frame that includes all the feature points with matching relationships as the image of the overlapping area in the second image frame. The detailed description of the feature points with matching relationships can be referred to the above description, which will not be repeated here.

[0137] It can be understood that, for the first video captured by the first image acquisition device and the second video captured by the second image acquisition device, the image format of the first video and the second video can be RGB (red, green, blue), YUV (luminance, chrominance), HSV (hue, saturation, value) and other formats, without limitation. The lens resolution of the first image acquisition device and the second image acquisition device can be 4K, or 1080P, 720P, 540P, 360P, etc., without limitation. Among them, P refers to progressive scan. In addition, the display ratio of the video screen captured by the first image acquisition device and the second image acquisition device can be 4:3 or 16:9, etc., without limitation.

[0138] Optionally, for the first image acquisition device and the second image acquisition device, the parameters of the lens resolution, display ratio, etc. of the two may be the same or different. When the parameters of the lens resolution, display ratio, etc. of the two are different, in S101, the image processing device needs to convert the video image acquired by the first image acquisition device and the video image acquired by the second image acquisition device into a unified format, and the specific process is not described in detail here.

[0139] S102: When the jitter amplitude of the image to be spliced ​​is less than or equal to a preset threshold, determine a first seam position.

[0140] The first stitching position is the position of the stitching that bypasses the target object in the reference sub-image (hereinafter referred to as the first stitching position), and the first stitching position is located on the same side of the target object in the reference sub-image and the target object in the historical sub-image. Here, the reference sub-image is one of the first sub-image or the second sub-image, the historical sub-image is the image of the overlapping area in the historical stitching image, and the historical stitching image is a stitching image obtained by stitching the historical frame located before the first image frame in the first video and the historical frame located before the second image frame in the second video.

[0141] The historical frames before the first image frame include a preset number of image frames before the first image frame in the first video, and the historical frames before the second image frame include a preset number of image frames before the second image frame in the second video, where the preset number is an integer greater than or equal to 1. In this case, when a spliced ​​image obtained by splicing the first sub-image and the second sub-image is a target spliced ​​image, and a video obtained by splicing the first video and the second video is a target spliced ​​video, the historical spliced ​​image is a preset number of image frames before the target spliced ​​image in the target spliced ​​video.

[0142] As an example, when the preset number is 2, and the first image frame is the mth image frame in the first video, and the second image frame is the mth image frame in the second video, then the historical frames before the first image frame include the (m-1)th image frame and the (m-2)th image frame in the first video, and the historical frames before the second image frame include the (m-1)th image frame and the (m-2)th image frame in the second video, the target stitched image is the mth image frame in the target stitched video, and the historical stitched images are the (m-1)th image frame and the (m-2)th image frame in the target stitched video. It should be understood that the (m-1)th image frame in the target stitched video is a stitched image obtained by stitching the (m-1)th image frame in the first video and the (m-1)th image frame in the second video, and the (m-2)th image frame in the target stitched video is a stitched image obtained by stitching the (m-2)th image frame in the first video and the (m-2)th image frame in the second video.

[0143] It should be understood that when determining the seam for stitching the first sub-image and the second sub-image, in order to avoid the incomplete display of the target object (such as a face, an object, etc.) in the overlapping area image caused by the seam (that is, the target object is cut off by the seam), the current stitching algorithm needs to use one of the first sub-image or the second sub-image as the reference sub-image, and make the seam bypass the target object with complex texture in the reference sub-image. In this way, the first sub-image and the second sub-image are stitched based on the seam that bypasses the target object in the reference sub-image, which can ensure the purpose of fully displaying the target object in the overlapping area of ​​the image to be stitched. Therefore, in an embodiment of the present application, for the target object in the reference sub-image, the first seam position determined by the image processing device not only needs to bypass the target object, but also needs to be located on the same side of the target object in the reference sub-image and the target object in the historical sub-image.

[0144] In other words, when the above-mentioned preset number is r, and r is an integer greater than or equal to 1, it means that the number of historical stitched images is r, that is, the historical stitched images actually include the first r frame images of the target stitched image in the target stitched video, and the historical sub-image includes the image of the overlapping area in the first r frame images, which is called the first r frame sub-image. In this case, the first seam position determined by the image processing device is located on the same side of the target object in the current reference sub-image and the target object in the first r frame sub-image. That is to say, in the target stitched video, in the first r frame images and the target image frame obtained by splicing the first image frame and the second image frame, the target object in the overlapping area is located on the same side of the seam, that is, in the image of consecutive r+1 frames in the target stitched video, the target objects in the overlapping area are all located on the same side of the seam. In this way, in the images of consecutive r+1 frames in the target stitched video, the target objects located in the overlapping area are all target objects displayed in the image frames in the same video to be stitched (the first video or the second video). This can ensure in the time domain that the position of the target object displayed in the stitched image will not change suddenly due to the change of the stitching position, thereby ensuring in the time domain that the image in the overlapping area of ​​the stitched image will not shake, thereby ensuring the stability of the video picture in the time domain.

[0145] As an example, refer to Fig. 9 , Fig. 9 A schematic diagram of a seam position change in the time domain provided by an embodiment of the present application is shown.

[0146] like Fig. 9 As shown in (a) in the figure, the reference sub-image is the first sub-image (i.e. Figure 8When the target object 80 is displayed as object 91 in the historical sub-image, and the seam 92 in the historical sub-image is located on the right side of the object 91, the target object 80 is displayed as object 81 in the reference sub-image. In this case, when the target object 80 moves in the direction indicated by the arrow in the historical sub-image, the image processing device determines that the seam that bypasses the object 81 is located on the right side of the object 81 in the reference sub-image. Fig. 9 That is, the seam position (ie, the first seam position) determined by the image processing device in the reference sub-image is located on the same side of the target object 80 in the reference sub-image and the target object 80 in the history sub-image.

[0147] like Fig. 9 As shown in (b) in the figure, the reference sub-image is the first sub-image (i.e. Figure 8 The target object 80 is displayed as object 81 in the reference sub-image. When the target object 80 is displayed as object 94 in the history sub-image, and the seam 95 in the history sub-image is located on the right side of the object 94. In this case, when the target object 80 moves in the direction indicated by the arrow in the history sub-image, in the reference sub-image, the image processing device determines that the seam that bypasses the object 81 is located on the right side of the object 81, as shown in FIG. Fig. 9 That is, the seam position (ie, the first seam position) determined by the image processing device in the reference sub-image is located on the same side of the target object 80 in the reference sub-image and the target object 80 in the history sub-image.

[0148] Specifically, S102 can be implemented through steps S1021 and S1022.

[0149] S1021. Determine the jitter amplitudes of the first image frame and the second image frame.

[0150] Optionally, the jitter of the first image frame and the second image frame may include at least one of the following: picture jitter caused by the movement of the camera / lens when shooting the first image frame and the second image frame, picture jitter caused by the movement / movement of the target object when shooting the first image frame and the second image frame, picture jitter caused by the change of ambient lighting in the camera / lens when shooting the first image frame and the second image frame, and the like.

[0151] Specifically, the image processing device may first determine the jitter amplitude of the first image frame and the jitter amplitude of the second image frame respectively. The embodiment of the present application does not specifically limit the specific method of determining the jitter amplitude of the first image frame and the jitter amplitude of the second image frame. For example, the jitter amplitude may be calculated only based on the image frame, or the jitter amplitude of the image frame may be detected in combination with sensors such as acceleration and gyroscope. Among them, the image-based jitter calculation method includes but is not limited to: optical flow method, block matching method, feature point matching method, grayscale projection method, etc.

[0152] In one example, taking the calculation of the jitter amplitude of the first image frame as an example, the grayscale projection method can be used to calculate the jitter amplitude of the first image frame. Specifically, the grayscale mean of the pixels in each row and column in the first image frame is first calculated. Then, grayscale projection is performed row (column) respectively, and the projected value is the average value of the grayscale mean of each row (column) minus the grayscale mean of all rows (columns). Then, the obtained row and column grayscale projections are expanded to obtain the grayscale projection curve corresponding to the first image frame. The image processing device can determine whether the picture jitter occurs in the first image frame by comparing the grayscale projection curve of the first image frame with the grayscale projection curve of the previous frame image in the first video, and the difference between the grayscale projection curve of the first image frame and the grayscale projection curve of the previous frame image in the first video is the jitter amplitude of the first image frame.

[0153] In another example, taking the calculation of the jitter amplitude of the first image frame as an example, the accelerometer, gyroscope and other sensors configured in the image acquisition device for shooting the first image frame can be used to detect whether the image acquisition device for shooting the first video is displaced when shooting the first image frame. If the sensor detects displacement, it is determined that the first image frame currently captured has jittered, and the image processing device can determine the jitter amplitude based on the displacement detected by the sensor. It should be understood that the displacement here refers to the displacement of the image acquisition device during the process from the image acquisition device shooting the previous frame of the first image frame to the image acquisition device shooting the first image frame.

[0154] In another example, taking the calculation of the jitter amplitude of the first image frame as an example, a motion detection method can be used to determine whether the first image frame is jittering and the amplitude of the jitter. For example, by performing frame difference calculation on the first image frame and the previous frame of the first image frame in the first video, or first detecting the position of the target object in the first image frame and the previous frame, and performing a difference calculation on the position box representing the target object in the two frames. When the difference result is greater than the set threshold, it means that the target object in the first video has moved rapidly, that is, the first image frame has jittered. Then, the jitter amplitude can be determined based on the difference result.

[0155] In another example, taking the calculation of the jitter amplitude of the first image frame as an example, a light sensor can be used to detect the light intensity and color change of the shooting environment. When the first image frame is shot, if the light intensity or color of the shooting environment changes suddenly, it is determined that the image frame is jittered, and the jitter amplitude of the first image frame is determined based on the sudden change value of the light intensity and the change value of the color. The light sensor can be integrated into the image acquisition device that shoots the first image frame, or it can be a device independent of the image acquisition device, and this is not limited.

[0156] It should be understood that the above-mentioned several exemplary methods for determining the image jitter amplitude may also be used in combination, and this is not limited.

[0157] After determining the jitter amplitude of the first image frame and the jitter amplitude of the second image frame, optionally, when the image processing device determines that the jitter amplitude of the first image frame is less than or equal to the first threshold, or determines that the jitter amplitude of the second image frame is less than or equal to the first threshold, the image processing device determines that the jitter amplitude of the first image frame and the second image frame is less than a preset threshold. The embodiment of the present application does not limit the value of the first threshold.

[0158] Optionally, when the image processing device determines that the sum of the jitter amplitude of the first image frame and the jitter amplitude of the second image frame is less than or equal to the second threshold, the image processing device determines that the jitter amplitude of the first image frame and the second image frame is less than a preset threshold. The embodiment of the present application does not limit the value of the second threshold.

[0159] It should be understood that when the shaking amplitude of the first image frame and the second image frame is less than the preset threshold, it can be understood that slight shaking occurs in the first image frame and the second image frame.

[0160] Furthermore, when the jitter amplitude of the first image frame and the second image frame is less than or equal to the preset threshold, the image processing device performs S1022. When the jitter amplitude of the first image frame and the second image frame is greater than the preset threshold, the image processing device determines the stitching position for stitching the first image frame and the second image frame based on the current algorithm.

[0161] S1022: When the jitter amplitudes of the first image frame and the second image frame are less than or equal to a preset threshold, determine a first seam position.

[0162] Specifically, the image processing device may first determine the reference sub-image in the first sub-image and the second sub-image. Optionally, the image processing device may randomly determine the reference sub-image in the first sub-image and the second sub-image. Alternatively, the image processing device may determine the reference sub-image in the first sub-image and the second sub-image based on a preset rule. The embodiments of the present application do not specifically limit the preset rule.

[0163] The image processing device may divide the reference sub-image into a plurality of sub-regions. Here, the embodiment of the present application does not limit the number of sub-regions obtained by the division, for example, the number may be a predetermined number. Optionally, the image processing device may divide the reference sub-image into a plurality of sub-regions with random sizes, or divide the reference sub-image into a plurality of sub-regions with preset sizes, and this is not limited.

[0164] As an example, refer to Fig.10 , Fig.10 FIG. 1 shows a schematic diagram of dividing a reference sub-image into multiple sub-areas provided by an embodiment of the present application. Fig.10 As shown, the image processing device divides the reference sub-image into 9 sub-regions, which are 9 sub-regions numbered 1 to 9 respectively.

[0165] In a first possible implementation manner, the image processing apparatus may determine the first seam position and the second seam position starting from the first sub-region in the reference sub-image.

[0166] The first sub-region is a sub-region at a preset position in the reference sub-image. Optionally, the sub-region at a preset position in the reference sub-image may be any sub-region in the reference sub-image, or a sub-region at a specified position, which is not limited. Fig.10 , the first sub-region can be Fig.10 The sub-region 1 numbered 1 or the sub-region 5 numbered 5 shown is not limited thereto.

[0167] The second seam position is the position of the seam other than the first seam in the reference sub-image (hereinafter referred to as the second seam). Fig.11 , Fig.11 FIG. 1 is a schematic diagram showing a first seam and a second seam provided in an embodiment of the present application. Fig.11 As shown, the joint 93 shown by the solid line in the reference sub-figure is the first joint, and the position of the joint 93 is the first joint position. The joints 111 and 112 other than the first joint shown by the dotted line in the reference sub-figure are the second joints, and the positions of the joints 111 and 112 are the second joint positions.

[0168] Specifically, the image processing apparatus determines whether the first sub-region includes the target object.

[0169] When the first sub-region does not include the target object, that is, the first sub-region is the background, the image processing device can determine the background sub-seam according to the first sub-region through S301 described below. It should be understood that since the background may be distributed in multiple sub-regions, the seam determined based on a sub-region as the background is a section of the second seam mentioned above, which is referred to as the background sub-seam. In other words, the background sub-seam is a section of the second seam.

[0170] When the first sub-region includes a target object, the image processing device determines a first sub-seam that bypasses the target object in the first sub-region in the reference sub-region based on the positional relationship between the seam that bypasses the target object in the historical sub-region and the target object, and the first sub-seam is located on the same side of the target object in the first sub-region and the target object in the historical sub-region. Since the target object may be distributed in several sub-regions, the seam that bypasses the target object determined based on the target object in a sub-region is a section of the first seam mentioned above, which is referred to as a sub-seam here. It should be understood that the first sub-seam determined by the image processing device in the reference sub-region may be located in the first sub-region or outside the first sub-region, and this is not limited to this.

[0171] Optionally, the image processing device may determine the first sub-seam according to the seam corresponding to the target object in the first sub-region in the historical sub-graph. The seam corresponding to the target object in the first sub-region in the historical sub-graph refers to a seam that bypasses the target object in the historical sub-graph, and hereinafter, "the seam corresponding to the target object in the first sub-region in the historical sub-graph" is referred to as the first historical sub-seam.

[0172] In some possible cases, when the first historical sub-seam does not pass through the target object in the first sub-region, the image processing device may determine the first historical sub-seam as the first sub-seam that bypasses the target object in the first sub-region, thereby determining the position of the first historical sub-seam in the historical sub-image as the position of the first sub-seam in the reference sub-image. Fig. 9 As shown in (a), for example, the target object 80 moves from the position of the object 91 in the historical sub-image to the position of the object 81 in the base sub-image, that is, the target object 80 moves slightly. Therefore, the seam 92 that bypasses the target object 80 in the historical sub-image is determined as the seam (i.e., seam 93) that bypasses the target object 80 in the base sub-image. In other words, the position of the seam 92 that bypasses the target object 80 in the historical sub-image is the same as the position of the seam 93 that bypasses the target object 80 in the base sub-image.

[0173] In some other possible situations, when the first historical sub-seam passes through the target object in the first sub-region, the first sub-seam that bypasses the target object in the first sub-region can be determined based on the first historical sub-seam. The position difference between the first sub-seam and the first historical sub-seam is less than a threshold value, which can ensure that the position of the seam that bypasses the target object is relatively close in two consecutive frames of stitched images. Since the seams in the stitched images can generally be clearly perceived by the user in the current technology, shortening the distance between the seam positions in two consecutive frames of stitched images can improve the image stability of the two consecutive frames of stitched images.

[0174] Optionally, when the value of the preset number mentioned above is greater than 1, that is, the number of historical stitching images is multiple, and the number of corresponding historical sub-images is also multiple, then the seams in the multiple historical sub-images that bypass the target object in the first sub-region are called the first group of historical sub-seams. In this case, the image processing device can determine the first sub-seams that bypass the target object in the first sub-region based on the first group of historical sub-seams. Specifically, the image processing device can determine the seam in the first group of historical sub-seams that is closest to the target edge of the historical sub-image as the first sub-seams that bypass the target object in the first sub-region, and the corresponding position is the position of the first sub-seams. The first group of historical sub-seams is located between the target edge of the historical sub-image and the target object.

[0175] As an example, combining Fig. 9 In (b), the target edge is the right side of the historical sub-image / reference sub-image / first sub-image, and the seam is located between the target edge and the target object 80. And taking the value of the preset number equal to 2 as an example, when the image processing device splices the next frame image of the first image frame and the next frame image of the second image frame, the first group of historical sub-seams includes Fig. 9 , and the seam 95 shown in (b) of FIG. 1 and the seam 93 in the reference sub-image (i.e., the first sub-image). If the image processing device determines that the seam 95 and the seam 93 are both located on the right side of the target object in the next frame image, and the seam 93 in the reference sub-image (i.e., the first sub-image) is closer to the target edge, the image processing device may determine the seam 93 as the seam of the next frame image, and determine the position in the next frame image that is the same as the position of the seam 93 in the reference sub-image as the position of the seam that bypasses the target object in the next frame image.

[0176] In this way, the image processing device can set the first sub-seam that bypasses the target object outside the motion range of the target object in the continuous multiple image frames. In this way, when the seam that bypasses the target object is subsequently determined, if the subsequent motion range of the target object does not exceed the aforementioned motion range, the position of the seam that bypasses the target object can be kept unchanged, which can improve the stability of the spliced ​​video picture.

[0177] Optionally, the above-mentioned preset number can periodically take a value greater than 1, and within a cycle, the preset number can be taken to be equal to 1. In this way, the image processing device can periodically determine the position of the sub-seam that bypasses the target object in the current reference sub-image based on the position of a group of historical sub-seams, and within a cycle, the image processing device determines the position of the sub-seam that bypasses the target object in the current reference sub-image only based on the position of the seam in the previous frame of the spliced ​​image. Through this combination, the computing power requirements of the image processing device can be reduced while ensuring the stability of the video picture.

[0178] After determining the background sub-seam or the first sub-seam according to the first sub-region through the first possible implementation method, the image processing device determines the second sub-region according to the determined background sub-seam or the first sub-seam. For example, after the image processing device determines the first sub-seam according to the first sub-region, the sub-region adjacent to the sub-region including the first sub-seam and intersecting with the first sub-seam is determined as the second sub-region, and the seam is determined according to the second sub-region through the first possible implementation method until a complete seam in the reference sub-image is determined, which will not be described in detail. Fig.11 , when the first sub-region is sub-region 3 numbered 3, and the image processing device determines the background sub-seam in sub-region 3, that is, Fig.11 As shown in the patchwork 111, the image processing device determines the sub-region 6 numbered 6 adjacent to the sub-region 3 including the patchwork 111 and intersecting the patchwork 111 as the second sub-region, and determines the patchwork according to the second sub-region through the first possible implementation method mentioned above, which is not repeated here.

[0179] In this way, the image processing device determines a plurality of sub-seams according to all sub-regions including the target object in the reference sub-image. The plurality of sub-seams constitute the first seam mentioned above, and the positions of the plurality of sub-seams are the first seam positions.

[0180] In addition, when at least one boundary of the first sub-region is a boundary of the reference sub-image, when the image processing device determines the above-mentioned first sub-seam, if the distance between the target object in the first sub-region and the boundary of the reference sub-image is less than a threshold, the first sub-seam is allowed to be located on both sides of the target object in the first sub-region and the target object in the historical sub-image, for example Figure 1 In this case, the image processing device uses the current stitching algorithm to determine the first sub-seam. The embodiment of the present application does not specifically limit the specific value of the threshold, for example, the threshold is 10 pixels, 8 pixels, etc.

[0181] In a second possible implementation, the image processing device may first determine one or more sub-regions including the target object in the reference sub-image. Then, the image processing device determines a first seam that bypasses the target object in the reference sub-image based on the positional relationship between the seam of the target object in the historical sub-image and the target object. The detailed process can refer to the description of determining the first sub-seam above, which will not be repeated here.

[0182] Then, optionally, the image processing device determines the second seam and the second seam position in the remaining sub-regions except the one or more sub-regions in the reference sub-image based on the current stitching algorithm, or determines the second seam and the second seam position based on the method described in S301 below, which is not limited to this. It should be understood that since the one or more sub-regions are sub-regions of the reference sub-image including the target object, the remaining sub-regions except the one or more sub-regions in the reference sub-image are the background areas.

[0183] It should be understood that when determining the first seam by the above two implementation methods, it depends on the seam position in the historical sub-image. In other words, the seam determined according to the initial frame of the video to be spliced ​​where the reference sub-image is located will continue to affect the determination of the seam position for the image frames in the subsequent video. Among them, the seam position of the initial frame is determined based on the current splicing algorithm. In order to improve the stability of the spliced ​​video picture, the image processing device can periodically determine the seam for splicing the image frames in the video to be spliced ​​based on the current splicing algorithm, so that the image processing device can use the seam as the initial seam within a cycle to determine the seam of the subsequent image frames to be spliced ​​within the cycle. Because in a short time, such as within a cycle, the jitter of the first video and the second video may be slight, such as the target object in the field of view of the first video and the second video shooting within a cycle moves slightly. In this case, the jitter amplitude of the image frame to be spliced ​​is less than or equal to the preset threshold, so the seam can be determined based on the above two methods to improve the picture stability of the spliced ​​video.

[0184] S103 , stitching the first image frame and the second image frame according to the first stitching seam position to obtain a target stitching image.

[0185] Specifically, the image processing device splices the first image frame and the second image frame according to the first splicing position and the second splicing position determined above, so as to obtain a target spliced ​​image. Here, the specific splicing process is not described in detail in the embodiment of the present application.

[0186] It should be understood that the target stitched image is a frame of image in the target stitched video obtained by stitching the first video and the second video, and the timestamp of the target stitched image in the target stitched video is the first timestamp mentioned above, that is, the timestamp of the first image frame in the first video, or the timestamp of the second image frame in the second video.

[0187] In this way, through the method described in S101 to S103, when the images in the to-be-joined videos shake slightly, the stability of the joined videos in the time domain can be improved.

[0188] In other embodiments, reference Fig.12 , Fig.12FIG. 1 is a flow chart of another image processing method provided by an embodiment of the present application. Optionally, the method can be applied to Figure 2 or Figure 3 The image processing system shown in FIG. Figure 6 The image processing device of the hardware structure shown is executed. After executing S101 described above, the method further executes the following steps.

[0189] S201: Determine a target object and a background in a first sub-image in a first image frame and a second sub-image in a second image frame.

[0190] Among them, the detailed description of the first sub-graph and the second sub-graph can be referred to the above description and will not be repeated here.

[0191] Optionally, the image processing device may determine the target object and background in the first sub-image by identifying the target object and background in the first image frame, and the image processing device may determine the target object and background in the second sub-image by identifying the target object and background in the second image frame.

[0192] Specifically, taking the first image frame as an example, the image processing device may first identify all target objects in the first image frame, and after identifying all target objects in the first image frame, determine the identified target objects located in the first sub-image as the target objects in the first sub-image, and determine the area other than the target objects in the first sub-image as the background of the first sub-image. Among them, the embodiment of the present application does not specifically limit the specific implementation method of identifying the target object in the first image frame. As can be seen from the above, the target object in the embodiment of the present application can also be called the foreground,

[0193] In one example, the image processing device can identify objects such as human bodies and faces in the first image frame and the second image frame through target detection technology, face detection technology, etc., and determine the identified human bodies, faces, etc. as target objects, and determine the area other than the target object as the background.

[0194] In another example, a moving object in an image is determined as a target object by a motion detection method. When detecting a moving object in an image, a frame difference method, a background difference method, an optical flow method, etc. may be used but are not limited to. For example, by performing a difference calculation on a first image frame and one or more frames of images located before the first image frame in a first video, pixels with a difference result greater than a threshold value may be determined as pixels showing a moving object (i.e., a target object), and pixels with a difference result less than a threshold value may be determined as background pixels. For another example, when the background is known and the background is stationary, a difference calculation is performed on the first image frame and a pre-stored background image, so that pixels with a difference result greater than a threshold value may be determined as pixels showing a moving object (i.e., a target object), and pixels with a difference result less than a threshold value may be determined as background pixels. For another example, the change in a two-dimensional image is evaluated based on the grayscale change of corresponding pixels in the first image frame and the previous frame of the first image frame in the first video. Specifically, pixels whose grayscale change difference between corresponding pixels in the first image frame and the previous image frame is greater than a threshold are determined as pixels displaying moving objects (ie, target objects), and pixels whose grayscale change difference is less than the threshold are determined as background pixels.

[0195] In another example, the foreground (i.e., the target object) and the background are separated by image segmentation, thereby determining the pixels showing the target object and the pixels showing the background in the image. The image segmentation algorithm is a pixel-level foreground-background separation algorithm, and the embodiment of the present application does not specifically limit the algorithm.

[0196] In another example, the above-mentioned methods of identifying the target object and the background in the image may also be used in combination, without limitation.

[0197] Through the above-mentioned example, the category probability of each pixel in the first image frame can be determined, and the category probability is the probability indicating that the pixel belongs to the target object or the background. Optionally, when the category probability of a pixel is greater than a threshold, it indicates that the pixel belongs to the target object. Correspondingly, when the category probability of a pixel is less than a threshold, it indicates that the pixel belongs to the background. Among them, when the value of the category probability is a normalized value, specifically any value between 0 and 1. In this case, the threshold can be 0.5, but is certainly not limited to this.

[0198] As an example, taking the threshold value of 0.5 as an example, when the category probability of pixel 1 in the first image frame is determined to be 0 through the above method, it means that pixel 1 belongs to the background. When the category probability of pixel 2 in the first image frame is determined to be 0.3 through the above method, it means that pixel 2 also belongs to the background. When the category probability of pixel 3 in the first image frame is determined to be 0.7 through the above method, it means that pixel 3 belongs to the target object.

[0199] In some embodiments, after the image processing device determines the category probability of each pixel in the first sub-image in the above manner, when the image processing device determines the first sub-image as a reference sub-image in S1022 and divides the reference sub-image into sub-regions, when the image processing device determines whether the sub-region includes the target object, it can determine whether the sub-region includes the target object or is a background that does not include the target object based on the category probability of each pixel in the sub-region.

[0200] For example, when a sub-region V[i][j] in the reference sub-image has N pixels, i, j, and N are all positive integers, where N of the N pixels f The category probability of pixels is greater than the threshold, that is, the N f In this case, the image processing apparatus determines that the probability value of sub-region V[i][j] being a region including the target object is p=N. f When the probability value p is greater than a certain threshold, the image processing device determines that the sub-region V[i][j] is a region including the target object, and when the probability value p is less than the threshold, the image processing device determines that the sub-region V[i][j] is the background.

[0201] For another example, when the sub-region V[i][j] in the reference sub-image includes at least one pixel whose class probability is greater than a threshold, the image processing device determines that the sub-region V[i][j] is a region including the target object. When there is no pixel whose class probability is greater than the threshold in the sub-region V[i][j], the image processing device determines that the sub-region V[i][j] is the background.

[0202] S202: Determine a geometric transformation relationship between the first image frame and the second image frame.

[0203] The geometric transformation relationship between images can also be understood as the geometric position relationship between images, or as the geometric position relationship between cameras / lenses that capture images. Through the geometric position relationship between two images, the two images can be projected onto the same plane. And, determining the geometric transformation relationship between two images can also be called image registration of the two images.

[0204] Specifically, the geometric transformation relationship between images can be represented by a homography matrix, and the geometric transformation relationship between the first image frame and the second image frame can be represented by a target homography matrix. The detailed description of the homography matrix can refer to the description in the technical terms above, which will not be repeated here.

[0205] It should be noted that when the image processing device solves the target homography matrix in S202, it is specifically calculated based on all the feature points with matching relationships in the entire image of the first image frame and the entire image of the second image frame, so the target homography matrix is ​​a global homography matrix. The detailed description of the feature points with matching relationships in the image can be referred to the above description and will not be repeated here.

[0206] In a possible implementation, the image processing device may detect feature points in the first sub-image and the second sub-image determined in S101, and match the feature points detected in the first sub-image and the second sub-image, thereby determining feature points having a matching relationship in the first sub-image and the second sub-image. The image processing device calculates the target homography matrix based on all the determined feature points having a matching relationship.

[0207] Among them, the method for the image processing device to detect feature points in the image includes but is not limited to: scale-invariant feature transform (SIFT), speeded-up robust features (SURF), Harris, Oriented FAST and Rotated BRIEF (ORB), optical flow method and other algorithms. And the method for the image processing device to determine the feature points with matching relationship among the detected feature points includes but is not limited to: brute-force matcher algorithm, random sample consensus (RANSAC) algorithm, nearest neighbor algorithm (K-Nearest Neighbor, KNN) matching algorithm, etc.

[0208] Alternatively, the image processing device may also utilize a deep learning model to detect and match feature points in the first sub-image and the second sub-image, thereby calculating a target homography matrix based on all determined feature points having a matching relationship.

[0209] In another possible implementation, the first image acquisition device for shooting the first image frame and the second image acquisition device for shooting the second image frame can be calibrated offline by camera calibration to obtain a global homography matrix that characterizes the geometric position relationship between the first image acquisition device and the second image acquisition device. The global homography matrix can characterize the geometric transformation relationship between the images shot by the two image acquisition devices. In other words, the global homography matrix can characterize the geometric transformation relationship between the first image frame and the second image frame, that is, the global homography matrix is ​​the target homography matrix mentioned above. Then, the global homography matrix obtained after offline calibration of the first image acquisition device and the second image acquisition device is pre-set in the image processing device.

[0210] The above-mentioned process of offline calibration of the image acquisition device may be performed before the image acquisition device leaves the factory, or when the image acquisition device is installed, or after the image acquisition device is installed, and there is no limitation on this.

[0211] Optionally, the first image acquisition device and the second image acquisition device may be offline calibrated using a conventional camera calibration method. Specifically, the first image acquisition device and the second image acquisition device may be calibrated using a chessboard as a calibration reference. The specific process is as follows: a chessboard is placed in a target scene of interest to the user (such as a conference room), and a plurality of images containing the chessboard are captured at different shooting distances and different shooting angles using the image acquisition device to be calibrated. Then, the corner positions of the chessboard in the image are detected, and the camera parameters of the image acquisition device to be calibrated are obtained by solving the calibration algorithm. The calibration algorithm may include but is not limited to the Zhang Zhengyou calibration algorithm. The camera parameters include camera intrinsic parameters and camera extrinsic parameters, the camera intrinsic parameters include an intrinsic parameter matrix and a distortion parameter matrix, and the camera extrinsic parameters include a rotation matrix and a translation matrix. According to the camera parameters obtained by calibrating the first image acquisition device and the camera parameters obtained by calibrating the second image acquisition device, a global homography matrix characterizing the geometric position relationship between the first image acquisition device and the second image acquisition device may be calculated.

[0212] Optionally, an active visual camera calibration method may be used to calibrate the camera using certain known camera motion information. Alternatively, a self-calibration algorithm, such as Hartley's QR decomposition method, absolute quadratic surface method, etc., may be used to achieve offline calibration of the image acquisition device. Among them, QR decomposition is to decompose a matrix into the product of an orthogonal matrix Q and an upper triangular matrix R.

[0213] S203 . Reproject the first image frame and the second image frame according to a geometric transformation relationship between the first image frame and the second image frame.

[0214] Specifically, the image processing device projects the first image frame and the second image frame onto the same plane according to a target homography matrix representing the geometric transformation relationship between the first image frame and the second image frame. This process is called a process of reprojecting the first image frame and the second image frame.

[0215] Optionally, when the image processing device reprojects the first image frame and the second image frame according to the geometric transformation relationship between the first image frame and the second image frame, the projection method adopted includes but is not limited to: rectilinear projection, equidistant cylindrical projection, equisolidangle projection, stereographic projection, orthographic projection, panini projection, etc.

[0216] The present embodiment of the application does not describe the image reprojection process in detail.

[0217] It should be noted that the embodiment of the present application does not specifically limit the order in which the image processing device executes S201 and S202-S203. For example, the image processing device may first execute S201, and then execute S202-S203. Alternatively, the image processing device may execute S201 and S202-S203 at the same time. Alternatively, the image processing device may first execute S202-S203, and then execute S201.

[0218] It should also be noted that, during the process of the image processing device executing S201, S202, and S203, the image processing device may also execute S1021 and S1022 described above. In this regard, the embodiment of the present application does not specifically limit the order in which the image processing device executes S1021, S1022, S201, S202, and S203. For example, the image processing device may execute S201→S202→S203→S1021→S1022 in sequence, or execute S201→S1021→S202→S203→S1022 in sequence, or execute S1021→S201→S202→S203→S1022 in sequence, and so on, without limitation thereto.

[0219] Next, the image processing apparatus executes S204 to replace S103 described above.

[0220] S204 . According to the first stitching seam position, stitch the reprojected first image frame and the reprojected second image frame to obtain a target stitching image.

[0221] Specifically, the image processing device splices the reprojected first image frame and the reprojected second image frame according to the first stitching position and the second stitching position determined in S1022, thereby obtaining a target stitching image. The specific stitching process is not described in detail in this embodiment of the application.

[0222] S205 (optional): smoothing the seams of the target stitched image to obtain a smoothed target stitched image.

[0223] Specifically, the image processing device uses an image fusion algorithm to smooth the seams in the target stitched image, so that the target stitched image can be smoothed at the seams. Optionally, the image fusion algorithm used by the image processing device includes but is not limited to: weighted average method, feathering algorithm, pyramid fusion algorithm, etc.

[0224] Among them, the weighted average method is to make the image transition from left to right by simply using weighted methods. The algorithm is simple and the speed is fast.

[0225] The feathering algorithm achieves a blurring effect by gradually processing the edge of the image, thereby achieving a smooth transition of the image. This method is more natural than the weighted average method.

[0226] Pyramid fusion algorithm, also known as multi-resolution fusion algorithm, multi-band fusion algorithm, etc. This method can build an image into a Laplace pyramid, and each layer of the Laplace pyramid contains different frequency band information of the image, and then fuse each frequency band separately to achieve the purpose of image fusion. This method has a good smoothing effect and a natural transition.

[0227] S206: Display the target stitched image.

[0228] Optionally, the image processing device may send the target stitched image to a display module (such as Figure 3 The display module 33 is shown, and the target stitched image is displayed by the display module.

[0229] Optionally, the display module may be a display module of a device to which the image processing apparatus belongs.

[0230] Optionally, the display module is a display module of a remote device connected to the device to which the image processing apparatus belongs. In this case, the image processing apparatus can display the image through a network module (such as a network module) of the device to which it belongs. Figure 3 The network module 35 shown in the figure sends the target spliced ​​image to the remote device to which the display module belongs, so that the remote device displays the target spliced ​​image through its own display module.

[0231] In this way, through the method described in S201 to S205, the stability of the target spliced ​​video can be ensured in terms of timing while the splicing of the image frames in the video to be spliced ​​can be completely achieved.

[0232] In some other embodiments, in order to further ensure the stability of the target spliced ​​video in terms of timing, reference is made to Fig.13 , Fig.13 A flowchart of another image processing method provided by an embodiment of the present application is shown. After executing S101 and S1021, and when it is determined in S1022 that the sub-region of the reference sub-image does not include the target object, the image processing device may use S301 to determine the second seam position.

[0233] S301: Determine a second seam position.

[0234] Specifically, when the image processing device determines that a sub-region in the above-mentioned reference sub-image, such as the first sub-region, does not include the target object, it means that the first sub-region is the background. In this case, the image processing device determines the first target region according to the position of the first sub-region in the reference sub-image, and the first target region refers to the region in the historical sub-image with the same position as the first sub-region in the reference sub-image.

[0235] Furthermore, the image processing device determines whether there is a seam in the historical sub-image in the first target area.

[0236] When the image processing device determines that there is a seam in the history sub-image in the first target area, the image processing device determines the seam as a background sub-seam in the first sub-area, and determines the position in the first sub-area that is the same as the seam position in the first target area as the position of the background sub-seam. The background sub-seam in the first sub-area is a section of the second seam mentioned above.

[0237] When the image processing device determines that there is no seam in the historical sub-image in the first target area, it determines that there is no seam in the first sub-area, determines the next sub-area to be processed according to the preset order, and performs S1022 and S301 for the sub-area, which will not be repeated. The embodiment of the present application does not limit the preset order.

[0238] In this case, for all sub-regions that do not include the target object in the reference sub-image, that is, the background region, the image processing device can determine multiple background sub-seams by executing S301, and the multiple background sub-seams constitute the second seam, and the positions of the multiple background sub-seams are the second seam positions. In combination with the above, it can be seen that the second seam position is the same position as the seam position of the target region in the historical sub-image in the reference sub-image, and the target region is the region in the historical sub-image that has the same background position as the reference sub-image.

[0239] After the image processing device determines the first stitching position through S1022 and determines the second stitching position through S301, the image processing device continues to execute S103 or S204 to stitch the first image frame and the second image frame according to the first stitching position and the second stitching position to obtain a target stitching video.

[0240] Thus, through S301, for the background in the reference sub-image, the seam at the same position as the background in the historical sub-image can be used. In other words, the seam position in the background of the reference sub-image is the seam position at the same position as the background in the historical sub-image. In this way, in a scene where the target object in the target scene has a small range of activity and the background in the target scene is static, the seam position of the background area in the target spliced ​​video is basically unchanged in the time domain, which can ensure the stability of the target spliced ​​video in the time domain.

[0241] In some other embodiments, to improve the accuracy of reprojecting the first image frame and the second image frame, reference Fig.14 , Fig.14 A schematic flow chart of another image processing method provided by an embodiment of the present application is shown. In the method, the image processing device may continue to execute S401 after executing S202.

[0242] S401 , performing time-domain smoothing processing on a target homography matrix representing a geometric transformation relationship between a first image frame and a second image frame according to a first weight and a historical frame homography matrix.

[0243] The historical homography matrix is ​​a homography matrix for reprojecting the historical frame mentioned above when obtaining the historical spliced ​​image mentioned above. One frame of historical spliced ​​image corresponds to one historical homography matrix, and the historical homography matrix is ​​a global homography matrix. For the description of the global homography matrix, reference may be made to the relevant description in S202. For the detailed description of the historical spliced ​​image and the historical frame, reference may be made to the relevant description in S102, which will not be repeated here.

[0244] Optionally, the historical stitched image includes one or more frames of images before the target stitched image in the target stitched video.

[0245] For the sake of simplicity, the historical stitching image is described below as an example of the previous frame image of the target stitching image in the target stitching video. In this case, the number of historical homography matrices corresponding to the historical stitching image is 1. Specifically, assuming that the previous frame image of the first image frame in the first video is called the first historical image frame, and the previous frame image of the second image frame in the second video is called the second historical image frame, then for the previous frame stitching image of the target stitching image in the target stitching video, the previous frame stitching image is the stitching image obtained by stitching the first historical image frame and the second historical image frame, and the historical homography matrix is ​​the homography matrix used to reproject the first historical image frame and the second historical image frame when stitching the first historical image frame and the second historical image frame. In addition, the homography matrix is ​​calculated based on all the feature points with matching relationships in the entire image of the first historical image frame and the entire image of the second historical image frame.

[0246] In this case, the image processing device can perform temporal smoothing on the target homography matrix according to the first weight and the historical homography matrix used to reproject the first historical image frame and the second historical image frame. The historical homography matrix can be a homography matrix that has been smoothed in the temporal domain or a homography matrix that has not been smoothed in the temporal domain, and there is no limitation on this.

[0247] Optionally, assuming that the first image frame and the second image frame are the nth spliced ​​image frame in the video to be spliced, the target homography matrix calculated based on the first image frame and the second image frame and not subjected to time domain smoothing is T(n), and the homography matrix calculated based on the n-1th frame image in the video to be spliced, that is, the historical homography matrix is ​​S(n-1). Among them, S(n-1) can be a homography matrix that has been smoothed in the time domain, or a homography matrix that has not been smoothed in the time domain, and there is no limitation on this. In this case, the target homography matrix S(n) that has been smoothed in the time domain can be calculated by the following formula (1):

[0248] Formula (1) S(n) = w × S(n-1) + (1-w) × T(n)

[0249] Wherein, w is the preset value of the first weight, and the value range of w is [0,1].

[0250] Optionally, when the difference between the two homography matrices is large, the effect of achieving time domain smoothing by weighted averaging the target homography matrix and the historical homography matrix is ​​limited. Therefore, assuming that the feature points with matching relationships used to calculate the target homography matrix are called target feature point pairs, the multiple coordinate mapping relationship equations established based on the target feature point pairs are called the first set of equations. Assuming that the feature points with matching relationships used to calculate the historical homography matrix are called historical feature point pairs, the multiple coordinate mapping relationship equations established based on the historical feature point pairs are called the second set of equations. In this case, the image processing device can weight the coordinate mapping relationship equations in the first set of equations and the second set of equations by a first weight with a value interval between [0,1], and jointly solve the target homography matrix according to all the weighted coordinate mapping relationship equations in the first set of equations and the second set of equations, and the target homography moment is the target homography matrix after time domain smoothing. It can be seen that the historical feature point pairs participate in the calculation of the target homography matrix of the current image to be spliced ​​and after time domain smoothing. In this way, the accuracy of the target homography matrix can be improved, thereby improving the accuracy of the reprojection.

[0251] As an example, assume that the first image frame and the second image frame are the nth stitched image frame in the video to be stitched, assume that the target homography matrix calculated based on the first image frame and the second image frame and not subjected to time domain smoothing is called T(n), and the homography matrix calculated based on the n-1th frame image in the video to be stitched, that is, the historical homography matrix is ​​called S(n-1), when the feature point pair used to calculate T(n) is p(n), then the set of coordinate mapping relationship equations established based on p(n) is T_p(n), and when the feature point pair used to calculate S(n-1) is p(n-1), then the set of coordinate mapping relationship equations established based on p(n) is S_p(n-1). When the first weight is w, the image processing device can use w to weight S_p(n-1) and use (1-w) to weight T_p(n). Then the image processing device jointly solves the weighted S_p(n-1) and T_p(n) to obtain the target homography matrix S(n) after time domain smoothing.

[0252] Next, the image processing apparatus executes S402 to replace S203.

[0253] S402 . Reproject the first image frame and the second image frame according to the time-domain smoothed target homography matrix.

[0254] Specifically, the image processing device projects the first image frame and the second image frame onto the same plane according to the target homography matrix processed by time domain smoothing, so as to complete the reprojection of the first image frame and the second image frame. For the related description, please refer to the description of S203, which will not be repeated here.

[0255] Then, the image processing device can perform S204 to S206 on the reprojected first image frame and the second image frame by using the determined first stitching position and the second stitching position to complete the stitching of the first image frame and the second image frame, thereby realizing the stitching of the first video and the second video. The specific process is described above and will not be repeated here.

[0256] In this way, through S401 to S402, it is possible to achieve smoothing in the time domain of the homography matrix used for reprojecting the current image to be stitched in the image stitching process. That is, the target homography matrix used for reprojecting the current image to be stitched is constrained by the historical homography matrix corresponding to the historical stitching image, so as to achieve smoothing in the time domain of the target homography matrix, which can improve the stability of the target stitching image obtained by the transition from the historical stitching image to the image stitching reprojected based on the target homography matrix, thereby improving the stability of the target stitching video.

[0257] In some other embodiments, to further improve the accuracy of reprojecting the first image frame and the second image frame, reference Fig.15 , Fig.15 A schematic flow chart of another image processing method provided by an embodiment of the present application is shown. In this method, the image processing device can use S501 to replace the above S202.

[0258] S501 , determining a target homography matrix representing a geometric transformation relationship between a first image frame and a second image frame, wherein the target homography matrix includes a global homography matrix and a plurality of local homography matrices.

[0259] Specifically, the image processing device can calculate the global homography matrix in the target homography matrix based on all feature points in the entire image of the first image frame and the entire image of the second image frame that have a matching relationship. The detailed process can refer to the relevant description in S202 and will not be repeated here.

[0260] The image processing device may also divide the overlapping area of ​​the first image frame and the second image frame into a plurality of sub-areas, the plurality of local homography matrices correspond to the plurality of sub-areas one by one, and a local homography matrix is ​​calculated based on an image of a sub-area of ​​the overlapping area of ​​the first image frame and the second image frame. Fig.16 , Fig.16 FIG. 4 shows a schematic diagram of dividing the overlapping area of ​​a first image frame and a second image frame into a plurality of sub-areas. Fig.16 As shown, the overlapping area of ​​the first image frame and the second image frame is divided into 9 sub-areas.

[0261] The specific manner of dividing the overlapping area of ​​the first image frame and the second image frame into a plurality of sub-areas can refer to the description of dividing the reference sub-image into a plurality of sub-areas in S1022, which will not be described in detail. It should be understood that the plurality of sub-areas into which the overlapping area of ​​the first image frame and the second image frame is divided can be the same as or different from the plurality of sub-areas into which the reference sub-image is divided in S1022, and this is not limited.

[0262] Furthermore, the image processing device calculates the local homography matrix corresponding to each sub-region based on the image of each sub-region of the first image frame and the second image frame in the overlapping region. Specifically, for any sub-region in the overlapping region, such as the second sub-region, the image processing device can calculate the local homography matrix corresponding to the second sub-region based on the feature points having a matching relationship between the image of the first image frame in the second sub-region and the image of the second image frame in the second sub-region.

[0263] Optionally, for the global homography matrix and / or each local homography matrix determined in S501, the image processing device can perform S401 to achieve time domain smoothing of the global homography matrix, and achieve smoothing of each local homography matrix in the time domain. The detailed process of the image processing device performing S401 for the global homography matrix determined in S501 can be referred to above, and will not be repeated here.

[0264] Assuming that the first image frame and the second image frame are the nth stitched image in the video to be stitched, for any local homography matrix determined in S501, such as the first local homography matrix, when the first local homography matrix corresponds to the second sub-region of the overlapping region, and the first local homography matrix is ​​a local homography matrix calculated based on the images of the first image frame and the second image frame in the second sub-region, when the image processing device performs time domain smoothing on the local homography matrix according to formula (1) in S401, S(n) in formula (1) is the local homography matrix calculated based on the images of the first image frame and the second image frame in the second sub-region and subjected to time domain smoothing, T(n) is the local homography matrix calculated based on the images of the first image frame and the second image frame in the second sub-region and not subjected to time domain smoothing, and S(n-1) is the local homography matrix calculated based on the images of the previous frame of the first image frame and the previous frame of the second image frame in the second sub-region.

[0265] Next, the image processing apparatus executes S502.

[0266] S502: Perform spatial domain smoothing processing on the above-mentioned multiple local homography matrices.

[0267] Optionally, since the number of feature points with matching relationships in the image of the sub-region in the overlapping region may be small (e.g., less than 4 pairs), the local homography matrix calculated based on the feature points with matching relationships in the image of the sub-region is inaccurate. When the image of the sub-region is reprojected, the inaccurate local homography matrix will cause the projected image to be distorted, thereby affecting the image quality and stitching quality of the stitched image.

[0268] To avoid such problems, the image processing device filters the multiple local homography matrices corresponding to the overlapping areas according to the difference between the global homography matrix in the target homography matrix and each local homography matrix before performing spatial domain smoothing on the multiple local homography matrices corresponding to the overlapping areas, so as to remove inaccurate local homography matrices from the multiple local homography matrices.

[0269] It can be understood that, since the global homography matrix is ​​calculated based on the feature points with matching relationship in the entire image of the first image frame and the entire image of the second image frame, and the local homography matrix is ​​calculated based on some of the feature points in the aforementioned feature points, the difference between the global homography matrix and the local homography matrix should be less than the set threshold. Therefore, the image processing device can determine at least one local homography matrix whose difference with the global homography matrix is ​​less than the set threshold as the local homography matrix to be processed, and discard the local homography matrix whose difference with the global homography matrix is ​​greater than the set threshold.

[0270] Next, the image processing device may perform spatial domain smoothing processing on the local homography matrix to be processed by the following method 1 and / or method 2. Among them, when the image processing device adopts the following method 1 and method 2 to perform spatial domain smoothing processing on the local homography matrix to be processed, the embodiment of the present application does not limit the order in which the image processing device adopts method 1 and method 2. For example, the image processing device may first adopt method 1 to perform spatial domain smoothing processing on the local homography matrix to be processed, and then adopt method 2 to perform spatial domain smoothing processing on the local homography matrix to be processed. For another example, the image processing device may first adopt method 2 to perform spatial domain smoothing processing on the local homography matrix to be processed, and then adopt method 1 to perform spatial domain smoothing processing on the local homography matrix to be processed, and so on.

[0271] Mode 1: The image processing device performs spatial domain smoothing on the local homography matrix to be processed according to the second weight and the global homography matrix in the target homography matrix. The global homography matrix and the local homography matrix to be processed may be homography matrices that have been smoothed in the time domain or may be homography matrices that have not been smoothed in the time domain, without limitation. The second weight is a preset value, which is not limited in the embodiment of the present application.

[0272] Optionally, assuming that the first image frame and the second image frame are the nth stitched images in the video to be stitched, and the overlapping area of ​​the first image frame and the second image frame is divided into s×t sub-areas, then the local homography matrix corresponding to any sub-area image in the overlapping area without spatial domain smoothing is T[i][j], and the global homography matrix calculated based on the first image frame and the second image frame is T(n). Wherein, i is a positive integer less than or equal to s, and j is a positive integer less than or equal to t. Wherein, T[i][j] and / or T(n) can be homography matrices that have been smoothed in the time domain, or can be homography matrices that have not been smoothed in the time domain, and there is no limitation on this. In this case, the local homography matrix T[i][j] that has been smoothed in the spatial domain is new It can be calculated by the following formula (2):

[0273] Formula (2) T[i][j] new =w[i][j]×T(n)+(1-w[i][j])×T[i][j]

[0274] Among them, w[i][j] is the second weight.

[0275] Optionally, assuming that T[i][j] corresponds to a sub-region V[i][j] in the overlapping region, the second weight w[i][j] can be pre-set according to the distance between V[i][j] and the edge of the first image frame on one side of the overlapping region, and according to the distance between V[i][j] and the edge of the second image frame on one side of the overlapping region. The closer V[i][j] is to the center of the overlapping region, the smaller the value of V[i][j] is, and the smaller the correlation between T[i][j] corresponding to V[i][j] and T(n). The closer V[i][j] is to the edge of the overlapping region, the larger the value of V[i][j] is, and the greater the correlation between T[i][j] corresponding to V[i][j] and T(n). When w[i][j] takes the value of 1, it means that T[i][j] is the same as T(n).

[0276] As an example, in a scenario where the overlapping area is divided into 3×3 sub-areas, the overlapping area is divided into 3×3 sub-areas. In this case, w[i][j] can be configured as:

[0277]

[0278] Through this possible implementation, it is possible to constrain multiple local homography matrices corresponding to the overlapping area of ​​the first image frame and the second image frame by calculating the global homography matrix based on the first image frame and the second image frame, thereby spatially smoothing the multiple local homography matrices, and then achieving a smooth transition from the non-overlapping area to the overlapping area of ​​the reprojected first image frame and the second image frame. Therefore, this method can improve the image quality of the target stitched image and improve the stitching quality of the image.

[0279] Mode 2: The image processing device performs spatial domain smoothing on the above-mentioned multiple local homography matrices according to the differences between adjacent homography matrices in the above-mentioned local homography matrices to be processed. The sub-regions in the overlapping region corresponding to adjacent homography matrices are adjacent sub-regions in the overlapping region. The local homography matrix to be processed may be a homography matrix that has been smoothed in the time domain or a homography matrix that has not been smoothed in the time domain, and there is no limitation on this.

[0280] It should be understood that this method realizes smoothing of multiple local homography matrices corresponding to the overlapping area by smoothing the local homography matrices corresponding to the adjacent sub-areas in the overlapping area.

[0281] Optionally, assuming that the overlapping area of ​​the first image frame and the second image frame is divided into s×t sub-areas, the local homography matrix corresponding to any sub-area image in the overlapping area without spatial domain smoothing is T[i][j], and T[i][j] corresponds to the sub-area V[i][j] in the overlapping area, i is a positive integer less than or equal to s, and j is a positive integer less than or equal to t. Among them, T[i][j] can be a homography matrix that has been smoothed in the time domain, or a homography matrix that has not been smoothed in the time domain, and there is no limitation on this. In this case, the local homography matrix T[i][j] that has been smoothed in the spatial domain is new It can be calculated by the following formula (3):

[0282]

[0283] Wherein, k and l are both integers, and their value range is [-1, 1]. T[i+k][j+l] represents the local homography matrix corresponding to the sub-regions adjacent to the sub-region V[i][j] in the overlapping region. K represents the preset normalized Gaussian filter kernel.

[0284] As an example, in a scenario where the overlapping area is divided into 3×3, the overlapping area is divided into 3×3 sub-areas. In this case, K can be configured as:

[0285]

[0286] Next, the image processing apparatus executes S503 to replace S203.

[0287] S503. Reproject the images of the non-overlapping areas in the first image frame and the second image frame according to the global homography matrix in the target homography matrix, and reproject the images of the overlapping areas in the first image frame and the second image frame according to the multiple local homography matrices that have been smoothed in the spatial domain.

[0288] For two images with overlapping areas, there are differences in the local homography matrices calculated based on the images of different sub-areas in the overlapping area of ​​the two images. The difference is mainly caused by the different distances between the object and the camera / lens in different sub-areas. Therefore, the image of a sub-area is reprojected by the local homography matrix calculated from the image of the sub-area, and the projection accuracy is high.

[0289] Therefore, in the embodiment of the present application, different homography matrices are used to specifically reproject the non-overlapping area images and the overlapping area images in the first image frame and the second image frame, respectively, so that the reprojection accuracy of the first image frame and the second image frame in the non-overlapping area and the overlapping area can be improved, thereby improving the image quality and stitching quality of the target stitched image.

[0290] Then, the image processing device can perform S204 to S206 on the reprojected first image frame and the second image frame by using the determined first stitching position and the second stitching position to complete the stitching of the first image frame and the second image frame, thereby realizing the stitching of the first video and the second video. The specific process is described above and will not be repeated here.

[0291] In this way, through S501 to S503, it is possible to achieve, in the image stitching process, targeted reprojection of images in different areas of the first image frame and the second image frame using the corresponding homography matrix that has been smoothed in the spatial domain, thereby improving the reprojection accuracy of the first image frame and the second image frame, and further improving the image quality and stitching quality of the target stitched image.

[0292] In order to further deepen the understanding of the method provided in the embodiments of the present application, it is further described below through specific examples.

[0293] Example 1: Reference Fig.17 , Fig.17 Another schematic diagram of the image processing method provided in the embodiment of the present application is shown. Optionally, the method can be applied to Figure 2 or Figure 3 The image processing system shown in FIG. Figure 6 The image processing device of the hardware structure shown in FIG. Fig.17 As shown, the method includes the following steps.

[0294] S601 , obtaining image 1 in video 1 and image 2 in video 2.

[0295] The detailed process can be referred to the description of S101 and will not be described in detail.

[0296] S602: Determine whether the jitter of image 1 and image 2 is greater than a preset threshold.

[0297] When the jitter of image 1 and image 2 is greater than a preset threshold, the image processing device uses the current stitching algorithm to determine the stitching seam and stitch image 1 and image 2, and outputs a stitched image of image 1 and image 2.

[0298] When the jitters of the image 1 and the image 2 are less than or equal to the preset threshold, the image processing apparatus executes S603.

[0299] Among them, the detailed description of S602 can refer to the description of S1021, which will not be repeated here.

[0300] S603: Identify target objects in image 1 and image 2.

[0301] The process of the image processing device identifying the target object in the image 1 and the image 2 can refer to the description of S201 and will not be repeated here.

[0302] Furthermore, the image processing apparatus executes S604 for the foreground area including the target object in the overlapping area of ​​image 1 and image 2, and executes S605 for the background area in the overlapping area of ​​image 1 and image 2.

[0303] S604: Determine a first stitching position according to a foreground area including a target object in an overlapping area of ​​image 1 and image 2.

[0304] For detailed description, please refer to the description of S1022, which will not be repeated here.

[0305] S605 : Determine a second stitching position according to a background area in an overlapping area of ​​image 1 and image 2 .

[0306] For detailed description, please refer to the description of S301, which will not be repeated here.

[0307] S606 , stitching the image 1 and the image 2 according to the first stitching position and the second stitching position to obtain a stitched image.

[0308] For detailed description, please refer to S103 or S204, which will not be described in detail.

[0309] S607: Output a stitched image of image 1 and image 2.

[0310] For detailed description, please refer to S206 and will not be repeated here.

[0311] Example 2: Combination Fig.17 ,refer to Fig.18 , Fig.18 Another schematic diagram of the process of the image processing method provided in the embodiment of the present application is shown. Optionally, the method can be applied to Figure 2 or Figure 3 The image processing system shown in FIG. Figure 6 The image processing device of the hardware structure shown in FIG. Fig.18 As shown, after the image processing apparatus executes S602, the method executes S701.

[0312] S701: Determine whether to update the initial seam.

[0313] The detailed description of the initial seam can be found in the above description and will not be repeated here.

[0314] Optionally, the image processing device may determine whether the cycle for updating the initial seam has arrived, or the image processing device may determine whether instruction information for instructing to update the initial seam has been received, etc., which is not limited thereto.

[0315] When the image processing device determines that the initial stitching seam needs to be updated, the image processing device uses the current stitching algorithm to determine the stitching seam and stitches the image 1 and the image 2, and outputs a stitched image of the image 1 and the image 2.

[0316] When the image processing apparatus determines that the initial seam does not need to be updated, S603 to S607 are executed, which will not be described in detail.

[0317] The above mainly introduces the solution provided in the embodiment of the present application from the perspective of method.

[0318] In order to achieve the above functions, Fig.19 As shown, Fig.19 FIG. 1 is a schematic diagram showing the structure of an image processing device 190 provided in an embodiment of the present application. The image processing device 190 can be used to execute the above-mentioned image processing method, for example, to execute Figure 7 , Fig.12 , Fig.13 , Fig.14 , Fig.15 , Fig.17 or Fig.18 The image processing device 190 may include an acquisition unit 1901 , a determination unit 1902 , and a splicing unit 1903 .

[0319] The acquisition unit 1901 is used to acquire an image to be stitched including a first image frame in a first video and a second image frame in a second video. There is an overlapping area between the shooting field of view of the first video and the shooting field of view of the second video, and the image to be stitched includes a first sub-image and a second sub-image, the first sub-image is an image of the overlapping area in the first image frame, and the second sub-image is an image of the overlapping area in the second image frame, and the first sub-image and the second sub-image include a target object. The determination unit 1902 is used to determine a first stitching position when the jitter amplitude of the image to be stitched is less than or equal to a preset threshold. The first stitching position is located on the same side of the target object in the reference sub-image and the target object in the historical sub-image, the reference sub-image is the first sub-image or the second sub-image, the historical sub-image is an image of the overlapping area in the historical stitching image, and the historical stitching image is obtained by stitching the historical frame in the first video before the first image frame and the historical frame in the second video before the second image frame. The stitching unit 1903 is used to stitch the first image frame and the second image frame according to the first stitching position to obtain a target stitching image.

[0320] As an example, combining Figure 7 , the acquisition unit 1901 can be used to execute S101, the determination unit 1902 can be used to execute S102, and the splicing unit 1903 can be used to execute S103.

[0321] Optionally, the first sub-image and the second sub-image also include a background, and the background refers to the area in the image other than the target object. The determination unit 1902 is also used to determine a second stitching position when the jitter amplitude of the image to be stitched is less than or equal to a preset threshold value, and the second stitching position is the same position as the stitching position of the target area in the historical sub-image in the reference sub-image, and the target area is the same area in the historical sub-image as the background position in the reference sub-image. The stitching unit 1903 is specifically used to stitch the first image frame and the second image frame according to the first stitching position and the second stitching position.

[0322] As an example, combining Fig.13 The determining unit 1902 may be used to execute S301. The splicing unit 1903 may be used to execute S103.

[0323] Optionally, the image processing device 190 further includes: a display unit 1904, configured to display the target stitched image.

[0324] As an example, combining Fig.12 , the display unit 1904 can be used to execute S206.

[0325] Optionally, the historical frames before the first image frame include a preset number of image frames before the first image frame in the first video, and the historical frames before the second image frame include a preset number of image frames before the second image frame in the second video, where the preset number is greater than or equal to 1.

[0326] Optionally, determination unit 1902 is also used to determine the jitter amplitude of the first image frame and the jitter amplitude of the second image frame; and, when the jitter amplitude of the first image frame is less than or equal to a first threshold, or, the jitter amplitude of the second image frame is less than or equal to the first threshold, or, the sum of the jitter amplitude of the first image frame and the jitter amplitude of the second image frame is less than or equal to the second threshold, determine that the jitter amplitude of the image to be spliced ​​is less than or equal to a preset threshold.

[0327] As an example, combining Figure 7 , the determination unit 1902 can be used to execute S1021.

[0328] Optionally, the determining unit 1902 is configured to determine a target object and a background in the first sub-image and the second sub-image.

[0329] As an example, combining Fig.12 , the determination unit 1902 can be used to execute S201.

[0330] Optionally, the image processing device 190 further includes a reprojection unit 1905. The determination unit 1902 is configured to determine a geometric transformation relationship between the first image frame and the second image frame. The reprojection unit 1905 is configured to reproject the first image frame and the second image frame according to the geometric transformation relationship. The splicing unit 1903 is specifically configured to splice the reprojected first image frame and the reprojected second image frame according to the first splicing seam position.

[0331] As an example, combining Fig.12 , the determination unit 1902 can be used to execute S202, the reprojection unit 1905 can be used to execute S203, and the stitching unit 1903 can be used to execute S204.

[0332] Optionally, the geometric transformation relationship is represented by a target homography matrix, and the image processing device 190 further includes: a smoothing unit 1906, which is used to perform temporal smoothing processing on the target homography matrix according to the first weight and the historical frame homography matrix, and the historical homography matrix is ​​a homography matrix for reprojecting the historical frame when the historical spliced ​​image is obtained. A reprojection unit 1905 is specifically used to reproject the first image frame and the second image frame based on the target homography matrix that has been smoothed in the temporal domain.

[0333] As an example, combining Fig.14 , the smoothing unit 1906 can be used to execute S401, and the reprojection unit 1905 can be used to execute S402.

[0334] Optionally, the above-mentioned geometric transformation relationship is represented by a target homography matrix, and the target homography matrix includes a global homography matrix and multiple local homography matrices. Wherein, the global homography matrix is ​​calculated based on the entire image of the first image frame and the entire image of the second image frame, and the multiple local homography matrices correspond to the multiple sub-regions divided by the overlapping area, and a local homography matrix is ​​calculated based on the image of a sub-region in the overlapping area. The image processing device 190 also includes: a smoothing unit 1906, which is used to perform spatial domain smoothing processing on the local homography matrix according to the second weight and the global homography matrix, or, according to the difference between adjacent homography matrices in the multiple local homography matrices, perform spatial domain smoothing processing on the local homography matrix, wherein the sub-regions corresponding to the adjacent homography matrices are adjacent sub-regions in the overlapping area. The reprojection unit 1905 is specifically used to reproject the images of the first image frame and the second image frame except the overlapping area based on the global homography matrix, and to reproject the images of the multiple sub-regions based on the multiple local homography matrices that have been spatially smoothed.

[0335] As an example, combining Fig.15 , the smoothing unit 1906 can be used to execute S502, and the reprojection unit 1905 can be used to execute S503.

[0336] Optionally, the first video and the second video are videos shot by adjacent lenses of a panoramic camera. Alternatively, the first video and the second video are videos shot by two cameras with overlapping shooting fields in the target scene.

[0337] Optionally, the jitter of the above-mentioned images to be stitched includes at least one of the following: image jitter caused by the movement of a camera / lens used to shoot the images to be stitched, image jitter caused by the movement / movement of a target object when shooting the images to be stitched, and image jitter caused by changes in ambient lighting where the camera / lens is located when shooting the images to be stitched.

[0338] For the detailed description of the above optional methods, please refer to the above method embodiments, which will not be repeated here. In addition, the explanation and description of the beneficial effects of any of the above image processing devices 190 can refer to the above corresponding method embodiments, which will not be repeated here.

[0339] As an example, combining Figure 6 The functions implemented by the determination unit 1902, the splicing unit 1903, the reprojection unit 1905 and the smoothing unit 1906 in the image processing device 190 can be realized by Figure 6 Processor 601 in the Figure 6 The functions implemented by the acquisition unit 1901 can be implemented by the program code in the memory 602 in FIG. Figure 6 The functions implemented by the display unit 1904 can be realized by Figure 6 The input and output interface 605 is implemented in, but is certainly not limited to this.

[0340] Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0341] It should be noted that Fig.19 The division of modules in the example is schematic and is only a logical function division. There may be other division methods in actual implementation. For example, two or more functions may be integrated into one processing module. The above integrated modules may be implemented in the form of hardware or software function modules.

[0342] The embodiment of the present application also provides an image processing system, which includes an image processing device, a first image acquisition device, and a second image acquisition device. The first image acquisition device is used to acquire a first video, and the second image acquisition device is used to acquire a second video. The shooting field of the first video and the shooting field of the second video have an overlapping area. The image processing device is used to perform the following steps: Figure 7 , Fig.12 , Fig.13 , Fig.14 , Fig.15 , Fig.17 or Fig.18 The method shown is used to achieve the splicing of image frames in the first video and the second video.

[0343] The present application also provides a computer program product and a computer-readable storage medium for storing the computer program product. The computer program product may include one or more program instructions, which, when executed by one or more processors, can provide the above Figure 7 , Fig.12 , Fig.13 , Fig.14 , Fig.15 , Fig.17 or Fig.18 Thus, for example, reference to Figure 7 One or more features of S101 to S103 may be undertaken by one or more instructions in the computer program product.

[0344] In some examples, such as for executing Figure 7 , Fig.12 , Fig.13 , Fig.14 , Fig.15 , Fig.17 or Fig.18 The image processing apparatus of the method may be configured to provide various operations, functions, or actions in response to one or more program instructions stored in a computer-readable storage medium.

[0345] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When executing instructions on a computer and executing a computer, a process or function according to an embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that contains one or more servers that can be integrated with a medium. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0346] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: include: Acquire an image to be stitched, the image to be stitched comprising a first image frame in a first video and a second image frame in a second video, there is an overlapping area between the shooting field of view of the first video and the shooting field of view of the second video, the image to be stitched comprises a first sub-image and a second sub-image, the first sub-image is an image of the overlapping area in the first image frame, the second sub-image is an image of the overlapping area in the second image frame, and the first sub-image and the second sub-image include a target object; When the jitter amplitude of the image to be stitched is less than or equal to a preset threshold, determining a first stitching position, the first stitching position is located on the same side of the target object in the reference sub-image and the target object in the historical sub-image, the reference sub-image is the first sub-image or the second sub-image, the historical sub-image is an image of the overlapping area in the historical stitching image, and the historical stitching image is obtained by stitching a historical frame in the first video that is located before the first image frame and a historical frame in the second video that is located before the second image frame; The first image frame and the second image frame are spliced ​​according to the first stitching position to obtain a target stitching image.

2. The method according to claim 1, characterized in that The first sub-image and the second sub-image also include a background, and the background is an area in the image other than the target object; the method further includes: When the jitter amplitude of the image to be stitched is less than or equal to the preset threshold, determining a second stitching position, where the second stitching position is the same position in the reference sub-image as the stitching position of the target area in the historical sub-image, and the target area is the same area in the historical sub-image as the background position in the reference sub-image; The step of splicing the first image frame and the second image frame according to the first stitching position comprises: The first image frame and the second image frame are spliced ​​according to the first stitching position and the second stitching position.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: The target stitched image is displayed.

4. The method according to any one of claims 1 to 3, characterized in that The historical frames located before the first image frame include a preset number of image frames before the first image frame in the first video, and the historical frames located before the second image frame include a preset number of image frames before the second image frame in the second video, where the preset number is greater than or equal to 1.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Determining a jitter amplitude of the first image frame and a jitter amplitude of the second image frame; When the jitter amplitude of the first image frame is less than or equal to a first threshold, or the jitter amplitude of the second image frame is less than or equal to the first threshold, or the sum of the jitter amplitude of the first image frame and the jitter amplitude of the second image frame is less than or equal to a second threshold, it is determined that the jitter amplitude of the image to be spliced ​​is less than or equal to a preset threshold.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: A target object and a background in the first sub-image and the second sub-image are determined.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Determining a geometric transformation relationship between the first image frame and the second image frame; Reprojecting the first image frame and the second image frame according to the geometric transformation relationship; The step of splicing the first image frame and the second image frame according to the first stitching position comprises: The reprojected first image frame and the reprojected second image frame are spliced ​​according to the first seam position.

8. The method according to claim 7, characterized in that The geometric transformation relationship is represented by a target homography matrix, and the method further comprises: Performing time domain smoothing processing on the target homography matrix according to the first weight and the homography matrix of the historical frame, wherein the historical homography matrix is ​​a homography matrix for reprojecting the historical frame when obtaining the historical spliced ​​image; The reprojecting the first image frame and the second image frame according to the geometric transformation relationship comprises: The first image frame and the second image frame are reprojected based on the target homography matrix that has been smoothed in the time domain.

9. The method according to claim 7 or 8, characterized in that: The geometric transformation relationship is represented by a target homography matrix, and the target homography matrix includes a global homography matrix and a plurality of local homography matrices; the global homography matrix is ​​calculated based on the entire image of the first image frame and the entire image of the second image frame, the plurality of local homography matrices correspond to the plurality of sub-regions divided by the overlapping region, and a local homography matrix is ​​calculated based on an image of a sub-region in the overlapping region; the method further includes: According to the second weight and the global homography matrix, the local homography matrix is ​​subjected to spatial domain smoothing processing, or, according to the difference between adjacent homography matrices in the multiple local homography matrices, the local homography matrix is ​​subjected to spatial domain smoothing processing, and the sub-regions corresponding to the adjacent homography matrices are adjacent sub-regions in the overlapping region; The reprojecting the first image frame and the second image frame according to the geometric transformation relationship comprises: The images of the first image frame and the second image frame except the overlapping area are reprojected based on the global homography matrix, and the images of the multiple sub-areas are reprojected based on the multiple local homography matrices that have been smoothed in the spatial domain.

10. The method according to any one of claims 1 to 9, characterized in that The first video and the second video are videos shot by adjacent lenses in a panoramic camera; or, the first video and the second video are videos shot by two cameras with overlapping shooting fields in a target scene.

11. The method according to any one of claims 1 to 10, characterized in that The jitter of the image to be stitched includes at least one of the following: image jitter caused by the movement of the camera / lens used to shoot the image to be stitched, image jitter caused by the movement / movement of the target object when shooting the image to be stitched, and image jitter caused by changes in ambient lighting where the camera / lens is located when shooting the image to be stitched.

12. An image processing device, characterized in that: include: an acquisition unit, configured to acquire an image to be stitched, the image to be stitched comprising a first image frame in a first video and a second image frame in a second video, wherein there is an overlapping area between a shooting field of view of the first video and a shooting field of view of the second video, the image to be stitched comprising a first sub-image and a second sub-image, the first sub-image being an image of the overlapping area in the first image frame, the second sub-image being an image of the overlapping area in the second image frame, and the first sub-image and the second sub-image including a target object; a determining unit, configured to determine a first stitching position when the jitter amplitude of the image to be stitched is less than or equal to a preset threshold, the first stitching position being located on the same side of a target object in a reference sub-image and a target object in a history sub-image, the reference sub-image being the first sub-image or the second sub-image, the history sub-image being an image of the overlapping area in a history stitching image, and the history stitching image being obtained by stitching a history frame in the first video that is located before the first image frame and a history frame in the second video that is located before the second image frame; A stitching unit is used to stitch the first image frame and the second image frame according to the first stitching seam position to obtain a target stitching image.

13. The device according to claim 12, characterized in that The first sub-image and the second sub-image also include a background, and the background is an area in the image except the target object; The determining unit is further configured to determine a second stitching position when the jitter amplitude of the image to be stitched is less than or equal to the preset threshold, wherein the second stitching position is a position in the reference sub-image that is the same as the stitching position of the target area in the historical sub-image, and the target area is an area in the historical sub-image that is the same as the background position in the reference sub-image; The splicing unit is specifically configured to splice the first image frame and the second image frame according to the first splicing seam position and the second splicing seam position.

14. The device according to claim 12 or 13, characterized in that The device also includes: A display unit is used to display the target stitched image.

15. The device according to any one of claims 12 to 14, characterized in that The historical frames located before the first image frame include a preset number of image frames before the first image frame in the first video, and the historical frames located before the second image frame include a preset number of image frames before the second image frame in the second video, where the preset number is greater than or equal to 1.

16. The device according to any one of claims 12 to 15, characterized in that The determination unit is also used to determine the jitter amplitude of the first image frame and the jitter amplitude of the second image frame; and, when the jitter amplitude of the first image frame is less than or equal to a first threshold, or the jitter amplitude of the second image frame is less than or equal to the first threshold, or the sum of the jitter amplitude of the first image frame and the jitter amplitude of the second image frame is less than or equal to the second threshold, determine that the jitter amplitude of the image to be spliced ​​is less than or equal to a preset threshold.

17. The device according to any one of claims 12 to 16, characterized in that The determining unit is used to determine the target object and the background in the first sub-image and the second sub-image.

18. The device according to any one of claims 12 to 17, characterized in that The device also includes a reprojection unit: The determining unit is used to determine a geometric transformation relationship between the first image frame and the second image frame; The reprojection unit is used to reproject the first image frame and the second image frame according to the geometric transformation relationship; The stitching unit is specifically configured to stitch the reprojected first image frame and the reprojected second image frame according to the first stitching seam position.

19. The device according to claim 18, characterized in that The geometric transformation relationship is represented by a target homography matrix, and the device further comprises: A smoothing unit, configured to perform time domain smoothing processing on the target homography matrix according to a first weight and a homography matrix of a historical frame, wherein the historical homography matrix is ​​a homography matrix for reprojecting the historical frame when obtaining the historical spliced ​​image; The reprojection unit is specifically configured to reproject the first image frame and the second image frame based on the target homography matrix that has been smoothed in the time domain.

20. The device according to claim 18 or 19, characterized in that The geometric transformation relationship is represented by a target homography matrix, and the target homography matrix includes a global homography matrix and a plurality of local homography matrices; the global homography matrix is ​​calculated based on the entire image of the first image frame and the entire image of the second image frame, the plurality of local homography matrices correspond to the plurality of sub-regions divided by the overlapping region, and a local homography matrix is ​​calculated based on an image of a sub-region in the overlapping region; the device further includes: a smoothing unit, configured to perform spatial domain smoothing processing on the local homography matrix according to a second weight and the global homography matrix, or to perform spatial domain smoothing processing on the local homography matrix according to a difference between adjacent homography matrices in the multiple local homography matrices, wherein the sub-regions corresponding to the adjacent homography matrices are adjacent sub-regions in the overlapping region; The reprojection unit is specifically used to reproject the images of the first image frame and the second image frame except the overlapping area based on the global homography matrix, and to reproject the images of the multiple sub-areas based on the multiple local homography matrices that have been smoothed in the spatial domain.

21. The device according to any one of claims 12 to 20, characterized in that The first video and the second video are videos shot by adjacent lenses in a panoramic camera; or, the first video and the second video are videos shot by two cameras with overlapping shooting fields in a target scene.

22. The device according to any one of claims 12 to 21, characterized in that The jitter of the image to be stitched includes at least one of the following: image jitter caused by the movement of the camera / lens used to shoot the image to be stitched, image jitter caused by the movement / movement of the target object when shooting the image to be stitched, and image jitter caused by changes in ambient lighting where the camera / lens is located when shooting the image to be stitched.

23. An image processing device, characterized in that: include: One or more processors and a memory, wherein the one or more processors are configured to call program instructions stored in the memory to execute the method according to any one of claims 1 to 11.

24. An image processing system, characterized in that: The invention comprises an image processing device, a first image acquisition device and a second image acquisition device, wherein the first image acquisition device is used to acquire a first video, the second image acquisition device is used to acquire a second video, there is an overlapping area between the shooting field of view of the first video and the shooting field of view of the second video, and the image processing device is used to execute the method as described in any one of claims 1 to 11 to realize the splicing of image frames in the first video and the second video.

25. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes program instructions. When the program instructions are executed on a computer or a processor, the computer or the processor executes the method according to any one of claims 1 to 11.

Images