Video image splicing method, device, electronic device and storage medium
Through cross-lens tracking and optical flow method, the problem of low stitching accuracy of low feature video images is solved, and high-quality video stitching effect is achieved, which is suitable for open area surveillance videos.
Patent Information
- Application Number
- CN202411171595.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-26
AI Technical Summary
The existing video stitching technology has low accuracy when processing low feature images, especially in open area monitoring videos. The traditional feature point-based stitching method is difficult to work, and the timeline calibration is difficult when stitching multiple videos, resulting in target flash and lag. Or ghosting.
By tracking the moving target across the mirror, the target image set and detection frame in the video segment are determined, the splicing position is adjusted according to the size change value of the target detection frame, and the optical flow method is used to confirm the optimal adjustment position to achieve high-precision splicing of the video segment.
It improves the accuracy and accuracy of video stitching, reduces manual intervention, avoids target flash, lag or ghosting, and is suitable for video stitching in low-character environments.
Smart Images

Figure CN118691466B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing, and in particular to a video image stitching method, device, electronic device and storage medium. Background Art
[0002] With the widespread application of video surveillance technology, video stitching technology has become an important research direction in the field of image processing. This technology aims to fuse video images from multiple cameras to provide a more comprehensive and coherent monitoring perspective. This technology has important application value in public security, traffic monitoring, smart cities and other fields. At present, the technology for image stitching of video images is relatively mature, but in practical applications, especially when processing low-feature images (such as surveillance videos in open areas such as rivers and highways), it still faces many challenges.
[0003] Existing video stitching technology mainly relies on methods for calculating feature points in images. For example, the feature points in an image are calculated using the SIFT algorithm (Scale-Invariant Feature Transform), and these feature points are used for matching to achieve image stitching. In this process, commonly used matching algorithms include FLANN (Fast Library for Approximate Nearest Neighbors) in OpenCV (Open Source Computer Vision Library). These algorithms can achieve good stitching results when there are enough feature points.
[0004] However, in practical applications, the problem of stitching images with low features is particularly prominent. For example, in open areas such as rivers or roads, due to the lack of obvious feature points in the image, traditional stitching methods based on feature points often fail to work. In particular, when the number of feature points is lower than a certain threshold, the subsequent image matching and stitching process will be seriously affected, and even lead to stitching failure.
[0005] In addition, multi-channel video stitching also faces the problem of timeline calibration. Since videos shot by different cameras may have time deviations, direct stitching may cause object flashing, lag or ghosting. Traditional methods often require manual adjustment of the video timeline, but this method is not only time-consuming and labor-intensive, but also difficult to achieve accurate alignment.
[0006] Currently, no effective solution has been proposed for the problem of low accuracy in splicing video images, especially low-feature video images, in related technologies. Summary of the invention
[0007] Embodiments of the present application provide a video image stitching method, apparatus, system, electronic device and storage medium to at least solve the problem of low accuracy in the stitching of video images, especially low-feature video images, in the related art.
[0008] In a first aspect, an embodiment of the present application provides a video image stitching method, the method comprising:
[0009] Acquire a first video segment and a second video segment respectively captured by different camera devices;
[0010] Based on the same moving target tracked in the first video segment and the second video segment, determine a first target image set in the first video segment and a second target image set in the second video segment;
[0011] Acquire a first target detection frame of the moving target in the first target image set, and a second target detection frame of the moving target in the second target image set; determine an initial splicing position according to the first target detection frame and the second target detection frame;
[0012] An adjustment result of the initial splicing position is determined according to a size change value from the first object detection frame to the second object detection frame, and the first video segment and the second video segment are spliced based on the initial splicing position and the adjustment result.
[0013] In some embodiments, determining a first target image set in the first video segment and a second target image set in the second video segment includes:
[0014] Determine a plurality of first associated images in the first video segment based on the moving target, and extend the plurality of first associated images according to a preset first time period to obtain the first target image set;
[0015] A plurality of frames of second associated images in the second video segment are determined based on the moving target, and the plurality of frames of second associated images are extended according to a preset second time period to obtain the second target image set.
[0016] In some embodiments, obtaining a first target detection frame of the moving target in the first target image set and a second target detection frame of the moving target in the second target image set includes:
[0017] Record first target position information of the moving target according to the first target image set, and record second target position information of the moving target according to the second target image set;
[0018] Based on the first target position information, obtaining a first target detection frame of the moving target in the first target image set;
[0019] Based on the second target position information, a second target detection frame of the moving target in the second target image set is obtained.
[0020] In some embodiments, determining an initial stitching position according to the first object detection frame and the second object detection frame includes:
[0021] Calculate a first detection size of the moving object according to the first object detection frame, and calculate a second detection size of the moving object according to the second object detection frame;
[0022] An initial splicing position is determined according to the first detection size and the second detection size.
[0023] In some embodiments, determining the adjustment result of the initial stitching position according to the size change value from the first object detection frame to the second object detection frame includes:
[0024] Compare the first detected size with the second detected size to obtain a size change value;
[0025] If it is detected that the size change value is smaller than a preset change threshold, an adjustment result indicating that the initial splicing position is not adjusted is generated; otherwise, an adjustment result indicating that the initial splicing position is adjusted is generated.
[0026] In some embodiments, splicing the first video segment and the second video segment based on the initial splicing position and the adjustment result includes:
[0027] If the adjustment result is to adjust the initial splicing position, obtaining a first target change amount in the first target detection frame and a second target change amount in the second target detection frame based on an optical flow method;
[0028] determining an optimal adjustment position based on a difference between the first target change amount and the second target change amount;
[0029] splicing the first video segment and the second video segment based on the optimal adjustment position and the initial splicing position;
[0030] If the adjustment result is not to adjust the initial splicing position, the first video segment and the second video segment are spliced based on the initial splicing position.
[0031] In some embodiments, based on the optical flow method, obtaining the first target change amount in the first target detection frame and the second target change amount in the second target detection frame, and confirming the optimal adjustment position based on the difference between the first target change amount and the second target change amount, includes:
[0032] Selecting a plurality of first target detection points in the first target detection frame, calculating the plurality of first target detection points based on an optical flow method, and obtaining the first target variation, where the first target variation is a vector velocity of the plurality of first target detection points;
[0033] Selecting a plurality of second target detection points in the second target detection frame, calculating the plurality of second target detection points based on an optical flow method, and obtaining a second target variation, where the second target variation is a vector velocity of the plurality of second target detection points;
[0034] Based on the initial splicing position, a vector difference operation is performed on the vector velocities of the plurality of first target detection points and the vector velocities of the plurality of second target detection points to determine an optimal adjustment position.
[0035] In a second aspect, an embodiment of the present application provides a video image stitching device, including:
[0036] An acquisition module, used to acquire a first video segment and a second video segment respectively captured by different camera devices;
[0037] A cross-lens tracking module, configured to determine a first target image set in the first video segment and a second target image set in the second video segment based on the same moving target tracked in the first video segment and the second video segment;
[0038] an initial stitching module, configured to obtain a first target detection frame of the moving target in the first target image set and a second target detection frame of the moving target in the second target image set; and determine an initial stitching position according to the first target detection frame and the second target detection frame;
[0039] An adjustment and splicing module is used to determine an adjustment result of the initial splicing position according to a target transformation trend from the first target detection frame to the second target detection frame, and splice the first video segment and the second video segment based on the initial splicing position and the adjustment result.
[0040] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the video image stitching method described in the first aspect above is implemented.
[0041] In a fourth aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the video image stitching method as described in the first aspect above.
[0042] Compared with the related art, the video image stitching method provided in the embodiment of the present application solves the problem of low accuracy of video images, especially low-feature video image stitching, by tracking moving targets across lenses and stitching video segments according to corresponding parameters, thereby achieving higher quality video stitching effects and improving the accuracy of video stitching.
[0043] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0045] Figure 1 is a hardware structure block diagram of a terminal of the video image splicing method according to an embodiment of the present invention;
[0046] Figure 2 is an operation flow chart of a video image stitching method according to an embodiment of the present application;
[0047] Figure 3 is a schematic diagram of a video image stitching method according to an embodiment of the present invention;
[0048] Figure 4 is a schematic diagram of another video image stitching method according to an embodiment of the present invention;
[0049] Figure 5 is a schematic diagram of another video image stitching method according to an embodiment of the present invention;
[0050] Figure 6 It is a structural block diagram of a video image stitching device according to an embodiment of the present application. DETAILED DESCRIPTION
[0051] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the contents disclosed in the present application, some changes such as design, manufacturing or production based on the technical contents disclosed in the present application are only conventional technical means, and should not be understood as insufficient contents disclosed in the present application.
[0052] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.
[0053] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantity limitation, and may indicate the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application means greater than or equal to two. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The terms "first", "second", "third" and the like involved in the present application are merely used to distinguish similar objects and do not represent a specific ordering of the objects.
[0054] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. Taking running on a terminal as an example, Figure 1 FIG. 1 is a hardware structure diagram of a terminal of the video image splicing method according to an embodiment of the present invention. Figure 1 As shown, the terminal 10 may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.
[0055] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the video image stitching method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0056] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by the communication provider of the terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0057] This embodiment provides a video image stitching method. Figure 2 is a flow chart of a video image stitching method according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:
[0058] Step S201: Acquire a first video segment and a second video segment respectively captured by different camera devices.
[0059] The first video segment and the second video segment of the present invention do not specifically refer to the splicing of two video segments, but can be the splicing of multiple videos. In this step, the video segments shot by multiple cameras can be connected to the video image splicing system or device through the video platform, and the video shot by each camera is cached, usually caching a video segment for a period of time (such as about 10 seconds) for subsequent processing. The cached video segment set is respectively used as the first video segment and the second video segment, etc.; then the cached video segment is decoded and converted into a format that can be processed by the image processing algorithm. For example, the video decoded image group of the first video segment has a total of n frames: (F A1 ,F A2 ,F A3 ,...,F An ), the video decoded picture group n frames of the second video segment (F B1 ,F B2 ,F B3 ,...,F Bn ). This step obtains multiple video segments from different cameras, which are located at different viewing angles or positions, capturing videos of different areas, covering different parts of the scene to be stitched, ensuring that there are multiple video segments from different sources that can be processed later.
[0060] Step S202: determining a first target image set in the first video segment and a second target image set in the second video segment based on the same moving target tracked in the first video segment and the second video segment.
[0061] The target tracking algorithm is used to find the same moving target that appears in the first video segment and the second video segment, and all image frames containing the moving target are extracted from the first video segment to form a first target image set. Similarly, all image frames containing the same moving target are extracted from the second video segment to form a second target image set, and the frame number A of the specific time interval in which the target appears and disappears from the camera is determined. tk To A tl ,B ti To B tj This step is the key to achieving accurate video stitching. By tracking the movement of the target object and establishing connections in different video segments, the stitching problem caused by insufficient number of feature points in traditional image stitching methods is overcome, making the video image stitching method of the present invention suitable for open areas and low-feature environments.
[0062] Step S203, obtaining a first target detection frame of the moving target in the first target image set and a second target detection frame of the moving target in the second target image set; and determining an initial splicing position according to the first target detection frame and the second target detection frame.
[0063] Among them, the detection frames of the moving targets are extracted from the first target image set and the second target image set respectively, such as Figure 3 As shown in FIG. 1 , these detection frames are usually rectangular frames that mark the position and size of the target object in the image. The target detection frame is used to extract the accurate position and size information of the target. By comparing the detection frames of the same moving target in two video segments, the target size changes evenly. Specifically, the first target image set (A tk ,......,A tl ) contains A tk To A tl Frame, the second target image set (B ti ,......,B tj ) contains B ti To B tj Frame, you can select A tk To A tl One of the frames, frame tu, is used as the reference frame, and B is calculated based on the detection frame. ti To B tj The target size of the frame and the tu frame is adjusted based on the position and size information to make the target size change evenly. At this time, the splicing position of the two video segments is preliminarily determined, that is, A tu =B tv , here B tv Frame is the second target image set and the reference frame A tu A frame of preliminary alignment. This step determines the initial splicing position through the target detection frame, preliminarily aligns the time axis and spatial position of the video segment, improves the accuracy of the splicing position, and reduces the splicing error caused by insufficient feature points.
[0064] Step S204: determining an adjustment result of the initial splicing position according to a size change value from the first target detection frame to the second target detection frame, and splicing the first video segment and the second video segment based on the initial splicing position and the adjustment result.
[0065] Among them, the size change value (such as width change, height change, etc.) from the first target detection frame to the second target detection frame is used to determine the size change of the moving target in the two video segments. These size changes may be caused by factors such as different viewing angles, distances, and focal lengths of each camera. According to the size change value of the target detection frame, it is confirmed whether the initial splicing position needs further adjustment, that is, the adjustment result. If further adjustment is required, the initial splicing position is adjusted and then the first video segment and the second video segment are spliced. If not, the first video segment and the second video segment are directly spliced. This step determines whether the splicing position needs to be further fine-tuned to ensure the continuity and smoothness of the splicing, realizes automated video splicing and calibration, reduces the need for manual intervention, and improves the efficiency and stability of splicing.
[0066] Through the above steps, the present invention uses cross-camera target tracking technology and a target recognition method to perform ID matching according to the feature value of the target, obtains the time and position information of the same target appearing in different cameras, effectively identifies and tracks targets moving across cameras, and provides accurate time and space information for subsequent image stitching, effectively solves the problems of low-feature image stitching and video timeline calibration, and realizes automated and high-precision video stitching effects. By calculating the size change value of the moving target and the correlation between adjacent frames, the corresponding relationship between frames is found, thereby calculating the motion information of the object, adjusting the stitching position during the video stitching process, and ensuring the accuracy and smoothness of the stitching.
[0067] In some embodiments, determining the first target image set in the first video segment and the second target image set in the second video segment in step S202 further includes:
[0068] Determine a plurality of first associated images in a first video segment based on a moving target, and extend the plurality of first associated images according to a preset first time period to obtain a first target image set;
[0069] A plurality of frames of second associated images in the second video segment are determined based on the moving target, and the plurality of frames of second associated images are extended according to a preset second time period to obtain a second target image set.
[0070] Among them, the first associated image A of the determined time interval in which the target appears and disappears in the first video segment is determined. tk To A tl , according to the preset first time period, the associated frame A tk To A tl Extending a part forward or backward, the first time period is set in advance by the staff, so as to obtain the first target image set, and the second target image set can also be obtained. The reason for extending the associated frames is that when the target just appears in the new camera, it contains less information and sometimes may not be recognized as the same target by the target tracking. Therefore, compared with the method of only obtaining the frame where the moving target is located to make an image set, extending the associated images forward and / or backward can avoid the failure of tracking some targets when the newly set camera captures less information, which easily affects the accuracy of stitching.
[0071] In some embodiments, obtaining a first target detection frame of the moving target in the first target image set and a second target detection frame of the moving target in the second target image set in step S203 further includes:
[0072] Record first target position information of the moving target according to the first target image set, and record second target position information of the moving target according to the second target image set;
[0073] Based on the first target position information, a first target detection frame of the moving target in the first target image set is obtained;
[0074] Based on the second target position information, a second target detection frame of the moving target in the second target image set is obtained.
[0075] Among them, in the first target image set A tk To A tl And the second target image set B ti To B tj The position information of the moving target is recorded in each frame, for example, the first target position information (X At1 ,Y At1 ), where X, Y contains the upper left and lower right coordinates, and X contains (X lefttop , X rightbot ), Y contains (Y lefttop , Y rightbot ), and also record the second target position information (X Bt1 ,Y Bt1 ), and then obtain the first target detection frame and the second target detection frame covering the moving target according to the position information, so that subsequent operations can extract the accurate position of the moving target and estimate the size information of the moving target based on the target detection frame. This step is the basis for determining the initial splicing position.
[0076] In some embodiments, determining the initial stitching position according to the first target detection frame and the second target detection frame in step S203 further includes:
[0077] Calculate a first detection size of the moving target according to the first target detection frame, and calculate a second detection size of the moving target according to the second target detection frame;
[0078] An initial splicing position is determined according to the first detection size and the second detection size.
[0079] Among them, by calculating the size of the first target detection frame, the size of the moving target is estimated to obtain the first detection size, and the second detection size is obtained in the same way. Then, the first detection size and the second detection size are compared to determine the initial splicing position. Specifically, A can be selected tk To A tl The tuth frame in the frame is used as the reference frame, and the second detection size (B ti To B tj The target size of the first frame) and the first detection size (selecting the target size of the tu frame), adjust the position of the image splicing based on the position information and size information so that the target size changes evenly. At this time, the splicing position of the two video segments is preliminarily determined, that is, A tu =B tv , here Btv Frame is B ti To B tj This step determines the initial splicing position by comparing the first detection size and the second detection size, preliminarily aligns the time axis and spatial position of the video segments, and realizes the rapid determination of the initial splicing position between the video segments, thereby reducing the splicing error caused by insufficient feature points, thereby improving the accuracy of the splicing position.
[0080] In some embodiments, determining the adjustment result of the initial stitching position according to the size change value from the first target detection frame to the second target detection frame in step S204 includes:
[0081] S241, comparing the first detected size with the second detected size to obtain a size change value;
[0082] S242: If it is detected that the size change value is less than a preset change threshold, an adjustment result indicating that the initial splicing position is not adjusted is generated; otherwise, an adjustment result indicating that the initial splicing position is adjusted is generated.
[0083] Among them, by calculating the first detection size and the second detection size, the size change value is obtained to determine whether the initial splicing position is accurate enough. If the size change value is less than the preset change threshold, it means that the initial splicing position can make the target size change evenly after the video segments are spliced, and there is no need to further accurately adjust the initial splicing position; if the size change value is not less than the preset change threshold, the target size changes unevenly after the video segments are spliced according to the initial splicing position. At this time, the splicing effect is not good, so the initial splicing position needs to be further accurately adjusted. By judging whether the initial splicing position is accurate enough through the size change value, automatic video splicing calibration is realized, so that the spliced video avoids problems such as target flashing, lag or ghosting, and improves the splicing effect.
[0084] In some embodiments, step S204 includes stitching the first video segment and the second video segment based on the initial stitching position and the adjustment result, including:
[0085] If the adjustment result is to adjust the initial splicing position, obtaining a first target change amount in the first target detection frame and a second target change amount in the second target detection frame based on an optical flow method;
[0086] Determining the best adjustment position based on the difference between the first target change amount and the second target change amount;
[0087] The first video segment and the second video segment are spliced based on the optimal adjustment position and the initial splicing position.
[0088] Among them, the motion information of the moving target between consecutive frames is calculated by the optical flow method, and the initial splicing position is further adjusted. The first target change is the change of the moving target between consecutive frames, which can be a speed change. The second target change is also obtained. The first target change is subtracted from the second target change to confirm the best adjustment position, thereby ensuring the consistency of the size of the moving target and the smoothness of the change direction after video splicing. The initial splicing position is accurately adjusted according to the best adjustment position, and finally the first video segment and the second video segment are spliced together. This step uses the optical flow method to quantify the motion information of the moving target between consecutive frames to confirm the best adjustment position, thereby achieving high-precision splicing of video segments and completely avoiding the problem of low accuracy in splicing low-feature video images.
[0089] If the adjustment result is not to adjust the initial splicing position, the first video segment and the second video segment are spliced based on the initial splicing position.
[0090] Among them, the adjustment result is not to adjust the initial splicing position, which means that after the video segments are spliced according to the initial splicing position, the size of the moving targets therein changes evenly, ensuring the continuity of the moving targets and a smooth splicing effect. At this time, the first video segment and the second video segment are directly spliced.
[0091] In some embodiments, based on the optical flow method, obtaining the first target change amount in the first target detection frame and the second target change amount in the second target detection frame; and confirming the optimal adjustment position based on the difference between the first target change amount and the second target change amount, including:
[0092] S241-1, selecting a plurality of first target detection points in the first target detection frame, calculating the plurality of first target detection points based on an optical flow method, and obtaining a first target variation, where the first target variation is a vector velocity of the plurality of first target detection points;
[0093] S241-2, selecting a plurality of second target detection points in the second target detection frame, calculating the plurality of second target detection points based on an optical flow method, and obtaining a second target variation, where the second target variation is a vector velocity of the plurality of second target detection points;
[0094] The first target detection point is the key corner point of the moving target. Multiple points can be selected in the first target detection frame to ensure the calculation accuracy. The first target change is the vector speed of each target detection point between consecutive frames, including the speed magnitude and direction, that is, (V1,...,V z ),like Figure 4The vector velocity of one of the target detection points is marked in the figure; similarly, the vector velocity of each target detection point between consecutive frames is calculated through the second target detection point. If the direction and magnitude of the target velocity vector change greatly in two consecutive frames, there is a deviation in the splicing position. By analyzing the change of the velocity vector, the consistency of the velocity magnitude and the smoothness of the direction are ensured to determine whether there is a deviation in the splicing position.
[0095] S241-3, based on the initial splicing position, perform vector difference calculation on the vector velocities of the plurality of first target detection points and the vector velocities of the plurality of second target detection points to confirm the best adjustment position.
[0096] Among them, Figure 5 Medium V ti To V ti+1 The vector difference between the two vector velocities is V ti,ti+1 , calculate the speed between Vti and Vtj (V ti,ti+1 , V ti+1,ti+2 ,......,V tj-1,tj ) The average value of the speed difference is calculated, and the best adjustment position is obtained with the tu frame as the benchmark. In addition, the scalar sum of the current speed difference |V| is recorded as the total error value to evaluate the stitching quality; with the frames before and after the tu frame as the benchmark, the corresponding adjustment steps are repeated to obtain a series of speed difference scalar sums (|V| tu ,|V| tu+a ,......,|V| tu+b ), select the scalar of the speed difference and the corresponding time T when it is minimum A =T B (to accurately align the time axis of the first video segment with the second) and spatial position LocA=LocB (to accurately align the first target position information with the second target position information) as the final time axis and spatial position alignment method, these parameters are applied to real-time stitching to complete the stitching of the first video segment and the second video segment. This step achieves seamless stitching between video segments by finely adjusting the stitching position, solves the stitching problem caused by device time axis deviation or target motion, and effectively solves the technical problems of low-feature image stitching and accurate calibration of video time axis and spatial position, improves the precision and accuracy of video stitching, reduces problems such as target flashing, lag or ghosting after stitching, and improves the overall quality of the stitching effect.
[0097] The embodiments of the present application are described and illustrated below through preferred embodiments.
[0098] First, the video platform caches the video segments (for example, about 10 seconds) of the multi-channel camera images that need to be spliced. The video decoding image group of camera A has a total of n frames: (F A1 ,F A2 ,FA3 ,...,F An ), Camera B:(F B1 ,F B2 ,F B3 ,...,F Bn ), connected to the subsequent stitching module, which mainly uses the image model to extract the information of the scene.
[0099] (1) Use the target tracking algorithm to determine the relevant frames (F) where a moving target with the same ID appears in two cameras At1 ,F At2 ,...,F Atn ) and (F Bk1 ,F Bk2 ,...,F Bkm ), where a portion of time will be extended forward or backward, while retaining the position information of the target in each frame image (X FAt1 ,Y FAt1 ), where X, Y are tuples (X lefttop , X rightbot ) contains the upper left and lower right coordinates.
[0100] (2) Based on the two sets of coordinate information, determine the frame number A of the time interval in which the target appears and disappears from the camera. tk To A tl ,B ti To B tj .
[0101] (3) Select the tu frame of camera A as the initial reference and calculate the B of camera B ti To B tj The target size is calculated by using the frame and tu frame and the subsequent target position, and the position of the image splicing is adjusted to make the target size change evenly. At this time, the time axis is preliminarily aligned and A tu =B tv .
[0102] (4) Since the detection frame of the target detection has a certain error, it is biased to simply locate the position based on the size of the target detection frame. Using optical flow calculation, the speed and direction of each corner point (V1,...,V z ), to ensure the consistency of velocity magnitude and smoothness of direction. If the direction and magnitude of the target velocity vector change greatly in two consecutive frames of images, there will be deviation in the splicing position.
[0103] (5) The direction of the vector difference between the two velocity vectors is the direction of the splicing adjustment, and the speed between ti and tj is calculated (V ti,ti+1 ,...,V tj-1,tj) The average value of the speed difference is obtained, and the splicing position is adjusted according to the speed difference to obtain the best splicing position based on the tu frame. The scalar and |V| of the current speed difference are recorded as the total error value.
[0104] (6) Repeat steps 3, 4, and 5 using the frames before and after the tu frame as the benchmark to obtain a series of scalar sums of velocity differences (|V| tu ,|V| tu+a , ..., |V| tu+b ), select the scalar of the speed difference and the minimum time T A =T B , LocA=LocB is used as the final alignment of the time axis and spatial position, and the output is added to the real-time stitching as a stitching parameter.
[0105] The present embodiment also provides a video image splicing device, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the terms "module", "unit", "subunit" and the like can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0106] Figure 6 is a structural block diagram of a video image splicing device according to an embodiment of the present application, such as Figure 6 As shown, the device comprises:
[0107] An acquisition module 10 is used to acquire a first video segment and a second video segment respectively captured by different camera devices;
[0108] A cross-lens tracking module 20, configured to determine a first target image set in the first video segment and a second target image set in the second video segment based on the same moving target tracked in the first video segment and the second video segment;
[0109] An initial stitching module 30 is used to obtain a first target detection frame of the moving target in the first target image set and a second target detection frame of the moving target in the second target image set; and determine an initial stitching position according to the first target detection frame and the second target detection frame;
[0110] The adjustment and splicing module 40 is used to determine the adjustment result of the initial splicing position according to the target transformation trend from the first target detection frame to the second target detection frame, and splice the first video segment and the second video segment based on the initial splicing position and the adjustment result.
[0111] The cross-lens tracking module 20 is further used to determine a plurality of first associated images in the first video segment based on the moving target, and extend the plurality of first associated images according to a preset first time period to obtain a first target image set;
[0112] A plurality of frames of second associated images in a second video segment are determined based on the moving target, and the plurality of frames of second associated images are extended according to a preset second time period to obtain a second target image set.
[0113] The initial stitching module 30 is further used to record the first target position information of the moving target according to the first target image set, and to record the second target position information of the moving target according to the second target image set;
[0114] Based on the first target position information, a first target detection frame of the moving target in the first target image set is obtained;
[0115] Based on the second target position information, a second target detection frame of the moving target in the second target image set is obtained.
[0116] The initial stitching module 30 is further used to calculate a first detection size of the moving target according to the first target detection frame, and calculate a second detection size of the moving target according to the second target detection frame;
[0117] An initial splicing position is determined according to the first detection size and the second detection size.
[0118] The adjustment and splicing module 40 is further used to compare the first detection size with the second detection size to obtain a size change value;
[0119] If it is detected that the size change value is less than a preset change threshold, an adjustment result indicating that the initial splicing position is not adjusted is generated; otherwise, an adjustment result indicating that the initial splicing position is adjusted is generated.
[0120] The adjustment and splicing module 40 is further configured to obtain a first target change amount in the first target detection frame and a second target change amount in the second target detection frame based on an optical flow method if the adjustment result is to adjust the initial splicing position;
[0121] Determining the best adjustment direction based on the difference between the first target change amount and the second target change amount;
[0122] splicing the first video segment and the second video segment based on the optimal adjustment direction and the initial splicing position;
[0123] If the adjustment result is not to adjust the initial splicing position, the first video segment and the second video segment are spliced based on the initial splicing position.
[0124] The adjustment and splicing module 40 is further used to select a plurality of first target detection points in the first target detection frame, calculate the plurality of first target detection points based on the optical flow method, and obtain a first target variation, where the first target variation is a vector velocity of the plurality of first target detection points;
[0125] Select multiple second target detection points in the second target detection frame, calculate the multiple second target detection points based on the optical flow method, and obtain the second target change amount, where the second target change amount is the vector speed of the multiple second target detection points;
[0126] Based on the initial splicing position, a vector difference operation is performed on the vector velocities of the plurality of first target detection points and the vector velocities of the plurality of second target detection points to determine the best adjustment direction.
[0127] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.
[0128] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0129] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.
[0130] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:
[0131] S1, obtaining a first video segment and a second video segment respectively captured by different camera devices.
[0132] S2: Determine a first target image set in the first video segment and a second target image set in the second video segment based on the same moving target tracked in the first video segment and the second video segment.
[0133] S3, obtaining a first target detection frame of the moving target in the first target image set and a second target detection frame of the moving target in the second target image set; and determining an initial splicing position according to the first target detection frame and the second target detection frame.
[0134] S4, determining an adjustment result of the initial splicing position according to a target transformation trend from the first target detection frame to the second target detection frame, and splicing the first video segment and the second video segment based on the initial splicing position and the adjustment result.
[0135] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.
[0136] In addition, in combination with the video image stitching method in the above embodiment, the embodiment of the present application can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, any one of the video image stitching methods in the above embodiment is implemented.
[0137] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0138] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A video image stitching method, characterized in that: The method comprises: Acquire a first video segment and a second video segment respectively captured by different camera devices; Based on the same moving target tracked in the first video segment and the second video segment, determine a first target image set in the first video segment and a second target image set in the second video segment; Acquire a first target detection frame of the moving target in the first target image set, and a second target detection frame of the moving target in the second target image set, wherein the first target detection frame and the second target detection frame are used to obtain position information and size information of the moving target; Calculate a first detection size of the moving object according to the first object detection frame, and calculate a second detection size of the moving object according to the second object detection frame; Determining an initial splicing position according to the first detection size and the second detection size; Compare the first detected size with the second detected size to obtain a size change value; If it is detected that the size change value is less than a preset change threshold, an adjustment result indicating that the initial splicing position is not adjusted is generated; otherwise, an adjustment result indicating that the initial splicing position is adjusted is generated; Based on the initial splicing position and the adjustment result, splicing the first video segment and the second video segment includes: If the adjustment result is to adjust the initial splicing position, obtaining a first target change amount in the first target detection frame and a second target change amount in the second target detection frame based on an optical flow method; determining an optimal adjustment position based on a difference between the first target change amount and the second target change amount; splicing the first video segment and the second video segment based on the optimal adjustment position and the initial splicing position; If the adjustment result is not to adjust the initial splicing position, the first video segment and the second video segment are spliced based on the initial splicing position.
2. The video image stitching method according to claim 1, characterized in that: The determining of a first target image set in the first video segment and a second target image set in the second video segment comprises: Determine a plurality of first associated images in the first video segment based on the moving target, and extend the plurality of first associated images according to a preset first time period to obtain the first target image set; A plurality of frames of second associated images in the second video segment are determined based on the moving target, and the plurality of frames of second associated images are extended according to a preset second time period to obtain the second target image set.
3. The video image stitching method according to claim 1, characterized in that: The acquiring of a first target detection frame of the moving target in the first target image set and a second target detection frame of the moving target in the second target image set comprises: Record first target position information of the moving target according to the first target image set, and record second target position information of the moving target according to the second target image set; Based on the first target position information, obtaining a first target detection frame of the moving target in the first target image set; Based on the second target position information, a second target detection frame of the moving target in the second target image set is obtained.
4. The video image stitching method according to claim 1, characterized in that: The method of obtaining a first target change amount in the first target detection frame and a second target change amount in the second target detection frame based on the optical flow method, and confirming an optimal adjustment position based on a difference between the first target change amount and the second target change amount, comprises: Selecting a plurality of first target detection points in the first target detection frame, calculating the plurality of first target detection points based on an optical flow method, and obtaining the first target variation, where the first target variation is a vector velocity of the plurality of first target detection points; Selecting a plurality of second target detection points in the second target detection frame, calculating the plurality of second target detection points based on an optical flow method, and obtaining a second target variation, where the second target variation is a vector velocity of the plurality of second target detection points; Based on the initial splicing position, a vector difference operation is performed on the vector velocities of the plurality of first target detection points and the vector velocities of the plurality of second target detection points to determine an optimal adjustment position.
5. A video image splicing device, characterized in that: include: An acquisition module, used to acquire a first video segment and a second video segment respectively captured by different camera devices; A cross-lens tracking module, configured to determine a first target image set in the first video segment and a second target image set in the second video segment based on the same moving target tracked in the first video segment and the second video segment; an initial stitching module, used to obtain a first target detection frame of the moving target in the first target image set, and a second target detection frame of the moving target in the second target image set, wherein the first target detection frame and the second target detection frame are used to obtain position information and size information of the moving target; The initial stitching module is further used to calculate a first detection size of the moving object according to the first object detection frame, and calculate a second detection size of the moving object according to the second object detection frame; The initial splicing module is further used to determine an initial splicing position according to the first detection size and the second detection size; An adjustment splicing module is used to compare the first detection size with the second detection size to obtain a size change value; The adjustment and splicing module is further configured to generate an adjustment result indicating that the initial splicing position should not be adjusted if it is detected that the size change value is less than a preset change threshold; otherwise, generate an adjustment result indicating that the initial splicing position should be adjusted; The adjustment and splicing module is further used to splice the first video segment and the second video segment based on the initial splicing position and the adjustment result; The adjustment and splicing module is further configured to obtain, based on an optical flow method, a first target change amount in the first target detection frame and a second target change amount in the second target detection frame if the adjustment result is to adjust the initial splicing position; The adjustment and splicing module is further used to determine the best adjustment position based on the difference between the first target change amount and the second target change amount; The adjustment and splicing module is further used to splice the first video segment and the second video segment based on the optimal adjustment position and the initial splicing position; The adjustment and splicing module is further configured to splice the first video segment and the second video segment based on the initial splicing position if the adjustment result is not to adjust the initial splicing position.
6. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the video image stitching method according to any one of claims 1 to 4.
7. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the video image stitching method according to any one of claims 1 to 4 when running.
