Digital refocusing algorithm and system applied to multi-focal matrix video stitching

Abstract

The invention relates to a digital refocusing algorithm and system applied to multi-focal matrix video stitching, and the algorithm comprises the steps: determining a focusing region according to a specified refocusing position, and creating a plurality of focusing planes in the focusing region, and the plurality of focusing planes are arranged in parallel; acquiring light field data associated with the focusing plane and generating a base image belonging to the focusing plane by using the light field data; determining a display object on the basic image and calculating a clear area and a fuzzy area of the display object; and acquiring a focusing plane associated with the clear area and the blurred area, recording the focusing plane as a reference focusing plane, and replacing the blurred area on the basic image with a corresponding clear area belonging to the basic image on the reference focusing plane to obtain an accurate display image. According to the digital refocusing algorithm and system applied to multi-focal matrix video stitching disclosed by the invention, the unclear content of the refocusing plane is repaired in a range content acquisition mode, so that an image meeting the definition requirement is obtained.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a digital refocusing algorithm and system applied to multi-focal matrix video splicing. Background Art

[0002] Due to the size and complexity of substations, traditional video surveillance systems may not be able to provide comprehensive coverage and high-quality images, mainly in the following aspects: Due to the limitation of the existing camera imaging technology principle, even if dozens of cameras are deployed, it is impossible to take into account the field of view of the entire station area and high-definition monitoring of the entire station area in the substation. At the same time, due to the limitations of installation location and installation issues, there are many monitoring blind spots; The chaotic structure of multiple monitoring screens and the weak correlation between the screens make it impossible to fully understand the operation status of the entire station through the monitoring screens. The monitoring contents of each monitoring screen are highly similar, making it difficult to quickly confirm the monitoring target; The splicing of surveillance images achieved by horizontally arranged multiple cameras results in serious distortion of the surveillance images, blurred details, and fragmented images. It is difficult to obtain important detailed information on the areas on both sides, and it is impossible to intuitively display the real surveillance scene.

[0003] Light field imaging technology is a method that can collect information from multiple points at a single point. It acquires data through cameras deployed in an array and uses sub-sampling technology to achieve high data compression and real-time transmission. This method can easily achieve coverage monitoring and multi-screen fusion. At the same time, with the help of adjustable focal length, it can also achieve rapid adjustment of range coverage and key monitoring areas.

[0004] However, due to the use of sub-sampling technology, a large amount of information is lost. When the picture is refocused, part of the picture will appear blurred, resulting in insufficient clarity of the image display content. Summary of the invention

[0005] The present invention provides a digital refocusing algorithm and system for multi-focus matrix video stitching, which repairs unclear content of a refocusing plane by acquiring range content to obtain an image that meets clarity requirements.

[0006] The above object of the present invention is achieved through the following technical solutions: In a first aspect, the present invention provides a digital refocusing algorithm for multi-focus matrix video stitching, comprising: Determine a focus area according to a specified refocusing position and create a plurality of focus planes inside the focus area, wherein the plurality of focus planes are arranged in parallel; Acquire light field data associated with the focal plane and generate a basis image attributable to the focal plane using the light field data; Determine the display object on the base image and calculate the clear area and the blurred area of ​​the display object; Obtaining a focus plane associated with the clear area and the blurred area, and recording it as a reference focus plane; The blurred areas on the base image are replaced with the corresponding clear areas belonging to the base image on the reference focus plane to obtain an accurate displayed image.

[0007] In a possible implementation manner of the first aspect, calculating a clear area and a fuzzy area of ​​a display object includes: Get the boundary outline of the display object; A reduced boundary contour is established inside the display object based on the boundary contour, and the coverage area of ​​the reduced boundary contour tends to decrease in the direction close to the center of the display object; Calculate the clarity of the content between two adjacent boundary contours to obtain a clarity calculation result; Use the sharpness calculation results to divide the displayed object into sharp areas and blurred areas.

[0008] In a possible implementation manner of the first aspect, calculating the clarity of content between two adjacent boundary contours includes: Creates a reference baseline on the content between two adjacent border outlines; Decompose the reference baseline using wavelet and calculate the high-frequency basis quantity and high-frequency basis quantity area of ​​the reference baseline; Divide the reference baseline according to the high-frequency base volume area to obtain the sub-construction reference baseline; Transfer the sub-construction reference baseline into the frequency domain to decompose and calculate the proportion of high-frequency basis in the sub-construction reference baseline; The clarity result of the content between two adjacent boundary contours is given according to the proportion of high-frequency basis in the sub-construction reference baseline.

[0009] In a possible implementation manner of the first aspect, replacing the blurred area on the base image with a corresponding clear area belonging to the base image on the reference focus plane includes: Adjust the reference focus plane to coincide with the focus plane of the base image; According to the outline of the blurred area on the basic image, the basic image belonging to the reference focus plane is cut to obtain a replacement supplementary image; Replace the blurred area on the base image with the replacement complementary image; The step of adjusting the reference focus plane to coincide with the focus plane where the basic image is located includes linear adjustment and spatial movement adjustment.

[0010] In a possible implementation manner of the first aspect, the method further includes fusing and replacing the supplementary image with the base image, wherein the fusing and replacing the supplementary image with the base image includes: Establishing a fusion reference curve on the base image and the replacement supplementary image, wherein the first part of the reference curve is located on the base image, and the second part is located on the replacement supplementary image; Decomposing the first part of the reference curve and the second part of the reference curve respectively by using wavelet to obtain two decomposition curve groups; Match the two decomposition curve groups and adjust the complementary image based on the matching results.

[0011] In a possible implementation manner of the first aspect, matching two decomposition curve groups includes: Determine a curve in the first decomposition curve group whose cutoff point is located at the junction of the first part of the reference curve and the second part of the reference curve, which is recorded as the first reference curve group; Determine a curve in the second decomposition curve group whose starting point is located at the junction of the first part of the reference curve and the second part of the reference curve, and record it as the second reference curve group; Matching is performed on the first reference curve group and the second reference curve group to obtain a matching result.

[0012] In a possible implementation manner of the first aspect, adjusting the supplementary image according to the matching result includes: Obtaining an adjustment parameter group according to the matching result, the adjustment parameter group including an adjustment amount; The curve of the complementary image is adjusted using the adjustment parameter group so that the curve of the complementary image is parallel to the reference curve.

[0013] In a second aspect, the present invention provides a digital refocusing device for multi-focus matrix video stitching, comprising: A focusing plane creating unit, used to determine a focusing area according to a specified refocusing position and create a plurality of focusing planes inside the focusing area, wherein the plurality of focusing planes are arranged in parallel; A basic image generation unit, configured to obtain light field data associated with the focus plane and generate a basic image belonging to the focus plane using the light field data; A basic image replacement unit, used for determining a display object on the basic image and calculating a clear area and a blurred area of ​​the display object; A focus plane acquisition unit, used to acquire a focus plane associated with the clear area and the blurred area, recorded as a reference focus plane; The content replacement unit is used to replace the blurred area on the basic image with the corresponding clear area belonging to the basic image on the reference focus plane to obtain an accurate display image.

[0014] In a third aspect, the present invention provides a digital refocusing system for multi-focal matrix video stitching, the system comprising: one or more memories for storing instructions; and One or more processors, used to call and run the instructions from the memory to execute the method as described in the first aspect and any possible implementation of the first aspect.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium, the computer-readable storage medium comprising: Program, when the program is executed by a processor, the method described in the first aspect and any possible implementation of the first aspect is executed.

[0016] In a fifth aspect, the present invention provides a computer program product, comprising program instructions, and when the program instructions are executed by a computing device, the method described in the first aspect and any possible implementation of the first aspect is executed.

[0017] In a sixth aspect, the present invention provides a chip system comprising a processor for implementing the functions involved in the above aspects, for example, generating, receiving, sending, or processing the data and / or information involved in the above methods.

[0018] The chip system may be composed of chips, or may include chips and other discrete devices.

[0019] In a possible design, the chip system also includes a memory, which is used to store necessary program instructions and data. The processor and the memory can be decoupled and respectively set on different devices, connected by wired or wireless means, or the processor and the memory can also be coupled on the same device. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram that contains both position and direction information in space.

[0021] Figure 2 It is a schematic diagram showing that different focal lengths result in different fields of view.

[0022] Figure 3 The present invention provides a schematic block diagram of the steps of a digital refocusing algorithm for multi-focal matrix video stitching.

[0023] Figure 4 It is a schematic diagram of the principle of subsampling provided by the present invention.

[0024] Figure 5 It is a schematic diagram of creating multiple focusing planes provided by the present invention.

[0025] Figure 6 It is a schematic diagram of establishing a reduced boundary contour provided by the present invention.

[0026] Figure 7 It is a schematic diagram provided by the present invention of obtaining an adjustment amount after matching a first reference curve group and a second reference curve group. DETAILED DESCRIPTION

[0027] In order to more clearly understand the technical solution of the present invention, the relevant technology is first introduced.

[0028] A light field image is a parameterized representation of a four-dimensional light radiation field in space that contains both position and direction information. In simple terms, it contains all the information about the light during its propagation, including the two-dimensional position information (u, v) and two-dimensional direction information (x, y) of the light. Figure 1 This four-dimensional information enables light field images to more fully reflect the lighting conditions in the real world.

[0029] For the method of acquiring colors, the moiré filter is commonly used. The moiré filter records the brightness information of the three colors of red, green, and blue from natural light, and then combines them into various colors in the photo.

[0030] For focal length, the focal length of a lens refers to the distance between the lens and the image sensor (CMOS sensor) when the image is in focus. Figure 2 , the dotted line represents the field of view. The shorter the focal length, the larger the field of view, and the longer the focal length, the smaller the field of view.

[0031] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0032] The present invention discloses a digital refocusing algorithm for multi-focal matrix video splicing, see Figure 3 In some examples, the digital refocusing algorithm for multi-focal matrix video stitching disclosed in the present invention includes the following steps: S101, determining a focus area according to a specified refocusing position and creating a plurality of focus planes inside the focus area, wherein the plurality of focus planes are arranged in parallel; S102, acquiring light field data associated with the focus plane and using the light field data to generate a basic image belonging to the focus plane; S103, determining a display object on the basic image and calculating a clear area and a blurred area of ​​the display object; S104, obtaining a focus plane associated with the clear area and the blurred area, recorded as a reference focus plane; S105, using a corresponding clear area of ​​the basic image on the reference focus plane to replace the blurred area on the basic image to obtain an accurate display image.

[0033] In general, the technical solution disclosed in the present invention is applied to image reconstruction at the display end. Image reconstruction refers to regenerating an image based on information obtained by light field device acquisition (subsampling). In the early light field device acquisition process, multiple cameras will work simultaneously and use different focal lengths to collect data. For example, multiple cameras work together to simultaneously collect far and near images.

[0034] The principle of subsampling is as follows Figure 4 As shown in the figure, x is the original signal of three sinusoidal signals superimposed on each other; the sparse matrix Ψ is the Fourier transform, which transforms the signal into the frequency domain S; and the observation matrix Φ corresponds to the random subsampling method we use; y is the final sampling result.

[0035] Theoretically, this acquisition method can achieve clear display of objects at different distances within the coverage range on the display end, but it will be affected by focal length adjustment (overall unclear), surrounding environment (causing insufficient local clarity) and subsampling (large data loss), resulting in the clarity of the generated image failing to meet monitoring requirements when refocusing.

[0036] In step S101, the focus area is first determined according to the specified refocusing position and multiple focus planes are created inside the focus area. The multiple focus planes are set in parallel. Generally speaking, the refocusing position is determined by the staff. For example, when observing a certain device or a certain area, refocusing is performed according to the position of the specified device or area.

[0037] Determining the focus area refers to determining the area where the refocus position is located. It should be noted that the focus area obtained at this time is a range area rather than a plane, because the specified device or area is distributed in three-dimensional space and has obvious depth characteristics. When taking pictures with a fixed focal length, the edges will be unclear.

[0038] In order to make each part of a device or a region clear, the present invention chooses to create multiple focus planes inside the focus area. Figure 5 As shown, it is then processed using a synthetic method.

[0039] In step S102, light field data associated with the focus plane is obtained and a basic image belonging to the focus plane is generated using the light field data. The light field data here refers to the data obtained after subsampling. Then, in step S103, the display object on the basic image is determined and the clear area and blurred area of ​​the display object are calculated. The clear area is retained, and the blurred area needs to be replaced.

[0040] In step S104, focus planes associated with clear areas and blurred areas are obtained. These focus planes are recorded as reference focus planes here. Then, the blurred areas on the base image are replaced with corresponding clear areas belonging to the base image on the reference focus planes to obtain an accurate displayed image, which is the content in step S105.

[0041] From the above description, it can be seen that the present invention repairs the blurred area of ​​the basic image on the focus plane by replacing the content on multiple focus areas. This method can generate an image at a specified refocusing position. When the refocusing position is respecified, the image can be generated at the refocusing position again.

[0042] This method breaks away from the fixed focal length image acquisition method. Using a light field device, it is possible to achieve full coverage monitoring of an area. At the same time, it can also adjust long and short distances through refocusing, and meet monitoring requirements in terms of clarity.

[0043] In some examples, the specific method of calculating the clear area and the blurred area of ​​the display object is: S201, obtaining the boundary outline of the display object; S202, establishing a reduced boundary contour inside the display object based on the boundary contour, and the coverage area of ​​the reduced boundary contour tends to decrease in the direction close to the center of the display object; S203, calculating the clarity of the content between two adjacent boundary contours to obtain a clarity calculation result; S204: Divide the display object into a clear area and a blurred area using the clarity calculation result.

[0044] In step S201 to step S204, firstly, a reduced boundary contour is established inside the display object based on the boundary contour, and the reduced boundary contour is equivalent to proportional reduction of the boundary contour. Figure 6 As shown, by establishing a reduced boundary contour, the display object can be divided into multiple rings, and then the clarity of the content between two adjacent boundary contours is calculated to obtain a clarity calculation result, and finally the clarity calculation result is used to divide the display object into a clear area and a blurred area.

[0045] It should be understood that there are many ways to judge clarity, such as making an overall judgment on the display object, such as using the Brenner gradient function, Tenengrad gradient function, Laplacian gradient function and SMD (grayscale variance) function, etc. The advantage of this judgment method is that the clarity of the display object can be evaluated as a whole, but it requires all the content contained in the display object to be processed and cannot be evaluated locally.

[0046] Of course there are ways to mesh the display objects first, but different meshing methods will still lead to different evaluation results.

[0047] In order to solve this problem, the present invention uses a method of calculating the clarity of the content between two adjacent boundary contours. The specific steps are as follows: S301, creating a reference baseline on the content between two adjacent boundary contours; S302, decomposing the reference baseline using wavelet and calculating the high-frequency basis quantity and high-frequency basis quantity area of ​​the reference baseline; S303, dividing the reference baseline according to the high-frequency base volume area to obtain a sub-construction reference baseline; S304, transferring the sub-construction reference baseline into the frequency domain to decompose and calculate the proportion of high-frequency basis in the sub-construction reference baseline; S305 , providing a clarity result of the content between two adjacent boundary contours according to the proportion of the high-frequency basis in the sub-constructed reference baseline.

[0048] In step S301 to step S305, a reference baseline is first created on the content between two adjacent boundary contours. The reference baseline has the same shape as the boundary contour and its length is greater than the inner boundary contour and the outer boundary contour of the two adjacent boundary contours.

[0049] The reference baseline is then decomposed using wavelet decomposition and the high-frequency basis quantity and high-frequency basis quantity area of ​​the reference baseline are calculated. The reason for using wavelet decomposition here is that the result obtained using wavelet decomposition has a starting point and a cutoff point. The high-frequency basis quantity area can be determined through the starting point and the cutoff point.

[0050] The high-frequency basis of the reference baseline mentioned in step S303 refers to the proportion of the high-frequency curve in the curve obtained after the reference baseline is decomposed. The higher the proportion of the high-frequency curve, the higher the clarity reflected by the corresponding reference baseline.

[0051] The high-frequency base volume region refers to the region where the high-frequency base volume exists.

[0052] Next, the reference baseline is divided according to the high-frequency base volume area to obtain the sub-construction reference baseline. The sub-construction reference baseline is then transferred to the frequency domain for decomposition and the proportion of the high-frequency base volume in the sub-construction reference baseline is calculated. The purpose of calculating the proportion of the high-frequency base volume in the sub-construction reference baseline is to verify the accuracy of the reference baseline division.

[0053] According to the above content, we can see that the high-frequency basis has a starting point and an end point, but the proportion of the high-frequency basis in the sub-construction reference baseline divided according to this may not meet the requirements (a set parameter). This is because the wavelet decomposition used here only calculates the high-frequency basis of the reference baseline, but does not involve the medium-frequency basis and the low-frequency basis. Therefore, it is necessary to use the method of calculating the proportion of the high-frequency basis in the sub-construction reference baseline for verification.

[0054] After the calculation is completed, the clarity result of the content between two adjacent boundary contours is given according to the proportion of high-frequency basis in the sub-construction reference baseline.

[0055] After completing the division of the sub-construction reference baseline, two points will be obtained. These two points are the dividing points between the clear area and the fuzzy area. By connecting these points in sequence, the clear area and the fuzzy area can be obtained.

[0056] In some examples, the specific manner of replacing the blurred area on the base image with the corresponding sharp area belonging to the base image on the reference focus plane is: S401, adjusting the reference focus plane to coincide with the focus plane of the base image; S402, cutting the base image on the reference focus plane according to the outline of the blurred area on the base image to obtain a replacement supplementary image; S403, using the replacement supplementary image to replace the blurred area on the basic image; The step of adjusting the reference focus plane to coincide with the focus plane where the basic image is located includes linear adjustment and spatial movement adjustment.

[0057] In step S401 to step S403, the reference focusing plane is first adjusted to coincide with the focusing plane where the base image is located. There are two adjustment methods, namely linear adjustment coincidence and spatial movement adjustment coincidence. Linear adjustment coincidence means that the reference focusing plane can be made coincident with the focusing plane where the base image is located only by zooming in and out; spatial movement adjustment coincidence means that the reference focusing plane can be made coincident with the focusing plane where the base image is located only by a combination of rotation adjustment and zooming in and out.

[0058] During the adjustment process, for areas that need to be stretched, the difference calculation method is used to supplement them, and for areas that need to be compressed, the fixed distance deletion method is used to compress the data and smooth it at the same time.

[0059] There are two reasons why adjustments exist because it is uncertain where the replacement supplemental image is located.

[0060] After the replacement is completed, the replacement supplementary image and the basic image need to be fused. This is because the replacement supplementary image and the basic image are generated based on different environments and there are differences. If only a translation-type local replacement is performed, the picture will have an obvious sense of splicing.

[0061] The specific method of fusing and replacing the supplementary image with the base image is as follows: S501, establishing a fusion reference curve on a basic image and a replacement supplementary image, wherein a first portion of the reference curve is located on the basic image, and a second portion of the reference curve is located on the replacement supplementary image; S502, decomposing the first part of the reference curve and the second part of the reference curve respectively by using wavelet to obtain two decomposition curve groups; S503, matching the two decomposition curve groups and adjusting the supplementary image according to the matching result.

[0062] The principle of the contents in step S501 to step S503 is that in the process of wavelet decomposition, if the starting point and the end point of a curve are located on both the basic image and the replacement supplementary image, then there should be continuity. However, when the replacement supplementary image is used, this part of the curve on the basic image will be broken. Therefore, in the present invention, the adjustment is made by reconnecting this part of the curve.

[0063] The specific way to match two decomposition curve groups is: Determine a curve in the first decomposition curve group whose cutoff point is located at the junction of the first part of the reference curve and the second part of the reference curve, which is recorded as the first reference curve group; Determine a curve in the second decomposition curve group whose starting point is located at the junction of the first part of the reference curve and the second part of the reference curve, and record it as the second reference curve group; Matching is performed on the first reference curve group and the second reference curve group to obtain a matching result.

[0064] In the above steps, refer to Figure 7 , a curve in which the cutoff point in the first decomposition curve group and the starting point in the second decomposition curve group are both located at the junction of the first part of the reference curve and the second part of the reference curve will be obtained, and then the first reference curve group and the second reference curve group are matched. The result of the matching process is movement in the longitudinal direction, and an adjustment amount, that is, the matching result, will be obtained.

[0065] The specific way to adjust the supplementary image according to the matching results is: Obtaining an adjustment parameter group according to the matching result, the adjustment parameter group including an adjustment amount; The curve of the complementary image is adjusted using the adjustment parameter group so that the curve of the complementary image is parallel to the reference curve.

[0066] From the above description, it can be seen that the fusion method in the present invention is to adjust only the broken curves, rather than to adjust the replacement and supplementary image as a whole. The advantage of this adjustment method is that there is no need to adjust the replacement and supplementary image as a whole under the premise of satisfying edge fusion, but only to adjust the content associated with the edge. For the replacement and supplementary image, its transition with the basic image will appear smoother. At the same time, the content associated with the edge on the replacement and supplementary image will also be appropriately adjusted, so that the replacement and supplementary image and the basic image appear more coordinated.

[0067] The present invention also provides a digital refocusing device for multi-focus matrix video splicing, comprising: A focusing plane creating unit, used to determine a focusing area according to a specified refocusing position and create a plurality of focusing planes inside the focusing area, wherein the plurality of focusing planes are arranged in parallel; A basic image generation unit, configured to obtain light field data associated with the focus plane and generate a basic image belonging to the focus plane using the light field data; A basic image replacement unit, used for determining a display object on the basic image and calculating a clear area and a blurred area of ​​the display object; A focus plane acquisition unit, used to acquire a focus plane associated with the clear area and the blurred area, recorded as a reference focus plane; The content replacement unit is used to replace the blurred area on the basic image with the corresponding clear area belonging to the basic image on the reference focus plane to obtain an accurate display image.

[0068] Furthermore, it also includes: A first acquisition unit, used to acquire a boundary contour of a display object; A reduced boundary contour establishing unit, used to establish a reduced boundary contour inside the display object based on the boundary contour, wherein the coverage area of ​​the reduced boundary contour tends to decrease in the direction close to the center of the display object; A first calculation unit, used to calculate the clarity of the content between two adjacent boundary contours to obtain a clarity calculation result; The area division unit is used to divide the display object into a clear area and a blurred area using the clarity calculation result.

[0069] Furthermore, it also includes: A reference baseline establishing unit, used for creating a reference baseline on the content between two adjacent boundary contours; A first decomposition processing unit, used for decomposing the reference baseline using wavelet and calculating the high-frequency basis quantity and high-frequency basis quantity area of ​​the reference baseline; A division unit is used to divide the reference baseline according to the high-frequency base volume area to obtain a sub-construction reference baseline; The second decomposition processing unit is used to transfer the sub-construction reference baseline into the frequency domain to decompose and calculate the proportion of the high-frequency basis in the sub-construction reference baseline; The first result unit is used to provide a clarity result of the content between two adjacent boundary contours according to the proportion of the high-frequency basis in the sub-construction reference baseline.

[0070] Furthermore, it also includes: A position adjustment unit, used for adjusting the reference focus plane to coincide with the focus plane where the base image is located; A cropping processing unit, used for cropping the basic image belonging to the reference focus plane according to the outline of the blurred area on the basic image to obtain a replacement supplementary image; A replacement processing unit, used for replacing the blurred area on the base image with the replacement supplementary image; The step of adjusting the reference focus plane to coincide with the focus plane where the basic image is located includes linear adjustment and spatial movement adjustment.

[0071] Furthermore, it also includes: A fusion reference curve establishing unit, used to establish a fusion reference curve on the basic image and the replacement supplementary image, wherein the first part of the reference curve is located on the basic image and the second part is located on the replacement supplementary image; A third decomposition processing unit is used to decompose the first part of the reference curve and the second part of the reference curve respectively by using wavelet to obtain two decomposition curve groups; The first matching unit is used to match the two decomposition curve groups and adjust the supplementary image according to the matching result.

[0072] Furthermore, it also includes: A first determining unit is used to determine a curve in the first decomposition curve group whose cutoff point is located at a junction of a first part of the reference curve and a second part of the reference curve, which is recorded as a first reference curve group; A second determining unit is used to determine a curve in the second decomposition curve group whose starting point is located at a junction of a first part of the reference curve and a second part of the reference curve, recorded as a second reference curve group; The second matching unit is used to perform matching processing on the first reference curve group and the second reference curve group to obtain a matching result.

[0073] Furthermore, it also includes: A second result unit, used to obtain an adjustment parameter group according to the matching result, wherein the adjustment parameter group includes an adjustment amount; The adjustment unit is used to adjust the curve of the supplementary image using the adjustment parameter group, wherein the curve of the supplementary image is parallel to the reference curve.

[0074] In one example, the unit in any of the above devices can be one or more integrated circuits configured to implement the above methods, such as: one or more application specific integrated circuits (ASICs), or, one or more digital signal processors (DSPs), or, one or more field programmable gate arrays (FPGAs), or a combination of at least two of these integrated circuit forms.

[0075] For another example, when the units in the device can be implemented in the form of a processing element scheduling program, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call a program. For another example, these units can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0076] Various objects such as various messages / information / equipment / network elements / systems / devices / actions / operations / processes / concepts that may appear in the present invention are named. It can be understood that these specific names do not constitute a limitation on the relevant objects. The names assigned may change with factors such as scenarios, contexts or usage habits. The understanding of the technical meaning of the technical terms in the present invention should be mainly determined from the functions and technical effects embodied / executed in the technical scheme.

[0077] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0078] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0079] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0080] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0081] It should also be understood that in various embodiments of the present invention, the first, second, etc. are only used to indicate that multiple objects are different. For example, the first time window and the second time window are only used to indicate different time windows. They should not have any impact on the time window itself, and the first, second, etc. mentioned above should not impose any limitations on the embodiments of the present invention.

[0082] It should also be understood that in the various embodiments of the present invention, unless otherwise specified or there is any logical conflict, the terms and / or descriptions between the different embodiments are consistent and can be referenced to each other, and the technical features in the different embodiments can be combined to form new embodiments according to their internal logical relationships.

[0083] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a computer-readable storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned computer-readable storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0084] The present invention also provides a digital refocusing system for multi-focal matrix video stitching, the system comprising: one or more memories for storing instructions; and One or more processors are used to call and run the instructions from the memory to execute the method as described above.

[0085] The present invention also provides a computer program product, which includes instructions. When the instructions are executed, the terminal device and the network device perform operations of the terminal device and the network device corresponding to the above method.

[0086] The present invention also provides a chip system, which includes a processor for implementing the functions involved in the above content, such as generating, receiving, sending, or processing the data and / or information involved in the above method.

[0087] The chip system may be composed of chips, or may include chips and other discrete devices.

[0088] The processor mentioned in any of the above places can be a CPU, a microprocessor, an ASIC, or one or more integrated circuits for executing programs for controlling the above-mentioned feedback information transmission method.

[0089] In a possible design, the chip system also includes a memory, which is used to store necessary program instructions and data. The processor and the memory can be decoupled and respectively set on different devices, connected by wire or wireless means to support the chip system to implement various functions in the above embodiments. Alternatively, the processor and the memory can also be coupled on the same device.

[0090] Optionally, the computer instructions are stored in a memory.

[0091] Optionally, the memory is a storage unit within the chip, such as a register, a cache, etc. The memory can also be a storage unit within the terminal located outside the chip, such as a ROM or other types of static storage devices that can store static information and instructions, RAM, etc.

[0092] It can be understood that the memory in the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories.

[0093] The non-volatile memory may be a ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), or a flash memory.

[0094] The volatile memory may be a RAM, which is used as an external cache. There are many different types of RAM, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct memory bus RAM.

[0095] The embodiments of this specific implementation method are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A digital refocusing algorithm for multi-focal matrix video stitching, characterized in that: include: Determine a focus area according to a specified refocusing position and create a plurality of focus planes inside the focus area, wherein the plurality of focus planes are arranged in parallel; Acquire light field data associated with the focal plane and generate a basis image attributable to the focal plane using the light field data; Determine the display object on the base image and calculate the clear area and the blurred area of ​​the display object; Obtaining a focus plane associated with the clear area and the blurred area, and recording it as a reference focus plane; The blurred areas on the base image are replaced with the corresponding clear areas belonging to the base image on the reference focus plane to obtain an accurate displayed image.

2. The digital refocusing algorithm for multi-focal matrix video stitching according to claim 1, characterized in that: Calculating the clear area and blurred area of ​​the displayed object includes: Get the boundary outline of the display object; A reduced boundary contour is established inside the display object based on the boundary contour, and the coverage area of ​​the reduced boundary contour tends to decrease in the direction close to the center of the display object; Calculate the clarity of the content between two adjacent boundary contours to obtain a clarity calculation result; Use the sharpness calculation results to divide the displayed object into sharp areas and blurred areas.

3. The digital refocusing algorithm for multi-focal matrix video stitching according to claim 2, characterized in that: Calculating the clarity of the content between two adjacent boundary contours includes: Creates a reference baseline on the content between two adjacent border outlines; Decompose the reference baseline using wavelet and calculate the high-frequency basis quantity and high-frequency basis quantity area of ​​the reference baseline; Divide the reference baseline according to the high-frequency base volume area to obtain the sub-construction reference baseline; Transfer the sub-construction reference baseline into the frequency domain to decompose and calculate the proportion of high-frequency basis in the sub-construction reference baseline; The clarity result of the content between two adjacent boundary contours is given according to the proportion of high-frequency basis in the sub-construction reference baseline.

4. The digital refocusing algorithm for multi-focal matrix video stitching according to claim 1, characterized in that: Replacing blurry areas on the base image with corresponding sharp areas attributed to the base image on the reference focus plane includes: Adjust the reference focus plane to coincide with the focus plane of the base image; According to the outline of the blurred area on the basic image, the basic image belonging to the reference focus plane is cut to obtain a replacement supplementary image; Replace the blurred area on the base image with the replacement complementary image; The step of adjusting the reference focus plane to coincide with the focus plane where the basic image is located includes linear adjustment and spatial movement adjustment.

5. The digital refocusing algorithm for multi-focal matrix video stitching according to claim 4, characterized in that: It also includes fusing and replacing the supplementary image with the base image. The fusing and replacing of the supplementary image with the base image includes: Establishing a fusion reference curve on the base image and the replacement supplementary image, wherein the first part of the reference curve is located on the base image, and the second part is located on the replacement supplementary image; Decomposing the first part of the reference curve and the second part of the reference curve respectively by using wavelet to obtain two decomposition curve groups; Match the two decomposition curve groups and adjust the complementary image based on the matching results.

6. The digital refocusing algorithm for multi-focal matrix video stitching according to claim 5, characterized in that: Matching two decomposition curve groups involves: Determine a curve in the first decomposition curve group whose cutoff point is located at the junction of the first part of the reference curve and the second part of the reference curve, and record it as the first reference curve group; Determine a curve in the second decomposition curve group whose starting point is located at the junction of the first part of the reference curve and the second part of the reference curve, and record it as the second reference curve group; Matching is performed on the first reference curve group and the second reference curve group to obtain a matching result.

7. The digital refocusing algorithm for multi-focal matrix video stitching according to claim 6, characterized in that: Adjustments to the supplementary image based on the matching results include: Obtaining an adjustment parameter group according to the matching result, the adjustment parameter group including an adjustment amount; The curve of the complementary image is adjusted using the adjustment parameter group so that the curve of the complementary image is parallel to the reference curve.

8. A digital refocusing device for multi-focal matrix video splicing, characterized in that: include: A focusing plane creating unit, used to determine a focusing area according to a specified refocusing position and create a plurality of focusing planes inside the focusing area, wherein the plurality of focusing planes are arranged in parallel; A basic image generation unit, configured to obtain light field data associated with the focus plane and generate a basic image belonging to the focus plane using the light field data; A basic image replacement unit, used for determining a display object on the basic image and calculating a clear area and a blurred area of ​​the display object; A focus plane acquisition unit, used to acquire a focus plane associated with the clear area and the blurred area, recorded as a reference focus plane; The content replacement unit is used to replace the blurred area on the basic image with the corresponding clear area belonging to the basic image on the reference focus plane to obtain an accurate display image.

9. A digital refocusing system for multi-focal matrix video stitching, characterized in that: The system comprises: one or more memories for storing instructions; and One or more processors, configured to call and execute the instructions from the memory to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer readable storage medium comprises: The program, when the program is executed by a processor, the method according to any one of claims 1 to 7 is executed.

Images