Video processing methods and apparatus, electronic devices and readable storage media

Abstract

This application discloses a video processing method, apparatus, electronic device, and readable storage medium, belonging to the field of image processing technology. The video processing method includes: acquiring a first image and a second image, both of which include a target object; performing target detection on the first image and the second image, and determining multiple first target regions in the first image and multiple second target regions in the second image based on the detection results; determining a first optical flow based on the multiple first target regions and the multiple second target regions; determining a target optical flow based on the first optical flow and a first optical flow algorithm, wherein the target optical flow is used to indicate the target offset of each pixel in the first image; adjusting the pixel values ​​of each pixel in the first image based on the target offset to obtain a target image; wherein the multiple first target regions and the multiple second target regions correspond one-to-one, and each of the multiple first target regions and the multiple second target regions includes a target object.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, specifically relating to a video processing method and apparatus, electronic device and readable storage medium. Background Technology

[0002] Currently, when calculating the optical flow of interpolated images using traditional sparse optical flow algorithms, the calculation process is not precise enough and includes unnecessary redundant steps, thus increasing the computational load and reducing the accuracy of determining the displacement of moving objects. This reduces the efficiency and accuracy of optical flow calculation for interpolated images, thereby decreasing the smoothness of the interpolated video. Summary of the Invention

[0003] The purpose of this application is to provide a video processing method, apparatus, electronic device, and readable storage medium that can solve the problems of low efficiency and accuracy in optical flow calculation of interpolated images, as well as low smoothness of the video after interpolation.

[0004] In a first aspect, embodiments of this application provide a video processing method, the method comprising: acquiring a first image and a second image from an original video, the first image and the second image being two consecutive frame images, both the first image and the second image including a target object; performing target detection on the first image and the second image, and determining multiple first target regions in the first image and multiple second target regions in the second image based on the detection results; determining a first optical flow based on the multiple first target regions and the multiple second target regions; determining a target optical flow based on the first optical flow and a first optical flow algorithm, the target optical flow being used to indicate the target offset of each pixel in the first image; adjusting the pixel values ​​of each pixel in the first image based on the target offset to obtain a target image; and inserting the target image between the first image and the second image to obtain a target video; wherein the multiple first target regions and the multiple second target regions correspond one-to-one, and both the multiple first target regions and the multiple second target regions include a target object.

[0005] Secondly, embodiments of this application provide a video processing apparatus, comprising: an acquisition unit, configured to acquire a first image and a second image from an original video, wherein the first image and the second image are two consecutive frame images, and both the first image and the second image include a target object; a processing unit, configured to perform target detection on the first image and the second image, and determine a plurality of first target regions in the first image and a plurality of second target regions in the second image based on the detection results; the processing unit is further configured to determine a first optical flow based on the plurality of first target regions and the plurality of second target regions; the processing unit is further configured to determine a target optical flow based on the first optical flow and a first optical flow algorithm, wherein the target optical flow is used to indicate the target offset of each pixel in the first image; the processing unit is further configured to adjust the pixel value of each pixel in the first image based on the target offset to obtain a target image; and the processing unit is further configured to insert the target image between the first image and the second image to obtain a target video; wherein the plurality of first target regions and the plurality of second target regions correspond one-to-one, and both the plurality of first target regions and the plurality of second target regions include a target object.

[0006] Thirdly, embodiments of this application provide an electronic device including a processor and a memory. The memory stores programs or instructions that can run on the processor, and when the programs or instructions are executed by the processor, they implement the steps of the video processing method as described in the first aspect.

[0007] Fourthly, embodiments of this application provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the video processing method as described in the first aspect.

[0008] Fifthly, embodiments of this application provide a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the steps of the video processing method as described in the first aspect.

[0009] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the steps of the video processing method as described in the first aspect.

[0010] In the video processing method provided in this application embodiment, a first image and a second image are obtained from the original video. The first image and the second image are two consecutive frame images, and both the first image and the second image include a target object. Target detection is performed on the first image and the second image, and multiple first target regions in the first image and multiple second target regions in the second image are determined based on the detection results. A first optical flow is determined based on the multiple first target regions and the multiple second target regions. A target optical flow is determined based on the first optical flow and the first optical flow algorithm. The target optical flow is used to indicate the target offset of each pixel in the first image. The pixel values ​​of each pixel in the first image are adjusted according to the target offset to obtain a target image. The target image is inserted between the first image and the second image to obtain a target video. The multiple first target regions and the multiple second target regions correspond one-to-one, and both the multiple first target regions and the multiple second target regions include a target object.

[0011] The video processing method described above involves interpolating two consecutive frames (a first image and a second image) from the original video. Both the first and second images contain a target object. Target detection is then performed on the first and second images, and based on the detection results, multiple first target regions containing the target object are determined in the first image, and multiple second target regions containing the target object are determined in the second image. Further, a first optical flow is determined for the first image based on the multiple first and second target regions. This first optical flow is the local optical flow of the first image (i.e., the optical flow of the multiple first target regions). Further, based on the first optical flow algorithm, the global optical flow and the local optical flow of the first image are fused to obtain a target optical flow. This target optical flow indicates the target offset of each pixel in the first image. Based on this, the pixel values ​​of each pixel in the first image are adjusted according to the target offset to obtain a target image. This target image is then inserted between the first and second images to obtain the target video.

[0012] In this way, during video frame interpolation, high-performance target detection quickly locates the target region in the image that requires dense optical flow calculation, reducing unnecessary regional optical flow calculations and thus reducing the computational load in the optical flow estimation process. Based on this, the global and local optical flow of the image are adaptively fused, efficiently preserving global sparse optical flow information while smoothly incorporating dense optical flow information from the target region. This fully utilizes the advantages of both dense and sparse optical flow information, resulting in an efficient and robust optical flow field. This improves the accuracy of displacement determination for moving objects, enhances the efficiency and accuracy of optical flow calculation, and ultimately ensures the accuracy of target image determination, thereby improving the smoothness of the target video. Attached Figure Description

[0013] Figure 1 A flowchart illustrating the video processing method provided in this application embodiment;

[0014] Figure 2 This is a structural block diagram of the video processing apparatus provided in the embodiments of this application;

[0015] Figure 3 A structural block diagram of the electronic device provided in the embodiments of this application;

[0016] Figure 4 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0018] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0019] The first aspect of this application provides a video processing method. The execution subject of the video processing method provided in this application can be a video processing device, which can be determined according to actual usage requirements. This application does not limit the specific implementation. To more clearly describe the video processing method provided in this application, the following method embodiments will executor the video processing method using a video processing device as the execution subject.

[0020] The video processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0021] like Figure 1 As shown, this application embodiment provides a video processing method, which may include the following steps S102 to S112:

[0022] Step S102: Obtain the first and second images from the original video.

[0023] The first image and the second image mentioned above are two images of the same scene or the same object acquired at different times. Specifically, the first image and the second image mentioned above are two consecutive frames from the original video.

[0024] Furthermore, both the first and second images mentioned above include a target object, which is a moving object. The pixel positions of the target object in the images differ between the first and second images.

[0025] Furthermore, the first image and the second image mentioned above can be two images captured from the video at different times, or the first image and the second image mentioned above can be two consecutive frame images extracted from the video in real time.

[0026] In other words, the video processing method proposed in this application can extract two frames (i.e., the first image and the second image) after the video recording is completed, and perform calculations on the two extracted images to determine the optical flow information of each pixel in the first image. The video processing method proposed in this application can also perform calculations on any two consecutive frames in the video in real time to determine the optical flow information of each frame in the video in real time.

[0027] In practical applications, users can choose the acquisition method and source of the first and second images according to their actual situation, and no specific restrictions are imposed here.

[0028] Step S104: Perform target detection on the first image and the second image, and determine multiple first target regions in the first image and multiple second target regions in the second image based on the detection results.

[0029] The first image and the second image are two images of the same scene or the same object acquired at different times, and both the first image and the second image include a target object, which is a moving object. The first pixel position of the target object in the first image differs from the second pixel position of the target object in the second image, and the change in pixel position is related to the motion state of the target object.

[0030] Furthermore, the aforementioned multiple first target regions and multiple second target regions correspond one-to-one, and each of the multiple first target regions and multiple second target regions includes the aforementioned target object.

[0031] Furthermore, the purpose of the aforementioned object detection is to perform saliency detection on the target object, thereby determining the location region of the target object in the first and second images. In practical applications, the aforementioned object detection can be implemented using saliency detection algorithms such as SR (Spectral Residual), FT (Frequency-Tuned), HC (Histogram-based Contrast), CA (Context-Aware), GR (Graph-Regularized), AIM (Attention based on Information Maximization), and LC (Linear-color Contrast). Users can choose the specific algorithm for object detection according to their actual needs; no specific restrictions are imposed here.

[0032] Specifically, when both the first image and the second image contain a target object, a first saliency map is obtained by performing saliency target detection on the first image, and a second saliency map is obtained by performing saliency target detection on the second image. The first saliency map is used to characterize the region location of the target object in the first image, and the second saliency map is used to characterize the region location of the target object in the second image.

[0033] Based on this, according to the salient target detection results, that is, according to the first and second salient maps, and specifically according to the regional positions of the target objects in the first and second images, the first and second images are subjected to overlapping block processing to determine multiple first target regions in the first image and multiple second target regions in the second image. The multiple first and second target regions correspond one-to-one, and each of the multiple first and second target regions includes the target object; that is, there are overlapping areas between the multiple first and second target regions. In this way, by detecting salient targets in the first and second images, the target regions (i.e., the first and second target regions) requiring dense optical flow calculations are quickly located, reducing unnecessary regional optical flow calculations and thus reducing the computational load in the optical flow estimation process. Simultaneously, it provides dense optical flow information (i.e., local optical flow information) of the target regions for subsequent optical flow calculations, thereby improving the accuracy of determining the motion information of each pixel in the target region.

[0034] Step S106: Determine the first optical flow based on multiple first target regions and multiple second target regions.

[0035] Among them, the aforementioned multiple first target regions and multiple second target regions correspond one-to-one, and the aforementioned target objects are included in all multiple first target regions and multiple second target regions.

[0036] Furthermore, the first optical flow is the local optical flow of the first image, specifically the weighted average optical flow of the optical flows of the plurality of first target regions.

[0037] Specifically, after determining multiple first target regions in the first image and multiple second target regions in the second image by performing salient target detection on the first and second images, the optical flow information of each first target region is determined based on a preset optical flow algorithm (i.e., the aforementioned second optical flow algorithm) according to the image parameters (i.e., the brightness of each pixel in the first and second target regions) of each first target region and the corresponding second target region, thereby obtaining multiple region optical flows. Based on this, a weighted average of the obtained multiple region optical flows is performed to obtain the aforementioned first optical flow. In this way, by determining the optical flow information (i.e., the first optical flow) of the target regions requiring dense optical flow calculation located by salient target detection in the first and second images, dense optical flow information (i.e., local optical flow information) of the target regions is provided for subsequent optical flow calculations, thereby improving the accuracy of determining the motion information of each pixel in the target region, and thus improving the accuracy of determining the motion information of the aforementioned target object.

[0038] Among them, the optical flow of the above-mentioned multiple regions, the above-mentioned multiple first target regions, and the above-mentioned multiple second target regions correspond one-to-one.

[0039] Furthermore, the aforementioned second optical flow algorithm can specifically take the LK (Lucas-Kanade) optical flow algorithm, etc. In practical applications, the specific form of the aforementioned second optical flow algorithm can be determined according to the actual situation, and no specific restrictions are imposed here.

[0040] For example, based on the detection results of salient target detection on the first image I0 and the second image I1, four first target regions I in the first image I0 are determined. 0a I 0b I 0c I 0d And the four second target regions I in the second image I1 1a I 1b I 1c I 1dBased on this, the regional optical flow of each first target region is calculated according to the LK optical flow algorithm, and the regional optical flow of each first target region is denoted as f. a f b f c f d Furthermore, a weighted average is applied to the optical flows of the four regions to obtain an average optical flow f (i.e., the first optical flow). Specifically, when the weight of the optical flow in each region is 1, the aforementioned average optical flow...

[0041] In practical applications, when the first optical flow is obtained by weighted averaging of optical flows from multiple regions, the user can set the weight value of each region's optical flow according to the actual situation of each target region, without making specific restrictions here.

[0042] Step S108: Determine the target optical flow based on the first optical flow and the first optical flow algorithm.

[0043] The first optical flow is the local optical flow of the first image, and specifically it is the weighted average optical flow of the optical flows of the plurality of first target regions.

[0044] Furthermore, the aforementioned target optical flow is used to characterize the motion information of each pixel in the first image, that is, the aforementioned target optical flow is used to indicate the target offset of each pixel in the first image.

[0045] Furthermore, the aforementioned first optical flow algorithm is a dense optical flow algorithm, which can be specifically implemented through a target optical flow equation. Unlike the objective function of traditional dense optical flow algorithms, the target optical flow equation of the first optical flow algorithm proposed in this application embodiment integrates global optical flow information and local optical flow information of the first image.

[0046] Specifically, the above-mentioned target optical flow equation includes two parts: a data term and a smoothing term. The target optical flow equation can be expressed by the following formula (1):

[0047] E(u,v)=E data +αE smooth (u,v) (1),

[0048] Where E(u,v) represents the optical flow energy, (u,v) represents the optical flow vector to be solved, i.e., the target optical flow, and E data E represents a data item. smooth (u,v) represents the smoothing term, and α is a hyperparameter of the target optical flow equation, used to balance the weights of the data term and the smoothing term.

[0049] Based on this, the aforementioned data items are used to describe the global sparse optical flow information of the first image, which can be determined by the traditional sparse optical flow equation. Specifically, the aforementioned data items can be described by the following formula (2):

[0050]

[0051] Wherein, I(x+u,y+v) represents the pixel brightness of each pixel in the second image, I(x,y) represents the pixel brightness of each pixel in the first image, and w is the window range of the first optical flow algorithm.

[0052] Specifically, the aforementioned window range w is the window width w. x With window height w y The convolution, i.e., w = w x *w y .

[0053] Based on this, substituting the window range w into formula (2) above, we can obtain:

[0054] E data =∫(w*I x ·u+w*I y ·v+w*I t ) 2 dxdy (3),

[0055] Furthermore, the above smoothing term can be specifically expressed by the following formula (4):

[0056]

[0057] Where λ is the error coefficient.

[0058] Based on this, substituting the above formulas (3) and (4) into the above formula (1), we can obtain:

[0059]

[0060] Furthermore, the above formula (5) is subjected to least squares transformation, and then the partial derivatives of E(u,v) with respect to u and v are solved simultaneously according to the target optical flow equation after least squares transformation, to obtain the following formulas (6) and (7):

[0061]

[0062]

[0063] in, It is the average optical flow of each pixel in the first image in the x-direction. It is the average optical flow of each pixel in the first image in the y-direction. In the embodiments of this application, the above... as well as All of them are equal to the value of the first optical flow mentioned above.

[0064] Based on this, by combining the above formulas (6) and (7), a recursive equation can be obtained. Substituting the initial value of the optical flow of each pixel in the first image (usually 0 by default, but can be set according to the actual situation) into the recursive equation, the target optical flow (u,v) of each pixel in the first image can be solved.

[0065] Step S110: Adjust the pixel values ​​of each pixel in the first image according to the target offset to obtain the target image.

[0066] The target offset mentioned above refers to the pixel position offset of each pixel in the first image.

[0067] Furthermore, the target image is an image obtained by moving each pixel in the first image according to the aforementioned target offset. Based on the motion trajectory of the target object, the target image is an intermediate image between the first and second images. That is, the pixel position of the target object in the target image lies between the first pixel position of the target object in the first image and the second pixel position of the target object in the second image.

[0068] Specifically, after determining the target offset of each pixel in the first image using the aforementioned video processing method, pixel remapping processing is performed on each pixel in the first image based on the target offset. That is, for each first pixel in the first image, the corresponding second pixel in the first image is determined based on the target offset, and then the pixel value of the first pixel is adjusted based on the determined pixel value of the second pixel to obtain the target image.

[0069] Step S112: Insert the target image between the first image and the second image to obtain the target video.

[0070] The target image is obtained by moving each pixel in the first image according to the target offset. Based on the trajectory of the target object, the target image is an intermediate image between the first and second images. That is, the pixel position of the target object in the target image lies between the first pixel position of the target object in the first image and the second pixel position of the target object in the second image.

[0071] Specifically, after performing pixel remapping on each pixel in the first image according to the determined target offset to obtain the target image, this target image is inserted as a new frame between the first and second images in the original image set, thus obtaining the target video. It can be understood that the frame rate of the target video is greater than that of the original video, meaning the smoothness of the target video is higher than that of the original video.

[0072] The video processing method provided in this application, when performing frame interpolation on the original video, acquires two consecutive frame images (i.e., a first image and a second image) from the original video, both of which contain a target object. Based on this, target detection is performed on the first and second images, and multiple first target regions containing the target object in the first image and multiple second target regions containing the target object in the second image are determined according to the target detection results. Further, a first optical flow of the first image is determined based on the multiple first and second target regions; this first optical flow is the local optical flow of the first image (i.e., the optical flow of the multiple first target regions). Further, based on the first optical flow algorithm, the global optical flow and the local optical flow of the first image are fused to obtain a target optical flow, which is used to indicate the target offset of each pixel in the first image. Based on this, the pixel values ​​of each pixel in the first image are adjusted according to the target offset to obtain a target image, which is then inserted between the first and second images to obtain the target video.

[0073] In this way, during video frame interpolation, high-performance target detection quickly locates the target region in the image that requires dense optical flow calculation, reducing unnecessary regional optical flow calculations and thus reducing the computational load in the optical flow estimation process. Based on this, the global and local optical flow of the image are adaptively fused, efficiently preserving global sparse optical flow information while smoothly incorporating dense optical flow information from the target region. This fully utilizes the advantages of both dense and sparse optical flow information, resulting in an efficient and robust optical flow field. This improves the accuracy of displacement determination for moving objects, thereby enhancing the efficiency and accuracy of optical flow calculation, ensuring the accuracy of target image determination, and ultimately improving the smoothness of the target video.

[0074] It should be noted that in the video processing method proposed in this application embodiment, the first image can be the previous frame of the second image. In this case, the target optical flow is the forward optical flow between the first image and the second image, that is, the second image can be obtained by back-mapping the target optical flow with the first image as the reference image.

[0075] Furthermore, the first image can also be the frame following the second image. In this case, the target optical flow is the backward optical flow between the first and second images, that is, the second image can be obtained by forward mapping the target optical flow using the first image as a reference image. The user can choose the order of the first and second images, i.e., the mapping direction of the target optical flow, according to the actual situation; no specific restrictions are imposed here.

[0076] Furthermore, in practical applications, users can use the first image and the second image as reference images respectively, and determine the forward and backward optical flows between the first image and the second image respectively. Based on this, the first target image and the second target image are determined according to the determined forward and backward optical flows, and the first target image and the second target image are fused to obtain the final target image for video frame interpolation, thereby further ensuring the accuracy of the target image determination and thus ensuring the smoothness of the target video.

[0077] Furthermore, the video processing method proposed in this application embodiment can also determine the optical flow information of each pixel in the first image by performing calculations on the two captured images (the first image and the second image) after shooting. That is to say, the first image and the second image can also be images stored in electronic devices (such as cameras, tablets, smartphones, etc.) or third-party storage areas (such as USB flash drives, optical discs, disks, etc.) after shooting.

[0078] In this embodiment of the application, step S104 specifically includes the following steps S104a to S104c:

[0079] Step S104a: Determine multiple first initial regions in the first image and multiple second initial regions in the second image based on the target detection results.

[0080] The first image and the second image are two images of the same scene or the same object acquired at different times, and both the first image and the second image include a target object, which is a moving object. The first pixel position of the target object in the first image differs from the second pixel position of the target object in the second image, and the change in pixel position is related to the motion state of the target object.

[0081] Furthermore, the aforementioned multiple first initial regions and multiple second initial regions correspond one-to-one, and the target object is included in all of the multiple first initial regions and multiple second initial regions. That is, there are overlapping regions among the multiple first initial regions, and there are overlapping regions among the multiple second initial regions.

[0082] Furthermore, the purpose of the aforementioned object detection is to perform saliency detection on the target objects, thereby determining the location region of the target objects in the first and second images. In practical applications, the aforementioned object detection can be implemented using saliency detection algorithms such as SR, FT, HC, CA, GR, AIM, and LC. Users can choose the specific algorithm for object detection according to their actual needs, and no specific restrictions are imposed here.

[0083] Specifically, when both the first image and the second image contain a target object, salient object detection is performed on the first image to obtain a first saliency map, and salient object detection is performed on the second image to obtain a second saliency map. The first saliency map characterizes the region location of the target object in the first image, and the second saliency map characterizes the region location of the target object in the second image. Based on this, according to the salient object detection results, i.e., according to the first and second saliency maps, i.e., according to the region location of the target object in the first and second images, the first and second images are subjected to overlapping block processing to determine multiple first initial regions in the first image and multiple second initial regions in the second image.

[0084] Step S104b: Fill the boundary pixels of multiple first initial regions according to the boundary pixel values ​​of the first image to obtain multiple first target regions.

[0085] Among them, the aforementioned multiple initial regions correspond one-to-one with the aforementioned multiple first target regions.

[0086] Furthermore, the aforementioned plurality of first initial regions are multiple image regions of irregular shape, while the aforementioned plurality of first target regions are multiple image regions of regular shape. That is, in this embodiment of the application, the boundary pixels of the plurality of first initial regions obtained by overlapping and segmenting are filled using the boundary pixel values ​​of the first image, thereby adjusting the plurality of irregularly shaped first initial regions into multiple first target regions of regular shape.

[0087] Specifically, after dividing the first image into overlapping blocks based on the target detection results to determine multiple irregularly shaped initial regions, additional pixels are set at the boundaries of each initial region to fill in the boundary areas, thereby transforming the irregularly shaped image regions into regularly shaped image regions. Based on this, the pixel values ​​of the additional pixels in each initial region are mapped and filled according to the boundary pixel values ​​of the first image to obtain the corresponding first target region.

[0088] Step S104c: Fill the boundary pixels of multiple second initial regions according to the boundary pixel values ​​of the second image to obtain multiple second target regions.

[0089] Among them, the aforementioned multiple second initial regions correspond one-to-one with the aforementioned multiple second target regions.

[0090] Furthermore, the aforementioned plurality of second initial regions are multiple image regions of irregular shape, while the aforementioned plurality of second target regions are multiple image regions of regular shape. That is, in this embodiment of the application, the boundary pixels of the plurality of second initial regions obtained by overlapping and segmenting are filled by the boundary pixel values ​​of the second image, thereby adjusting the plurality of irregularly shaped second initial regions into multiple second target regions of regular shape.

[0091] Specifically, after dividing the second image into overlapping blocks based on the target detection results to determine multiple irregularly shaped initial regions, additional pixels are set at the boundaries of each initial region to fill in the boundary areas, thereby transforming the irregularly shaped initial regions into regularly shaped ones. Based on this, the pixel values ​​of the additional pixels in each initial region are mapped and filled according to the boundary pixel values ​​of the second image, thus obtaining the corresponding second target region.

[0092] In the embodiments provided in this application, when determining multiple first target regions in a first image and multiple second target regions in a second image based on target detection results, multiple first initial regions in the first image and multiple second initial regions in the second image are first determined based on the target detection results. Based on this, the boundary pixels of the multiple first initial regions are filled according to the boundary pixel values ​​of the first image to obtain multiple first target regions, and the boundary pixels of the multiple second initial regions are filled according to the boundary pixel values ​​of the second image to obtain multiple second target regions. In this way, by filling the boundary pixels of the multiple first initial regions and multiple second initial regions obtained by overlapping and segmenting using the boundary pixel values ​​of the first and second images, the irregularly shaped multiple first initial regions and multiple second initial regions are adjusted into regular-shaped multiple first target regions and multiple second target regions, respectively. In the subsequent optical flow calculation process, the accuracy of registration between the first target regions and the corresponding second target regions is improved, thereby ensuring the accuracy of the first optical flow determination.

[0093] In this embodiment of the application, step S106 may specifically include the following steps S106a and S106b:

[0094] Step S106a: Determine multiple second optical flows based on the second optical flow algorithm, multiple first target regions, and multiple second target regions' image parameters.

[0095] Among them, the aforementioned multiple first target regions, multiple second target regions, and multiple second optical flows correspond one-to-one. The multiple first target regions and multiple second target regions all include the aforementioned target objects. The second optical flow is used to characterize the motion information of each pixel in the corresponding first target region.

[0096] Furthermore, the aforementioned second optical flow algorithm can specifically take the LK or other optical flow algorithms. In practical applications, the specific form of the aforementioned second optical flow algorithm can be determined according to the actual situation, and no specific restrictions are imposed here.

[0097] Specifically, after determining multiple first target regions in the first image and multiple second target regions in the second image by performing salient target detection on the first and second images, the optical flow information of each first target region is determined based on a preset optical flow algorithm (i.e., the aforementioned second optical flow algorithm) according to the image parameters of each first target region and the corresponding second target region (i.e., the brightness of each pixel in the first and second target regions), thereby obtaining multiple second optical flows.

[0098] Step S106b: The first optical flow is obtained by weighted averaging of multiple second optical flows.

[0099] The second optical flow is used to characterize the motion information of each pixel in the corresponding first target region. The first optical flow is the local optical flow of the first image, specifically the weighted average optical flow of the optical flows of the multiple first target regions.

[0100] Specifically, after determining the optical flow information of each of the aforementioned first target regions to obtain multiple second optical flows, a weighted average of the obtained second optical flows is performed to obtain the aforementioned first optical flow. Thus, by determining the optical flow information (i.e., the first optical flow) of the target regions requiring dense optical flow calculation identified by salient target detection in the first and second images, dense optical flow information (i.e., local optical flow information) of the target regions is provided for subsequent optical flow calculations, thereby improving the accuracy of determining the motion information of each pixel in the target region, and thus improving the accuracy of determining the motion information of the aforementioned target object.

[0101] In practical applications, when the first optical flow is obtained by weighted averaging of multiple second optical flows, the user can set the weight value of each second optical flow according to the actual situation of each target area, without making specific restrictions here.

[0102] For example, based on the detection results of salient target detection on the first image I0 and the second image I1, four first target regions I in the first image I0 are determined. 0a I 0b I 0c I 0d And the four second target regions I in the second image I1 1a I 1b I 1c I 1d Based on this, the regional optical flow (i.e., the second optical flow) of each first target region is calculated according to the LK optical flow algorithm, and the regional optical flow of each first target region is denoted as f. a f b f c f d Furthermore, a weighted average is applied to the optical flows of the four regions to obtain an average optical flow f (i.e., the first optical flow). Specifically, when the weight of the optical flow in each region is 1, the aforementioned average optical flow...

[0103] The embodiments provided in this application, when determining the first optical flow based on multiple first target regions and multiple second target regions, specifically determine multiple second optical flows based on a second optical flow algorithm and image parameters of the multiple first target regions and multiple second target regions, and then perform a weighted average of the multiple second optical flows to obtain the first optical flow. Each of the multiple first target regions corresponds one-to-one with a single second optical flow, and the second optical flow is used to characterize the motion information of each pixel in the corresponding first target region. In this way, by calculating the regional optical flow (i.e., the second optical flow) of multiple first target regions containing the target object, and obtaining the first optical flow based on the weighted average of the multiple second optical flows, the computational load in the optical flow calculation process is reduced, while providing local optical flow information (i.e., the first optical flow) for subsequent target optical flow calculation, thereby improving the accuracy of determining the displacement of the target object, and thus improving the efficiency and accuracy of optical flow calculation.

[0104] In this embodiment, the second optical flow is a set of multiple optical flow values, and each of the multiple optical flow values ​​corresponds one-to-one with a pixel in the first target region corresponding to the second optical flow. Based on this, before step S106b, the video processing method may further include the following steps S106c and S106d:

[0105] Step S106c: Determine the norm between each optical flow value and the window value of the second optical flow algorithm.

[0106] The window value mentioned above is the window width w. x With window height w y The convolution, i.e., w = wx *w y .

[0107] Furthermore, the aforementioned norm value can specifically be the Euclidean norm, which is expressed as the square root of the sum of the squares of the absolute values ​​of the vector elements.

[0108] Step S106d: If the norm value corresponding to the first optical flow value among multiple optical flow values ​​is greater than a preset threshold, remove the first optical flow value.

[0109] Specifically, after determining the Euclidean value between each optical flow value and the window value of the second optical flow algorithm, the Euclidean value corresponding to each optical flow value is compared with a preset threshold. If the Euclidean value corresponding to the first optical flow value among multiple optical flow values ​​is greater than the preset threshold, it indicates that the first optical flow value is an isolated optical flow segment in the second optical flow. At this time, the first optical flow value is removed from its corresponding second optical flow to ensure the accuracy of subsequent determination of the first optical flow.

[0110] The specific value of the aforementioned preset threshold can be set by the user according to the actual situation, and no specific restrictions are imposed here.

[0111] In the embodiments provided in this application, the second optical flow is a set of multiple optical flow values, and each of the multiple optical flow values ​​corresponds one-to-one with a pixel in the first target region corresponding to the second optical flow. Based on this, before obtaining the first optical flow by weighted averaging of the multiple second optical flows, for each optical flow value in the second optical flow, its Euclidean norm is calculated relative to the window value of the second optical flow algorithm. If the Euclidean norm calculated from a certain optical flow value is greater than a preset threshold, that optical flow value is removed from the corresponding second optical flow. Thus, by comparing the Euclidean norm between each optical flow value in the second optical flow and the window value of the second optical flow algorithm with the preset threshold, isolated optical flow segments in the second optical flow are filtered, ensuring the accuracy of the subsequent determination of the first optical flow, thereby ensuring the accuracy of determining the motion information of the target object and improving the accuracy of the target optical flow calculation.

[0112] In this embodiment of the application, step S108 may specifically include the following steps S108a to S108e:

[0113] Step S108a: Determine the target optical flow equation based on the first optical flow and the first optical flow algorithm.

[0114] The target optical flow equation is related to the pixel offset of each pixel in the first image.

[0115] Furthermore, the first optical flow is the local optical flow of the first image, specifically the weighted average optical flow of the optical flows of the plurality of first target regions.

[0116] Furthermore, the aforementioned first optical flow algorithm is a dense optical flow algorithm, which can be specifically implemented through a target optical flow equation. Unlike the objective function of traditional dense optical flow algorithms, the target optical flow equation of the first optical flow algorithm proposed in this application embodiment integrates global optical flow information and local optical flow information of the first image (i.e., the aforementioned first optical flow).

[0117] Specifically, the above-mentioned target optical flow equation includes two parts: a data term and a smoothing term. The target optical flow equation can be expressed by the following formula (1):

[0118] E(u,v)=E data +αE smooth (u,v) (1),

[0119] Where E(u,v) represents the optical flow energy, (u,v) represents the optical flow vector to be solved, i.e., the target optical flow, and E data E represents a data item. smooth (u,v) represents the smoothing term, and α is a hyperparameter of the target optical flow equation, used to balance the weights of the data term and the smoothing term.

[0120] Based on this, the aforementioned data items are used to describe the global sparse optical flow information of the first image, which can be determined by the traditional sparse optical flow equation. Specifically, the aforementioned data items can be described by the following formula (2):

[0121]

[0122] Wherein, I(x+u,y+v) represents the pixel brightness of each pixel in the second image, I(x,y) represents the pixel brightness of each pixel in the first image, and w is the window range of the first optical flow algorithm.

[0123] Specifically, the aforementioned window range w is the window width w. x With window height w y The convolution, i.e., w = w x *w y .

[0124] Based on this, substituting the window range w into formula (2) above, we can obtain:

[0125] E data =∫(w*I x ·u+w*I y ·v+w*I t ) 2 dxdy (3),

[0126] Furthermore, the aforementioned smoothing term is used to describe the local dense optical flow information of the first image, which is related to the first optical flow determined above. Specifically, the aforementioned smoothing term can be expressed by the following formula (4):

[0127]

[0128] Where λ is the error coefficient.

[0129] Based on this, substituting the above formulas (3) and (4) into the above formula (1) yields the target optical flow equation:

[0130]

[0131] Step S108b: Perform least-squares transformation on the target optical flow equation.

[0132] Step S108c: Determine the partial differential equation of the processed target optical flow equation with respect to the pixel offset.

[0133] Specifically, the above formula (5) is subjected to least squares transformation, and then the partial derivatives of E(u,v) with respect to u and v are solved simultaneously according to the target optical flow equation after least squares transformation, to obtain the following formulas (6) and (7):

[0134]

[0135]

[0136] in, It is the average optical flow of each pixel in the first image in the x-direction. It is the average optical flow of each pixel in the first image in the y-direction. In the embodiments of this application, the above... as well as All of them are equal to the value of the first optical flow mentioned above.

[0137] Step S108d: Determine the target recursive equation based on the partial differential equation.

[0138] Step S108e: Substitute the initial optical flow value of each pixel in the first image into the target recursive equation to obtain the target optical flow of each pixel in the first image.

[0139] Specifically, by combining the above formulas (6) and (7), a target recursive equation can be obtained. Substituting the initial value of the optical flow of each pixel in the first image (usually 0 by default, but can be set according to the actual situation) into the target recursive equation, the target optical flow (u,v) of each pixel in the first image can be solved.

[0140] In the embodiments provided in this application, when determining the target optical flow based on the first optical flow and the first optical flow algorithm, the target optical flow equation is specifically determined based on the first optical flow and the first optical flow algorithm. This target optical flow equation is related to the pixel offset of each pixel in the first image. Based on this, the target optical flow equation is subjected to least-squares processing, and the partial differential equation of the processed target optical flow equation with respect to the pixel offset is determined. Then, the target recursive equation is determined based on the partial differential equation. Substituting the initial optical flow values ​​of each pixel in the first image into the target recursive equation yields the target optical flow of each pixel in the first image. Thus, through the first optical flow algorithm, the global and local optical flow information of the first image are adaptively fused, efficiently preserving global sparse optical flow information while smoothly incorporating dense optical flow information from the target region. This fully utilizes the advantages of both dense and sparse optical flow information, resulting in an efficient and robust optical flow field, thereby improving the accuracy of displacement determination for moving objects and ultimately enhancing the efficiency and accuracy of optical flow calculation.

[0141] In this embodiment of the application, step S110 may specifically include the following steps S110a to S110c:

[0142] Step S110a: Obtain the first pixel value and coordinates of the first pixel in the first image.

[0143] It is understandable that an image is composed of multiple pixels, and different pixels are located at different pixel positions. If we consider the first image as a two-dimensional coordinate system, then each pixel in the image corresponds to a unique and definite coordinate point, which is the pixel coordinate of the corresponding pixel.

[0144] Specifically, after determining the target offset of each pixel in the first image using the video processing method described above, for each first pixel in the first image, the coordinates of the first pixel and the value of the first pixel are determined for subsequent use.

[0145] Step S110b: Determine the coordinates of the second pixel in the first image based on the coordinates of the first pixel and the target offset.

[0146] The target offset mentioned above refers to the pixel position offset of each pixel in the first image.

[0147] Furthermore, the first pixel and the second pixel correspond to each other. Specifically, there is a mapping relationship between the second pixel and the first pixel, which can be expressed by the aforementioned target offset. For each first pixel in the first image, a unique and definite second pixel can be determined in the first image using the aforementioned target offset.

[0148] Specifically, the target offset mentioned above is a coordinate offset, and the coordinates of the second pixel point = the coordinates of the first pixel point + the target offset.

[0149] Step S110c: Adjust the pixel value of the second pixel point corresponding to the coordinates of the second pixel point to the value of the first pixel point.

[0150] Specifically, after determining the target offset of each pixel in the first image using the above video processing method, pixel remapping processing is performed on each pixel in the first image based on the target offset.

[0151] That is, for each first pixel in the first image, the coordinates of a second pixel in the first image are determined based on the first pixel's coordinates and the target offset. These second pixel coordinates correspond to the second pixel in the first image. Based on this, the pixel value of the second pixel is adjusted to the first pixel value of the first pixel to obtain the target image.

[0152] Therefore, the target image is obtained by moving each pixel in the first image according to the aforementioned target offset. Based on the trajectory of the target object, the target image is an intermediate image between the first and second images. That is, the pixel position of the target object in the target image lies between the first pixel position of the target object in the first image and the second pixel position of the target object in the second image.

[0153] The embodiments provided in this application, when adjusting the pixel values ​​of each pixel in the first image according to the target offset, specifically obtain the first pixel value and coordinates of the first pixel in the first image, then determine the coordinates of the second pixel in the first image based on the first pixel coordinates and the target offset, and adjust the pixel value of the second pixel corresponding to the second pixel coordinates to the first pixel value. In this way, by performing pixel remapping processing on the first image, the target image obtained by moving each pixel in the first image according to the aforementioned target offset is determined, ensuring the accuracy of the target image determination and thereby improving the smoothness of the target video.

[0154] The video processing method provided in the first aspect of this application can be executed by a video processing device. This application uses the execution of the above-described video processing method by a video processing device as an example to illustrate the video processing device provided in the second aspect of this application.

[0155] like Figure 2 As shown, this application embodiment provides a video processing apparatus 200, which may include the acquisition unit 202 and the processing unit 204 described below.

[0156] Acquisition unit 202 is used to acquire a first image and a second image from the original video. The first image and the second image are two consecutive frame images, and both the first image and the second image include the target object.

[0157] The processing unit 204 is configured to perform target detection on the first image and the second image, and determine multiple first target regions in the first image and multiple second target regions in the second image based on the detection results;

[0158] Processing unit 204 is further configured to determine a first optical flow based on a plurality of first target regions and a plurality of second target regions;

[0159] The processing unit 204 is further configured to determine the target optical flow based on the first optical flow and the first optical flow algorithm, wherein the target optical flow is used to indicate the target offset of each pixel in the first image;

[0160] The processing unit 204 is also used to adjust the pixel values ​​of each pixel in the first image according to the target offset to obtain the target image;

[0161] The processing unit 204 is also used to insert the target image between the first image and the second image to obtain the target video;

[0162] Among them, multiple first target areas and multiple second target areas correspond one-to-one, and each of the multiple first target areas and multiple second target areas includes a target object.

[0163] The video processing apparatus 200 provided in this application embodiment, when performing frame interpolation processing on the original video, acquires two consecutive frame images (i.e., a first image and a second image) from the original video through the acquisition unit 202. Both the first image and the second image contain a target object. Based on this, the processing unit 204 performs target detection on the first image and the second image, and determines multiple first target regions containing the target object in the first image and multiple second target regions containing the target object in the second image based on the target detection results. Further, the processing unit 204 determines a first optical flow of the first image based on the multiple first target regions and the multiple second target regions. This first optical flow is the local optical flow of the first image (i.e., the optical flow of the multiple first target regions). Further, based on the first optical flow algorithm, the processing unit 204 fuses the global optical flow of the first image and the local optical flow to obtain a target optical flow. The target optical flow is used to indicate the target offset of each pixel in the first image. Based on this, the processing unit 204 adjusts the pixel values ​​of each pixel in the first image according to the target offset to obtain the target image, and inserts the obtained target image between the first image and the second image to obtain the target video.

[0164] In this way, during video frame interpolation, high-performance target detection quickly locates the target region in the image that requires dense optical flow calculation, reducing unnecessary regional optical flow calculations and thus reducing the computational load in the optical flow estimation process. Based on this, the global and local optical flow of the image are adaptively fused, efficiently preserving global sparse optical flow information while smoothly incorporating dense optical flow information from the target region. This fully utilizes the advantages of both dense and sparse optical flow information, resulting in an efficient and robust optical flow field. This improves the accuracy of displacement determination for moving objects, enhances the efficiency and accuracy of optical flow calculation, and ultimately ensures the accuracy of target image determination, thereby improving the smoothness of the target video.

[0165] In this embodiment of the application, the processing unit 204 is specifically used to: determine a plurality of first initial regions in a first image and a plurality of second initial regions in a second image based on the detection results; fill the boundary pixels of the plurality of first initial regions according to the boundary pixel values ​​of the first image to obtain a plurality of first target regions; and fill the boundary pixels of the plurality of second initial regions according to the boundary pixel values ​​of the second image to obtain a plurality of second target regions.

[0166] In the embodiments provided in this application, when the processing unit 204 determines multiple first target regions in the first image and multiple second target regions in the second image based on the target detection results, it first determines multiple first initial regions in the first image and multiple second initial regions in the second image based on the target detection results. Based on this, the boundary pixels of the multiple first initial regions are filled according to the boundary pixel values ​​of the first image to obtain multiple first target regions, and the boundary pixels of the multiple second initial regions are filled according to the boundary pixel values ​​of the second image to obtain multiple second target regions. In this way, by filling the boundary pixels of the multiple first initial regions and multiple second initial regions obtained by overlapping and segmenting using the boundary pixel values ​​of the first and second images, the irregularly shaped multiple first initial regions and multiple second initial regions are adjusted into regular-shaped multiple first target regions and multiple second target regions, respectively. In the subsequent optical flow calculation process, the accuracy of the registration between the first target region and the corresponding second target region is improved, thereby ensuring the accuracy of the first optical flow determination.

[0167] In this embodiment of the application, the processing unit 204 is specifically used to: determine a plurality of second optical flows based on the second optical flow algorithm, the image parameters of a plurality of first target regions and a plurality of second target regions; and perform a weighted average of the plurality of second optical flows to obtain a first optical flow; wherein, the plurality of first target regions and the plurality of second optical flows correspond one-to-one, and the second optical flow is used to characterize the motion information of each pixel in the corresponding first target region.

[0168] In the embodiments provided in this application, when the processing unit 204 determines the first optical flow based on multiple first target regions and multiple second target regions, it specifically determines multiple second optical flows based on a second optical flow algorithm and image parameters of the multiple first target regions and multiple second target regions. Then, it performs a weighted average of the multiple second optical flows to obtain the first optical flow. Each of the multiple first target regions corresponds one-to-one with a single second optical flow, and the second optical flow is used to characterize the motion information of each pixel in the corresponding first target region. In this way, by calculating the regional optical flow (i.e., the second optical flow) of multiple first target regions containing the target object, and obtaining the first optical flow based on the weighted average of the multiple second optical flows, the computational load in the optical flow calculation process is reduced. Simultaneously, local optical flow information (i.e., the first optical flow) is provided for subsequent target optical flow calculation, thereby improving the accuracy of determining the displacement of the target object and thus enhancing the efficiency and accuracy of optical flow calculation.

[0169] In this embodiment of the application, the second optical flow is a set of multiple optical flow values, and the multiple optical flow values ​​correspond one-to-one with each pixel in the first target region corresponding to the second optical flow. The processing unit 204 is further configured to: determine the norm value between each optical flow value and the window value of the second optical flow algorithm; and remove the first optical flow value if the norm value corresponding to the first optical flow value among the multiple optical flow values ​​is greater than a preset threshold.

[0170] In the embodiments provided in this application, the second optical flow is a set of multiple optical flow values, and each of the multiple optical flow values ​​corresponds one-to-one with a pixel in the first target region corresponding to the second optical flow. Based on this, before the processing unit 204 performs a weighted average of the multiple second optical flows to obtain the first optical flow, for each optical flow value in the second optical flow, its Euclidean norm is calculated relative to the window value of the second optical flow algorithm. If the Euclidean norm calculated from a certain optical flow value is greater than a preset threshold, that optical flow value is removed from the corresponding second optical flow. Thus, by comparing the Euclidean norm between each optical flow value in the second optical flow and the window value of the second optical flow algorithm with the preset threshold, isolated optical flow segments in the second optical flow are filtered, ensuring the accuracy of the subsequent determination of the first optical flow, thereby ensuring the accuracy of the determination of the motion information of the target object and improving the accuracy of the target optical flow calculation.

[0171] In this embodiment, the processing unit 204 is specifically used to: determine the target optical flow equation based on the first optical flow and the first optical flow algorithm, wherein the target optical flow equation is related to the pixel offset of each pixel in the first image; perform least squares processing on the target optical flow equation; determine the partial differential equation of the processed target optical flow equation with respect to the pixel offset; determine the target recursive equation based on the partial differential equation; and substitute the initial optical flow value of each pixel in the first image into the target recursive equation to obtain the target optical flow of each pixel in the first image.

[0172] In the embodiments provided in this application, when the processing unit 204 determines the target optical flow based on the first optical flow and the first optical flow algorithm, it specifically determines the target optical flow equation based on the first optical flow and the first optical flow algorithm. This target optical flow equation is related to the pixel offset of each pixel in the first image. Based on this, the target optical flow equation is subjected to least-squares processing, and the partial differential equation of the processed target optical flow equation with respect to the pixel offset is determined. Then, the target recursive equation is determined based on the partial differential equation. Substituting the initial optical flow values ​​of each pixel in the first image into the target recursive equation yields the target optical flow of each pixel in the first image. Thus, through the first optical flow algorithm, the global and local optical flow information of the first image are adaptively fused, efficiently preserving global sparse optical flow information while smoothly incorporating dense optical flow information from the target region. This fully utilizes the advantages of both dense and sparse optical flow information, resulting in an efficient and robust optical flow field, thereby improving the accuracy of displacement determination for moving objects and ultimately enhancing the efficiency and accuracy of optical flow calculation.

[0173] In this embodiment of the application, the processing unit 204 is specifically used to: obtain the first pixel value and the coordinates of the first pixel in the first image; determine the coordinates of the second pixel in the first image based on the coordinates of the first pixel and the target offset; and adjust the pixel value of the second pixel corresponding to the coordinates of the second pixel to the first pixel value.

[0174] In the embodiments provided in this application, when the processing unit 204 adjusts the pixel values ​​of each pixel in the first image according to the target offset, it specifically obtains the first pixel value and coordinates of the first pixel in the first image, then determines the coordinates of the second pixel in the first image based on the first pixel coordinates and the target offset, and adjusts the pixel value of the second pixel corresponding to the second pixel coordinates to the first pixel value. In this way, by performing pixel remapping processing on the first image, the target image obtained by moving each pixel in the first image according to the aforementioned target offset is determined, ensuring the accuracy of the target image determination and thereby improving the smoothness of the target video.

[0175] The video processing device 200 in this embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This embodiment does not specifically limit the functionality of the electronic device.

[0176] The video processing device 200 in this embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this embodiment does not specifically limit its use.

[0177] The video processing apparatus 200 provided in the second aspect embodiment of this application is capable of realizing Figure 1 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0178] Optionally, such as Figure 3 As shown, this application embodiment also provides an electronic device 300, including a processor 302 and a memory 304. The memory 304 stores a program or instructions that can run on the processor 302. When the program or instructions are executed by the processor 302, they implement the various steps of the video processing method embodiment of the first aspect described above and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0179] It should be noted that the electronic devices in the embodiments of this application include the aforementioned mobile electronic devices and non-mobile electronic devices.

[0180] Figure 4 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0181] The electronic device 400 includes, but is not limited to, components such as: radio frequency unit 401, network module 402, audio output unit 403, input unit 404, sensor 405, display unit 406, user input unit 407, interface unit 408, memory 409, and processor 410.

[0182] Those skilled in the art will understand that the electronic device 400 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 410 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0183] The electronic device 400 of this application embodiment can be used to implement the various steps of the video processing method embodiment of the first aspect described above.

[0184] The input unit 404 is used to acquire a first image and a second image from the original video. The first image and the second image are two consecutive frame images, and both the first image and the second image include the target object.

[0185] The processor 410 is configured to perform target detection on the first image and the second image, and determine multiple first target regions in the first image and multiple second target regions in the second image based on the detection results.

[0186] The processor 410 is also used to determine a first optical flow based on a plurality of first target regions and a plurality of second target regions.

[0187] The processor 410 is also configured to determine a target optical flow based on the first optical flow and the first optical flow algorithm, wherein the target optical flow is used to indicate the target offset of each pixel in the first image;

[0188] The processor 410 is also used to adjust the pixel values ​​of each pixel in the first image according to the target offset to obtain the target image.

[0189] The processor 410 is also used to insert the target image between the first image and the second image to obtain the target video.

[0190] Among them, multiple first target areas and multiple second target areas correspond one-to-one, and each of the multiple first target areas and multiple second target areas includes a target object.

[0191] In this embodiment, when performing frame interpolation on the original video, the electronic device 400 acquires two consecutive frame images (i.e., a first image and a second image) from the original video through the input unit 404. Both the first and second images contain a target object. Based on this, the processor 410 performs target detection on the first and second images, and determines multiple first target regions containing the target object in the first image and multiple second target regions containing the target object in the second image based on the target detection results. Further, the processor 410 determines a first optical flow of the first image based on the multiple first and second target regions. This first optical flow is the local optical flow of the first image (i.e., the optical flow of the multiple first target regions). Further, based on the first optical flow algorithm, the processor 410 fuses the global optical flow and the local optical flow of the first image to obtain a target optical flow. The target optical flow is used to indicate the target offset of each pixel in the first image. Based on this, the processor 410 adjusts the pixel values ​​of each pixel in the first image according to the target offset to obtain a target image, and inserts the obtained target image between the first and second images to obtain the target video.

[0192] In this way, during video frame interpolation, high-performance target detection quickly locates the target region in the image that requires dense optical flow calculation, reducing unnecessary regional optical flow calculations and thus reducing the computational load in the optical flow estimation process. Based on this, the global and local optical flow of the image are adaptively fused, efficiently preserving global sparse optical flow information while smoothly incorporating dense optical flow information from the target region. This fully utilizes the advantages of both dense and sparse optical flow information, resulting in an efficient and robust optical flow field. This improves the accuracy of displacement determination for moving objects, enhances the efficiency and accuracy of optical flow calculation, and ultimately ensures the accuracy of target image determination, thereby improving the smoothness of the target video.

[0193] Optionally, the processor 410 is specifically configured to: determine multiple first initial regions in a first image based on the detection results, and determine multiple second initial regions in a second image; fill the boundary pixels of the multiple first initial regions according to the boundary pixel values ​​of the first image to obtain multiple first target regions; and fill the boundary pixels of the multiple second initial regions according to the boundary pixel values ​​of the second image to obtain multiple second target regions.

[0194] In the embodiments provided in this application, when the processor 410 determines multiple first target regions in a first image and multiple second target regions in a second image based on the target detection results, it first determines multiple first initial regions in the first image and multiple second initial regions in the second image based on the target detection results. Based on this, the boundary pixels of the multiple first initial regions are filled according to the boundary pixel values ​​of the first image to obtain multiple first target regions, and the boundary pixels of the multiple second initial regions are filled according to the boundary pixel values ​​of the second image to obtain multiple second target regions. In this way, by filling the boundary pixels of the multiple first initial regions and multiple second initial regions obtained by overlapping and segmenting using the boundary pixel values ​​of the first and second images, the irregularly shaped multiple first initial regions and multiple second initial regions are adjusted into regular-shaped multiple first target regions and multiple second target regions, respectively. In the subsequent optical flow calculation process, the accuracy of the registration between the first target regions and the corresponding second target regions is improved, thereby ensuring the accuracy of the first optical flow determination.

[0195] Optionally, the processor 410 is specifically configured to: determine multiple second optical flows based on the second optical flow algorithm, multiple first target regions, and multiple second target regions' image parameters; and perform a weighted average of the multiple second optical flows to obtain a first optical flow; wherein the multiple first target regions and the multiple second optical flows correspond one-to-one, and the second optical flows are used to characterize the motion information of each pixel in the corresponding first target region.

[0196] In the embodiments provided in this application, when the processor 410 determines the first optical flow based on multiple first target regions and multiple second target regions, it specifically determines multiple second optical flows based on a second optical flow algorithm and image parameters of the multiple first target regions and multiple second target regions. Then, it performs a weighted average of the multiple second optical flows to obtain the first optical flow. Each of the multiple first target regions corresponds one-to-one with a single second optical flow, and the second optical flow is used to characterize the motion information of each pixel in the corresponding first target region. In this way, by calculating the regional optical flow (i.e., the second optical flow) of multiple first target regions containing the target object, and obtaining the first optical flow based on the weighted average of the multiple second optical flows, the computational load in the optical flow calculation process is reduced. Simultaneously, local optical flow information (i.e., the first optical flow) is provided for subsequent target optical flow calculation, thereby improving the accuracy of determining the displacement of the target object and thus enhancing the efficiency and accuracy of optical flow calculation.

[0197] Optionally, the second optical flow is a set of multiple optical flow values, and the multiple optical flow values ​​correspond one-to-one with each pixel in the first target region corresponding to the second optical flow. The processor 410 is further configured to: determine the norm value between each optical flow value and the window value of the second optical flow algorithm; and remove the first optical flow value if the norm value corresponding to the first optical flow value among the multiple optical flow values ​​is greater than a preset threshold.

[0198] In the embodiments provided in this application, the second optical flow is a set of multiple optical flow values, and each of the multiple optical flow values ​​corresponds one-to-one with a pixel in the first target region corresponding to the second optical flow. Based on this, before the processor 410 performs a weighted average of the multiple second optical flows to obtain the first optical flow, for each optical flow value in the second optical flow, its Euclidean norm is calculated relative to the window value of the second optical flow algorithm. If the Euclidean norm calculated from a certain optical flow value is greater than a preset threshold, that optical flow value is removed from the corresponding second optical flow. Thus, by comparing the Euclidean norm between each optical flow value in the second optical flow and the window value of the second optical flow algorithm with the preset threshold, isolated optical flow segments in the second optical flow are filtered, ensuring the accuracy of the subsequent determination of the first optical flow, thereby ensuring the accuracy of the determination of the motion information of the target object and improving the accuracy of the target optical flow calculation.

[0199] Optionally, the processor 410 is specifically configured to: determine a target optical flow equation based on a first optical flow and a first optical flow algorithm, wherein the target optical flow equation is related to the pixel offset of each pixel in the first image; perform least squares processing on the target optical flow equation; determine the partial differential equation of the processed target optical flow equation with respect to the pixel offset; determine a target recursive equation based on the partial differential equation; and substitute the initial optical flow values ​​of each pixel in the first image into the target recursive equation to obtain the target optical flow of each pixel in the first image.

[0200] In the embodiments provided in this application, when the processor 410 determines the target optical flow based on the first optical flow and the first optical flow algorithm, it specifically determines the target optical flow equation based on the first optical flow and the first optical flow algorithm. This target optical flow equation is related to the pixel offset of each pixel in the first image. Based on this, the target optical flow equation is subjected to least-squares processing, and the partial differential equation of the processed target optical flow equation with respect to the pixel offset is determined. Then, the target recursive equation is determined based on the partial differential equation. Substituting the initial optical flow values ​​of each pixel in the first image into the target recursive equation yields the target optical flow of each pixel in the first image. Thus, through the first optical flow algorithm, the global and local optical flow information of the first image are adaptively fused, efficiently preserving global sparse optical flow information while smoothly incorporating dense optical flow information from the target region. This fully utilizes the advantages of both dense and sparse optical flow information, resulting in an efficient and robust optical flow field, thereby improving the accuracy of displacement determination for moving objects and ultimately enhancing the efficiency and accuracy of optical flow calculation.

[0201] Optionally, the processor 410 is specifically configured to: obtain the first pixel value and coordinates of the first pixel in the first image; determine the coordinates of the second pixel in the first image based on the coordinates of the first pixel and the target offset; and adjust the pixel value of the second pixel corresponding to the coordinates of the second pixel to the first pixel value.

[0202] In the embodiments provided in this application, when the processor 410 adjusts the pixel values ​​of each pixel in the first image according to the target offset, it specifically obtains the first pixel value and coordinates of the first pixel in the first image, then determines the coordinates of the second pixel in the first image based on the first pixel coordinates and the target offset, and adjusts the pixel value of the second pixel corresponding to the second pixel coordinates to the first pixel value. In this way, by performing pixel remapping processing on the first image, the target image obtained by moving each pixel in the first image according to the aforementioned target offset is determined, ensuring the accuracy of the target image determination and thereby improving the smoothness of the target video.

[0203] It should be understood that, in this embodiment, the input unit 404 may include a graphics processing unit (GPU) 4041 and a microphone 4042. The GPU 4041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 406 may include a display panel 4061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 407 includes at least one of a touch panel 4071 and other input devices 4072. The touch panel 4071 is also called a touch screen. The touch panel 4071 may include a touch detection device and a touch controller. Other input devices 4072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0204] The memory 409 can be used to store software programs and various data. The memory 409 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 409 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 409 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0205] Processor 410 may include one or more processing units; optionally, processor 410 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 410.

[0206] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the video processing method embodiment of the first aspect described above and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0207] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0208] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described first aspect video processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0209] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0210] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the video processing method embodiment of the first aspect described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0211] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0212] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0213] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method of video processing, the method comprising: The method comprises: obtaining a first image and a second image in an original video, the first image and the second image being two continuous frame images, the target object being included in the first image and the second image; target detection is performed on the first image and the second image to determine the position region of the target object in the first image and the second image, and a plurality of first target regions in the first image and a plurality of second target regions in the second image are determined according to the detection result; determining a first optical flow according to the plurality of first target regions and the plurality of second target regions; determining a target optical flow according to the first optical flow and a first optical flow algorithm, the target optical flow being used to indicate a target offset amount of each pixel point in the first image, and the first optical flow algorithm being a dense optical flow algorithm; adjusting the pixel value of each pixel point in the first image according to the target offset amount to obtain a target image, the target image being an image obtained by moving each pixel point in the first image according to the target offset amount; inserting the target image between the first image and the second image to obtain a target video; wherein the plurality of first target regions and the plurality of second target regions correspond one-to-one, and the target object is included in the plurality of first target regions and the plurality of second target regions.

2. The video processing method of claim 1, wherein, The method comprises: determining a plurality of first initial regions in the first image and a plurality of second initial regions in the second image according to the detection result; filling the boundary pixels of the plurality of first initial regions according to the boundary pixel value of the first image to obtain the plurality of first target regions; filling the boundary pixels of the plurality of second initial regions according to the boundary pixel value of the second image to obtain the plurality of second target regions.

3. The video processing method of claim 1, wherein, The method comprises: determining a plurality of second optical flows according to a second optical flow algorithm, image parameters of the plurality of first target regions and the plurality of second target regions; weighting and averaging the plurality of second optical flows to obtain the first optical flow; wherein the plurality of first target regions and the plurality of second optical flows correspond one-to-one, and the second optical flow is used to represent the motion information of each pixel point in the corresponding first target region.

4. The video processing method of claim 3, wherein, The second optical flow is a set of a plurality of optical flow values, the plurality of optical flow values corresponding one-to-one to each pixel point in the first target region corresponding to the second optical flow, and before the weighting and averaging of the plurality of second optical flows to obtain the first optical flow, the method further comprises: determining the norm value between each optical flow value and the window value of the second optical flow algorithm; in the case that the norm value corresponding to a first optical flow value in the plurality of optical flow values is greater than a preset threshold value, removing the first optical flow value.

5. The video processing method of any of claims 1-4, wherein, The method comprises: determine a target optical flow equation according to the first optical flow and the first optical flow algorithm, the target optical flow equation being related to a pixel displacement of each pixel point in the first image; perform least square processing on the target optical flow equation; determine a partial differential equation of the processed target optical flow equation with respect to the pixel displacement; determine a target recursive equation according to the partial differential equation; substitute an initial optical flow value of each pixel point in the first image into the target recursive equation to obtain the target optical flow of each pixel point in the first image.

6. The video processing method of any of claims 1-4, wherein, The adjusting the pixel value of each pixel point in the first image according to the target displacement comprises: obtaining a first pixel value of a first pixel point and a first pixel point coordinate in the first image; determining a second pixel point coordinate in the first image according to the first pixel point coordinate and the target displacement; adjusting a pixel value of a second pixel point corresponding to the second pixel point coordinate to the first pixel value.

7. A video processing apparatus, comprising: The video processing apparatus comprises: an obtaining unit configured to obtain a first image and a second image in an original video, the first image and the second image being two continuous frame images, and the first image and the second image both comprising a target object; a processing unit configured to perform target detection on the first image and the second image to determine a position region of the target object in the first image and the second image, and determine a plurality of first target regions in the first image and a plurality of second target regions in the second image according to a detection result; the processing unit is further configured to determine a first optical flow according to the plurality of first target regions and the plurality of second target regions; the processing unit is further configured to determine a target optical flow according to the first optical flow and a first optical flow algorithm, the target optical flow being used to indicate a target displacement of each pixel point in the first image, and the first optical flow algorithm being a dense optical flow algorithm; the processing unit is further configured to adjust a pixel value of each pixel point in the first image according to the target displacement to obtain a target image, the target image being an image obtained by moving each pixel point in the first image according to the target displacement; the processing unit is further configured to insert the target image between the first image and the second image to obtain a target video; wherein the plurality of first target regions and the plurality of second target regions correspond to each other one by one, and the target object is included in the plurality of first target regions and the plurality of second target regions.

8. The video processing apparatus of claim 7, wherein, The processing unit is specifically configured to: determine a plurality of first initial regions in the first image and a plurality of second initial regions in the second image according to the detection result; fill boundary pixels of the plurality of first initial regions according to a boundary pixel value of the first image to obtain the plurality of first target regions; fill boundary pixels of the plurality of second initial regions according to a boundary pixel value of the second image to obtain the plurality of second target regions.

9. The video processing apparatus of claim 7, wherein, The processing unit is specifically configured to: determine a plurality of second optical flows according to a second optical flow algorithm, image parameters of the plurality of first target regions and the plurality of second target regions; perform weighted average on the plurality of second optical flows to obtain the first optical flow; wherein the plurality of first target regions and the plurality of second target regions are in one-to-one correspondence, and the second optical flow is used to represent motion information of each pixel point in the corresponding first target region.

10. The video processing apparatus of claim 9, wherein, The second optical flow is a set of a plurality of optical flow values, the plurality of optical flow values are in one-to-one correspondence with each pixel point in the first target region corresponding to the second optical flow, and the processing unit is further configured to: determine a norm value between each optical flow value and a window value of the second optical flow algorithm; remove a first optical flow value in the plurality of optical flow values in a case where a norm value corresponding to the first optical flow value is greater than a preset threshold.

11. The video processing apparatus of any of claims 7-10, wherein, The processing unit is specifically configured to: determine a target optical flow equation according to the first optical flow and a first optical flow algorithm, the target optical flow equation being related to a pixel offset of each pixel point in the first image; perform least square processing on the target optical flow equation; determine a partial differential equation of the processed target optical flow equation with respect to the pixel offset; determine a target recursive equation according to the partial differential equation; substitute an optical flow initial value of each pixel point in the first image into the target recursive equation to obtain the target optical flow of each pixel point in the first image.

12. The video processing apparatus of any of claims 7-10, wherein, The processing unit is specifically configured to: obtain a first pixel value and a first pixel point coordinate of a first pixel point in the first image; determine a second pixel point coordinate in the first image according to the first pixel point coordinate and the target offset; adjust a pixel value of a second pixel point corresponding to the second pixel point coordinate to the first pixel value.

13. An electronic device, comprising: The device comprises a processor and a memory, the memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to implement the steps of the video processing method in any one of claims 1 to 6.

14. A readable storage medium, characterized by, The readable storage medium stores programs or instructions, and the programs or instructions are executed by the processor to implement the steps of the video processing method in any one of claims 1 to 6.

Images