Image processing method and apparatus, device, and medium

Abstract

Embodiments of the present disclosure relate to an image processing method and apparatus, a device, and a medium. The method comprises: acquiring a first target image frame and a second target image frame in a target video, wherein the first target image frame is a previous image frame of the second target image frame; acquiring grayscale information of a target point in a first target planar region in the first target image frame; and on the basis of the grayscale information of the target point, determining a target mapping relationship between the second target image frame and the first target image frame, and on the basis of the target mapping relationship, determining a second target planar region in the second target image frame, wherein the second target planar region is matched with the first target planar region.

Description

Image processing method, device, equipment and medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 202310779515.5, filed on June 30, 2023, entitled “Image processing method, device, equipment and medium”. The entire contents of that application are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, apparatus, device, and medium. Background Art

[0004] In the field of image processing, in order to perform edge lighting effects, content replacement, and other processing on a specific planar area in a video (such as a billboard, photo frame, or screen), it is necessary to track the position of the planar area within the video frame. However, the planar tracking methods provided by related technologies suffer from high processing loss and inaccurate tracking results.

[0005] Summary of the Invention

[0006] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides an image processing method, apparatus, device and medium.

[0007] An embodiment of the present disclosure provides an image processing method, the method comprising: acquiring a first target frame image and a second target frame image in a target video; wherein the first target frame image is a previous frame image of the second target frame image; acquiring grayscale information of a target point in a first target plane area in the first target frame image; determining a target mapping relationship between the second target frame image and the first target frame image based on the grayscale information of the target point, so as to determine a second target plane area in the second target frame image based on the target mapping relationship; wherein the second target plane area matches the first target plane area.

[0008] Optionally, obtaining the first target frame image and the second target frame image in the target video includes: performing feature point matching on adjacent frame images in the target video to obtain feature point matching results; and taking adjacent frame images whose matching errors corresponding to the feature point matching results are greater than a preset error threshold as the first target frame image and the second target frame image.

[0009] Optionally, the target point is a point in the first target plane area whose grayscale gradient is greater than a preset gradient threshold.

[0010] Optionally, the grayscale information includes grayscale and grayscale gradient; determining the target mapping relationship between the second target frame image and the first target frame image based on the grayscale information of the target point includes: determining the corresponding point of the target point in the second target frame image based on a preset initial mapping relationship between the second target frame image and the first target frame image, and obtaining the grayscale of the corresponding point; performing an adjustment operation on the initial mapping relationship according to the grayscale gradient of the target point and the difference between the grayscale of the target point and the grayscale of the corresponding point until a target mapping relationship is obtained; wherein, the adjustment operation is used to reduce the difference.

[0011] Optionally, performing an adjustment operation on the initial mapping relationship until a target mapping relationship is obtained includes: performing an adjustment operation a specified number of times based on the initial mapping relationship, so as to determine a target mapping relationship according to a mapping relationship obtained after the adjustment is performed a specified number of times.

[0012] Optionally, determining the second target plane area in the second target frame image based on the target mapping relationship includes: obtaining a first mapping relationship between the base frame image of the target video and the first target frame image; obtaining a second mapping relationship between the base frame image and the second target frame image based on the target mapping relationship and the first mapping relationship; determining the second target plane area in the second target frame image based on the base target plane area in the base frame image and the second mapping relationship; wherein the base target plane area, the first target plane area and the second target plane area all correspond to the same plane.

[0013] Optionally, obtaining the second mapping relationship between the reference frame image and the second target frame image based on the target mapping relationship and the first mapping relationship includes: determining the estimated mapping relationship between the reference frame image and the second target frame image based on the target mapping relationship and the first mapping relationship; obtaining a projected frame image obtained by reprojecting the second target frame image onto the reference frame image based on the estimated mapping relationship; obtaining correction information of the estimated mapping relationship based on the projected frame image and the reference frame image; and correcting the estimated mapping relationship using the correction information to obtain the second mapping relationship between the reference frame image and the second target frame image.

[0014] Optionally, obtaining correction information of the estimated mapping relationship based on the projection frame image and the reference frame image includes: adjusting the brightness of the projection frame image so that the adjusted brightness matches the brightness of the reference frame image; and obtaining correction information of the estimated mapping relationship based on the projection frame image and the reference frame image after brightness adjustment.

[0015] Optionally, the correction information of the estimated mapping relationship is obtained based on the projection frame image after brightness adjustment and the reference frame image, including: determining the mapping relationship between the projection frame image after brightness adjustment and the reference frame image according to the grayscale and grayscale gradient of the target point in the reference target plane area in the reference frame image; and obtaining the correction information of the estimated mapping relationship based on the mapping relationship between the projection frame image after brightness adjustment and the reference frame image.

[0016] An embodiment of the present disclosure also provides an image processing device, including: a frame image acquisition module, used to acquire a first target frame image and a second target frame image in a target video; wherein the first target frame image is a previous frame image of the second target frame image; a grayscale information acquisition module, used to acquire grayscale information of a target point in a first target plane area in the first target frame image; a plane area determination module, used to determine a target mapping relationship between the second target frame image and the first target frame image based on the grayscale information of the target point, so as to determine a second target plane area in the second target frame image based on the target mapping relationship; wherein the second target plane area matches the first target plane area.

[0017] An embodiment of the present disclosure further provides an electronic device, comprising: a storage device storing a computer program; and a processing device for executing the computer program in the storage device to implement the steps of any one of the above-mentioned image processing methods.

[0018] An embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the image processing method provided by the embodiment of the present disclosure.

[0019] The above technical solution provided by the embodiment of the present disclosure can obtain the grayscale information of the target point of the first target plane area in the first target frame image (i.e., in the previous frame), and determine the target mapping relationship between the second target frame image (i.e., in the subsequent frame) and the first target frame image based on the grayscale information of the target point, so as to determine the second target plane area in the second target frame image that matches the first target plane area based on the target mapping relationship. Compared with the method of using traditional deep learning algorithms for plane tracking, the above method requires less power consumption and lower processing loss; moreover, the above method has better effect of plane tracking using grayscale information, and the tracking result is more accurate, especially it can better guarantee the plane registration accuracy of image frames with characteristics such as blur or weak texture.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0022] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] FIG1 is a schematic flow chart of an image processing method provided by an embodiment of the present disclosure;

[0024] FIG2 is a schematic flow chart of an image processing method provided by an embodiment of the present disclosure;

[0025] FIG3 is a schematic diagram of an image processing flow provided by an embodiment of the present disclosure;

[0026] FIG4 is a schematic structural diagram of an image processing device provided by an embodiment of the present disclosure;

[0027] FIG5 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0029] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0030] FIG1 is a flow chart of an image processing method provided by an embodiment of the present disclosure. The method can be performed by an image processing device, wherein the device can be implemented using software and / or hardware and can generally be integrated into an electronic device. As shown in FIG1 , the method mainly includes the following steps S102 to S106:

[0031] Step S102 : Acquire a first target frame image and a second target frame image in a target video; wherein the first target frame image is a frame image preceding the second target frame image.

[0032] The embodiments of the present disclosure do not restrict the video content of the target video. For example, the target video may contain a plane area. The first target frame image and the second target frame image are two adjacent frames in the target video. In some embodiments, any two adjacent frames in the target video can be used as the first target frame image and the second target frame image, respectively. In other embodiments, adjacent frames that meet specified conditions can be used as the first target frame image and the second target frame image. For example, a special frame image such as a blurred frame image or a frame image with weak texture in the target video can be used as the second target frame image, and the previous frame image can be used as the first target frame image. In addition, after the target plane area of ​​a special frame image is determined using the method provided by the embodiments of the present disclosure, the special frame image can continue to serve as the first target frame image to determine the target plane area of ​​the frame image following the special frame image. In addition, other methods such as feature point matching algorithms can be used to first determine the target plane area of ​​each frame image in the target video, and then select adjacent frames with poor plane area registration results as the first target frame image and the second target frame image. The selection method of the first target frame image and the second target frame image can be flexibly set according to needs and is not limited here.

[0033] Step S104: Obtain grayscale information of a target point in a first target plane area in the first target frame image. Exemplarily, the grayscale information may include grayscale and grayscale gradient.

[0034] The disclosed embodiments do not restrict the shape of the target plane area. For example, it can be a quadrilateral plane such as a common screen or plaque, or other polygonal planes or irregular planes such as triangles that appear in daily life. The first target plane area in the first target frame image is the plane to be tracked in the target video. The number of first target plane areas can be one or more, and there is no limit here. It will be understood that the first target plane area in the first target frame image is known. The disclosed embodiments primarily track the corresponding plane area in the second target frame image based on the position of the first target plane area in the first target frame image. The disclosed embodiments do not restrict the target points. For example, the target points can be all points in the first target plane area, multiple specified points, multiple random points, or multiple points distributed in a preset manner, and are not limited to feature points. This makes it more suitable for blurred frames or frames with weak textures. To better ensure plane tracking accuracy, in some specific examples, the target points can be points with a high grayscale gradient, such as points in the first target plane area whose grayscale gradient exceeds a preset gradient threshold. The preset gradient threshold can be flexibly set according to needs. In practical applications, methods such as the Canny operator can be used to determine target points with strong gradients in the first target plane area, and record the grayscale information (grayscale and grayscale gradient) of the target points. It can be understood that even blurred images or images with weak textures usually contain more points with a certain grayscale gradient. Compared with the method of using only feature points for plane tracking, the method of plane tracking based on the grayscale information of points proposed in the embodiment of the present disclosure is more universal.

[0035] Step S106, determining a target mapping relationship between the second target frame image and the first target frame image based on the grayscale information of the target point, so as to determine a second target plane area in the second target frame image based on the target mapping relationship; wherein the second target plane area matches the first target plane area.

[0036] In practical applications, the initial mapping relationship between two adjacent frame images can be preset first, and then the initial mapping relationship can be adjusted and iterated multiple times until the target mapping relationship is obtained, wherein the second target plane area determined based on the target mapping relationship matches the first target plane area, that is, corresponds to the same plane, which is the specified plane to be tracked in the target video.

[0037] Compared to traditional deep learning algorithms for plane tracking, this approach consumes less power and has lower processing overhead. Furthermore, it utilizes grayscale information for plane tracking, resulting in more accurate tracking results. This approach can particularly effectively ensure plane registration accuracy for image frames with characteristics such as blur or weak textures. The disclosed embodiments can effectively ensure plane tracking accuracy while consuming less power.

[0038] In order to further reduce power consumption and lower the requirements for device performance, the embodiment of the present disclosure provides an implementation example of obtaining a first target frame image and a second target frame image in a target video, including: performing feature point matching on adjacent frame images in the target video to obtain a feature point matching result; and using adjacent frame images whose matching errors corresponding to the feature point matching results are greater than a preset error threshold as the first target frame image and the second target frame image. The embodiment of the present disclosure does not limit the feature point matching method. For example, feature point matching can be performed on adjacent frame images in the target video based on a corner detection algorithm and a sparse optical flow algorithm. In some specific implementation examples, the corner detection algorithm can be a Shi-Tomasi corner detection algorithm, and the sparse optical flow algorithm can be a KLT sparse optical flow algorithm. In actual applications, other corner detection algorithms and / or other sparse optical flow algorithms can also be used, which are not limited here.

[0039] It is understandable that for most image frames, a good registration effect can be achieved by using feature point matching, that is, the target plane can be matched more accurately. However, for special frame images with fewer or less obvious feature points, such as blurred frame images or weak texture frame images, the plane matching error obtained by using feature point matching is large, and the target plane in the subsequent frame obtained by tracking does not match the target plane in the previous frame. That is, the position of the target plane in the subsequent frame determined based on the feature points of the previous frame is not accurate. In order to balance the efficiency and accuracy of plane tracking, the embodiment of the present disclosure can first use feature point matching to perform plane tracking on adjacent frame images in the target video, thereby improving the plane tracking efficiency, reducing the required power consumption and reducing the requirements for device performance. Then, based on the size of the matching error of the feature point matching result, the frame images that need to be re-matched are selected to re-perform region matching based on the grayscale information. For example, a gradient descent method can be used to optimize the grayscale difference of the target plane area of ​​adjacent frame images (i.e., the first target frame image and the second target frame image), so as to achieve an accurate region matching effect with greater robustness. On the basis that grayscale information includes grayscale and grayscale gradient, the embodiment of the present disclosure provides a specific implementation example of determining the target mapping relationship between the second target frame image and the first target frame image according to the grayscale information of the target point, which can be performed with reference to the following steps (1) and (2):

[0040] Step (1) determines the corresponding point of the target point in the second target frame image based on the preset initial mapping relationship between the second target frame image and the first target frame image, and obtains the grayscale of the corresponding point. In practical applications, the mapping relationship can be represented in the form of a matrix, and specifically, a unit three-dimensional matrix can be pre-set as the initial mapping relationship. When the initial mapping relationship between the two frames of images and the target point of the first target frame image are known, the corresponding point of the target point in the second target frame image can be directly obtained. It should be noted that the initial mapping relationship is only pre-set, so it is not accurate. At this time, the corresponding point determined in the second target frame image is also not accurate, but the initial mapping relationship can be iteratively adjusted multiple times, that is, the position of the corresponding point can be adjusted multiple times until a more accurate corresponding point can be found.

[0041] Step (2) performs an adjustment operation on the initial mapping relationship according to the grayscale gradient of the target point and the difference between the grayscale of the target point and the grayscale of the corresponding point until a target mapping relationship is obtained; wherein the adjustment operation is used to reduce the difference. That is, the adjustment can be made starting from the initial mapping relationship in the direction of reducing the difference. Each time the adjustment is made, the corresponding corresponding point and the difference between the grayscale of the target point and the grayscale of the corresponding point are re-determined based on the adjusted mapping relationship, and then the mapping relationship is adjusted again based on the newly determined difference until a target mapping relationship that meets the requirements is obtained.

[0042] In some implementation examples, a specified number of adjustment operations are performed based on the initial mapping relationship to determine a target mapping relationship based on the mapping relationship obtained after the specified number of adjustments. For example, the mapping relationship obtained after the specified number of adjustments can be directly used as the target mapping relationship. The specified number of adjustments can be set based on empirical values, and the mapping relationship obtained after the specified number of adjustment operations can approximate the accurate mapping relationship between the first target frame image and the second target frame image.

[0043] Exemplarily, a target point P with a strong gradient in the first target plane area in the first target frame image is known, and the grayscale value gradient of the target point P is obtained. First, the mapping relationship H between the first target frame image and the second target frame image can be initialized, which can be represented by a unit three-dimensional matrix. Then, the mapping relationship H is updated and iterated K times to gradually reduce the grayscale difference between the target point P of the first target frame image and the corresponding point P' of the second target frame image. In specific implementation, the corresponding point P' of the second target frame image is H*P. Assuming that the grayscale of P' in the second target frame image is I', and the grayscale of P in the first target frame image is I, then the grayscale difference D between P' and P is I'-I. In some implementation examples, the formula H'=H+α*D can be updated according to the mapping relationship. T*D*G*d(P′) is used to iteratively update the mapping relationship H between the first target frame image and the second target frame image. For each adjustment operation, H in the above formula is the mapping relationship before the update, H' is the mapping relationship after the update, α is the learning rate, which can be set to 0.1 for example, and d(P′) represents the differential of P′. Specifically, the mapping relationship can be represented by a homography matrix with a total of 8 parameters. The differential of P′ represents the offset of P′ when each of the 8 parameters is slightly changed by a small value (change amount). The change amount can be obtained by calculation. For example, H = [h1,h2,.....h8,1]. Assuming that the coordinates of P are (x, y), the coordinates of P′ are: (h1*x+h2*y+h3) / (h7*x+h8*y+1), (h4*x+h5*y+h6) / (h7*x+h8*y+1). The above formula can be used to derive h1, h2, h3...h8, and the above update formula can be further used to iterate H multiple times until the latest H obtained after K cycles is the target mapping relationship between the first target frame image and the second target frame image.

[0044] After the target mapping relationship is determined in the above manner, the second target plane area in the second target frame image can be determined based on the target mapping relationship. In some embodiments, the target mapping relationship can be directly used as the mapping relationship between the first target frame image and the second target frame image, and the second target plane area in the second target frame image can be determined based on the first target plane area in the first target frame image and the target mapping relationship. In other embodiments, taking into account the problems such as cumulative errors that may exist in the plane tracking process, the reference frame image of the target video can be further introduced, and the mapping relationship between the second target frame image and the reference frame image can be determined based on the target mapping relationship, so as to more accurately determine the second target plane area in the second target frame image. For example, it can be performed with reference to the following steps A to C:

[0045] Step A, obtaining a first mapping relationship between the reference frame image of the target video and the first target frame image. Exemplarily, the reference frame image can be the first frame image of the target video. In practical applications, based on the known first target plane area of ​​the first target frame image and the reference target plane area of ​​the reference frame image, the first mapping relationship can be directly obtained, wherein the first target plane area of ​​the first target frame image is also obtained by tracking the target plane area of ​​the previous frame image, and so on. In other words, by recursively calculating the mapping relationship between adjacent frame images in the target video, the first mapping relationship between the reference frame image of the target video and the first target frame image can be more accurately known.

[0046] Step B, based on the target mapping relationship and the first mapping relationship, obtain the second mapping relationship between the reference frame image and the second target frame image. Since the target mapping relationship is the mapping relationship between the first target frame image and the second target frame image, and the first mapping relationship is the mapping relationship between the first target frame image and the reference frame image, based on the target mapping relationship and the first mapping relationship, the second mapping relationship between the reference frame image and the second target frame image can be reasonably and reliably obtained. In order to fully guarantee the accuracy of the second mapping relationship, illustratively, the following steps B1 to B4 can be referred to for execution:

[0047] Step B1: Determine an estimated mapping relationship between the reference frame image and the second target frame image based on the target mapping relationship and the first mapping relationship. It is understood that the disclosed embodiments fully account for issues such as cumulative errors, which may result in the target mapping relationship between the first target frame image and the second target frame image being acquired containing certain errors and not being very precise. Therefore, the mapping relationship between the reference frame image and the second target frame image determined based on the target mapping relationship and the first mapping relationship is also an estimate, not necessarily very precise, and requires further adjustment.

[0048] Step B2: Based on the estimated mapping relationship, a projection frame image is obtained by reprojecting the second target frame image onto the reference frame image. The reprojected projection frame image may also be referred to as the reprojected second target frame image.

[0049] Step B3: Obtain correction information for the estimated mapping relationship based on the projected frame image and the reference frame image. In practical applications, the projected frame image and the reference frame image can be compared to determine the correction information for the estimated mapping relationship. In some specific implementation examples, the following steps B3.1 to B3.2 can be referred to:

[0050] Step B3.1, adjust the brightness of the projected frame image so that the adjusted brightness matches the brightness of the reference frame image. That is, the difference between the brightness of the projected frame image after adjustment and the brightness of the reference frame image is less than the preset brightness threshold, that is, the brightness difference between the projected frame image and the reference frame image is small. Exemplarily, a method such as histogram matching can be used to adjust the brightness of the projected frame image to be basically consistent with the brightness of the reference frame image. The embodiment of the present disclosure fully takes into account that there may be differences in the brightness of different frame images, especially for non-adjacent frame images, the brightness difference may be large. Therefore, the embodiment of the present disclosure can first adjust the brightness of the projected frame image, and adjust the brightness of the reference frame of the projected frame image to be basically consistent, so as to effectively eliminate the influence of brightness on grayscale.

[0051] Step B3.2: Based on the brightness-adjusted projected frame image and the reference frame image, correction information for the estimated mapping relationship is obtained. In practical applications, a mapping relationship between the brightness-adjusted projected frame image and the reference frame image can be obtained. This mapping relationship can serve as correction information for the aforementioned estimated mapping relationship. Specifically, it can be represented in matrix form, also referred to as a correction matrix. The mapping relationship between the brightness-adjusted projected frame image and the reference frame image can be determined using the aforementioned grayscale information-based method to ensure the accuracy of the mapping relationship between the brightness-adjusted projected frame image and the reference frame image. That is, in some specific implementation examples, the mapping relationship between the brightness-adjusted projected frame image and the reference frame image can be determined based on the grayscale and grayscale gradient of the target point in the reference target plane region of the reference frame image. Then, based on the mapping relationship between the brightness-adjusted projected frame image and the reference frame image, correction information for the estimated mapping relationship is obtained. In other words, the reference frame image and the brightness-adjusted projected frame image are aligned using the aforementioned grayscale information-based method to obtain a more accurate mapping relationship between the two.

[0052] In step B4, the estimated mapping relationship is corrected using the correction information to obtain a second mapping relationship between the reference frame image and the second target frame image. For example, the correction information is represented by a correction matrix, and the estimated mapping relationship is represented by an estimated mapping matrix. Multiplying the correction matrix by the estimated mapping matrix yields a result matrix that corrects the estimated mapping matrix. This result matrix represents the second mapping relationship between the reference frame image and the second target frame image. This method eliminates accumulated errors and produces a more accurate and reliable second mapping relationship between the reference frame image and the second target frame image.

[0053] Step C, determining the second target plane area in the second target frame image based on the baseline target plane area in the baseline frame image and the second mapping relationship; wherein the baseline target plane area, the first target plane area and the second target plane area all correspond to the same plane, for example, all correspond to the same plane in the target scene, and the target scene is the scene presented by the target video.

[0054] Since the second mapping relationship is more accurate and can effectively eliminate cumulative errors, the position of the second target plane area obtained based on this is also more accurate, achieving an accurate and reliable plane tracking effect.

[0055] Furthermore, based on the above, the embodiment of the present disclosure further provides a flowchart of an image processing method as shown in FIG2 , which mainly includes the following steps S202 to S214:

[0056] Step S202 : Acquire a first target frame image and a second target frame image in a target video; wherein the first target frame image is a frame image preceding the second target frame image.

[0057] Step S204 , obtaining the grayscale and grayscale gradient of the target point in the first target plane area in the first target frame image.

[0058] Step S206 : Based on the preset initial mapping relationship between the second target frame image and the first target frame image, a corresponding point of the target point in the second target frame image is determined, and the grayscale of the corresponding point is obtained.

[0059] Step S208 , performing a specified number of adjustment operations on the initial mapping relationship according to the grayscale gradient of the target point and the difference between the grayscale of the target point and the grayscale of the corresponding point, and using the mapping relationship obtained after the specified number of adjustments as the target mapping relationship.

[0060] Step S210 : Acquire a first mapping relationship between a reference frame image and a first target frame image of a target video.

[0061] Step S212 : acquiring a second mapping relationship between the reference frame image and the second target frame image according to the target mapping relationship and the first mapping relationship.

[0062] Step S214 : determining a second target plane area in the second target frame image according to the reference target plane area in the reference frame image and the second mapping relationship.

[0063] The specific implementation of the above steps can refer to the aforementioned related content and will not be repeated here. Through the above method, compared with the traditional deep learning method, the required power consumption is lower, the device performance requirements are not high, and the mapping relationship between adjacent frame images can be reasonably determined based on the grayscale information. On this basis, a reference frame image can be introduced to finally determine the second mapping relationship between the reference frame image and the second target frame image, thereby achieving a more accurate plane tracking effect.

[0064] For ease of understanding, you can also refer to an image processing flow diagram shown in Figure 3. In this flow diagram, the first target frame image is represented by the n-1th frame, and the second target frame image is represented by the nth frame. First, the target mapping relationship H(n) between the n-1th frame and the nth frame is determined by inter-frame alignment, and then the first mapping relationship H*(n-1) between the reference frame (such as the first frame of the video) and the n-1th frame is obtained. The target mapping relationship H(n) is multiplied by the first mapping relationship H*(n-1) to obtain the estimated mapping relationship H*(n)' between the reference frame and the nth frame. The estimated mapping relationship H*(n)' is further corrected twice using the reference frame image (for details, refer to the aforementioned steps B2 to B4), and finally the second mapping relationship H*(n) between the reference frame and the nth frame is obtained.

[0065] Through the above method, not only can a more accurate inter-frame registration effect be achieved, but also precise registration can be achieved even for blurred frame images or weakly textured frame images. Moreover, the introduction of the reference frame image for secondary correction on this basis can effectively improve the problem of cumulative error and achieve a better plane tracking effect in an integrated manner.

[0066] Corresponding to the aforementioned image processing method, FIG4 is a schematic structural diagram of an image processing device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated into an electronic device. As shown in FIG4 , the device includes:

[0067] The frame image acquisition module 402 is configured to acquire a first target frame image and a second target frame image in a target video; wherein the first target frame image is a frame image preceding the second target frame image;

[0068] Grayscale information acquisition module 404, used to acquire grayscale information of a target point in a first target plane area in a first target frame image;

[0069] The plane area determination module 406 is used to determine the target mapping relationship between the second target frame image and the first target frame image based on the grayscale information of the target point, so as to determine the second target plane area in the second target frame image based on the target mapping relationship; wherein the second target plane area matches the first target plane area.

[0070] Compared with the method of using traditional deep learning algorithms for plane tracking, the above-mentioned device requires less power consumption, has lower processing loss, and does not require high equipment performance. Moreover, the above-mentioned method uses grayscale information for plane tracking more effectively, and the tracking results are more accurate, especially it can better ensure the plane registration accuracy of image frames with characteristics such as blur or weak texture.

[0071] In some embodiments, the frame image acquisition module 402 is specifically used to: perform feature point matching on adjacent frame images in the target video to obtain feature point matching results; and use adjacent frame images whose matching errors corresponding to the feature point matching results are greater than a preset error threshold as the first target frame image and the second target frame image.

[0072] In some embodiments, the target point is a point in the first target plane area having a grayscale gradient greater than a preset gradient threshold.

[0073] In some embodiments, the grayscale information includes grayscale and grayscale gradient; the plane area determination module 406 is specifically used to: determine the corresponding point of the target point in the second target frame image based on a preset initial mapping relationship between the second target frame image and the first target frame image, and obtain the grayscale of the corresponding point; according to the grayscale gradient of the target point and the difference between the grayscale of the target point and the grayscale of the corresponding point, perform an adjustment operation on the initial mapping relationship until a target mapping relationship is obtained; wherein, the adjustment operation is used to reduce the difference.

[0074] In some implementations, the plane area determination module 406 is specifically configured to: perform a specified number of adjustment operations based on the initial mapping relationship, so as to determine a target mapping relationship according to the mapping relationship obtained after the specified number of adjustments.

[0075] In some embodiments, the plane area determination module 406 is specifically used to: obtain a first mapping relationship between the base frame image of the target video and the first target frame image; obtain a second mapping relationship between the base frame image and the second target frame image based on the target mapping relationship and the first mapping relationship; determine the second target plane area in the second target frame image based on the base target plane area in the base frame image and the second mapping relationship; wherein the base target plane area, the first target plane area and the second target plane area all correspond to the same plane.

[0076] In some embodiments, the plane area determination module 406 is specifically used to: determine the estimated mapping relationship between the reference frame image and the second target frame image based on the target mapping relationship and the first mapping relationship; obtain the projection frame image obtained by reprojecting the second target frame image onto the reference frame image based on the estimated mapping relationship; obtain correction information of the estimated mapping relationship based on the projection frame image and the reference frame image; and use the correction information to correct the estimated mapping relationship to obtain a second mapping relationship between the reference frame image and the second target frame image.

[0077] In some embodiments, the plane area determination module 406 is specifically used to: adjust the brightness of the projected frame image so that the adjusted brightness matches the brightness of the reference frame image; and obtain correction information of the estimated mapping relationship based on the projection frame image after brightness adjustment and the reference frame image.

[0078] In some embodiments, the plane area determination module 406 is specifically used to: determine the mapping relationship between the projection frame image after brightness adjustment and the reference frame image based on the grayscale and grayscale gradient of the target point in the reference target plane area in the reference frame image; and obtain correction information of the estimated mapping relationship based on the mapping relationship between the projection frame image after brightness adjustment and the reference frame image.

[0079] The image processing device provided by the embodiments of the present disclosure can execute the image processing method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0080] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0081] An embodiment of the present disclosure provides an electronic device, which includes: a storage device storing a computer program; and a processing device configured to execute the computer program in the storage device to implement the steps of any one of the methods in the present disclosure.

[0082] Reference is now made to FIG5 , which illustrates a schematic diagram of the structure of an electronic device 500 suitable for implementing embodiments of the present disclosure. Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device illustrated in FIG5 is merely an example and should not limit the functionality or scope of use of embodiments of the present disclosure.

[0083] As shown in Figure 5, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0084] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although FIG5 shows the electronic device 500 with various devices, it should be understood that not all of the devices shown are required to be implemented or present. More or fewer devices may alternatively be implemented or present.

[0085] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0086] In addition to the above-mentioned method and device, the embodiment of the present disclosure may also be a computer program product, which includes computer program instructions. When the computer program instructions are executed by a processor, the processor executes the image processing method provided by the embodiment of the present disclosure.

[0087] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0088] In addition, the embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor is enabled to execute the image processing method provided by the embodiment of the present disclosure.

[0089] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0090] The embodiment of the present disclosure further provides a computer program product, including a computer program / instruction, which implements the image processing method in the embodiment of the present disclosure when executed by a processor.

[0091] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0092] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0093] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0094] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0095] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0096] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, comprising: Acquire a first target frame image and a second target frame image in a target video; wherein the first target frame image is a frame image preceding the second target frame image; Acquire grayscale information of a target point in a first target plane area in the first target frame image; A target mapping relationship between the second target frame image and the first target frame image is determined according to the grayscale information of the target point, so as to determine a second target plane area in the second target frame image based on the target mapping relationship; wherein the second target plane area matches the first target plane area.

2. The method according to claim 1, wherein: The step of acquiring a first target frame image and a second target frame image in a target video includes: Performing feature point matching on adjacent frame images in the target video to obtain a feature point matching result; Adjacent frame images whose matching errors corresponding to the feature point matching results are greater than a preset error threshold are used as the first target frame image and the second target frame image.

3. The method according to claim 1, wherein: The target point is a point in the first target plane area whose grayscale gradient is greater than a preset gradient threshold.

4. The method according to claim 1, wherein: The grayscale information includes grayscale and grayscale gradient; and determining the target mapping relationship between the second target frame image and the first target frame image according to the grayscale information of the target point includes: Based on a preset initial mapping relationship between the second target frame image and the first target frame image, determining a corresponding point of the target point in the second target frame image, and acquiring a grayscale of the corresponding point; According to the grayscale gradient of the target point and the difference between the grayscale of the target point and the grayscale of the corresponding point, an adjustment operation is performed on the initial mapping relationship until a target mapping relationship is obtained; wherein the adjustment operation is used to reduce the difference.

5. The method according to claim 4, wherein: The performing an adjustment operation on the initial mapping relationship until a target mapping relationship is obtained includes: A specified number of adjustment operations are performed based on the initial mapping relationship to determine a target mapping relationship according to a mapping relationship obtained after the specified number of adjustments.

6. The method according to any one of claims 1 to 5, wherein: The determining a second target plane area in the second target frame image based on the target mapping relationship includes: Acquire a first mapping relationship between a reference frame image of the target video and the first target frame image; Acquire a second mapping relationship between the reference frame image and the second target frame image according to the target mapping relationship and the first mapping relationship; According to the reference target plane area in the reference frame image and the second mapping relationship, a second target plane area in the second target frame image is determined; wherein the reference target plane area, the first target plane area and the second target plane area all correspond to the same plane.

7. The method according to claim 6, wherein: The acquiring, according to the target mapping relationship and the first mapping relationship, a second mapping relationship between the reference frame image and the second target frame image comprises: Determining an estimated mapping relationship between the reference frame image and the second target frame image according to the target mapping relationship and the first mapping relationship; Based on the estimated mapping relationship, obtaining a projection frame image obtained by reprojecting the second target frame image onto the reference frame image; Acquire correction information of the estimated mapping relationship according to the projection frame image and the reference frame image; The estimated mapping relationship is corrected using the correction information to obtain a second mapping relationship between the reference frame image and the second target frame image.

8. The method according to claim 7, wherein: The obtaining, according to the projection frame image and the reference frame image, correction information of the estimated mapping relationship includes: Adjusting the brightness of the projection frame image so that the adjusted brightness matches the brightness of the reference frame image; Based on the projection frame image after brightness adjustment and the reference frame image, correction information of the estimated mapping relationship is obtained.

9. The method according to claim 8, wherein: The obtaining correction information of the estimated mapping relationship based on the projection frame image after brightness adjustment and the reference frame image includes: Determining a mapping relationship between the projection frame image after brightness adjustment and the reference frame image according to the grayscale and grayscale gradient of the target point in the reference target plane area in the reference frame image; Based on the mapping relationship between the projection frame image after brightness adjustment and the reference frame image, correction information of the estimated mapping relationship is obtained.

10. An image processing device, comprising: A frame image acquisition module, used to acquire a first target frame image and a second target frame image in a target video; wherein the first target frame image is a frame image preceding the second target frame image; A grayscale information acquisition module, used to acquire grayscale information of a target point in a first target plane area in the first target frame image; A plane area determination module is used to determine a target mapping relationship between the second target frame image and the first target frame image according to the grayscale information of the target point, so as to determine a second target plane area in the second target frame image based on the target mapping relationship; wherein the second target plane area matches the first target plane area.

11. An electronic device, comprising: a storage device having a computer program stored thereon; A processing device, used to execute the computer program in the storage device to implement the steps of the image processing method according to any one of claims 1 to 9.

12. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the image processing method according to any one of claims 1 to 9.

Images