A data processing method, apparatus, device, medium, and product

Abstract

The application discloses a data processing method, device, equipment, medium and product. The method comprises the following steps: for any pair of adjacent images in a video, the position of a target plane in an image with a later arrangement position is determined according to a plurality of optical flow algorithms and the position of the target plane in an image with an earlier arrangement position, so that the position change of the target plane in the video can be automatically determined in an iterative manner. The plurality of optical flow algorithms are used to determine the motion information of the target plane between the adjacent images, so that the motion information can more accurately represent the position change of the target plane between the adjacent images, and the plane position determined based on the motion information is more accurate. Therefore, the defects caused by the fact that a single optical flow algorithm is difficult to simultaneously ensure the tracking quantity and the tracking quality, such as poor robustness, can be effectively overcome, and the tracking effect is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, medium, or product. Background Technology

[0002] In certain scenarios, such as video processing or augmented reality (AR) scenarios, there is a need to update the content displayed on a common plane (such as a large screen) shared by all frames of a video after acquiring it. This could involve adding new content to the plane or replacing the existing content with something else. To achieve this, the plane's position within each frame of the video could be manually determined by relevant personnel, allowing for subsequent content updates based on that position. However, this method of position determination requires significant manual intervention, resulting in high costs. Summary of the Invention

[0003] To address the aforementioned technical problems, this application provides a data processing method, apparatus, device, medium, or product that can automatically determine or track the positional changes of a plane in a video, thereby reducing costs.

[0004] To achieve the above objectives, the technical solution provided in this application is as follows:

[0005] This application provides a data processing method, the method comprising: acquiring a video, the video including a first image and a second image, wherein the first image is arranged adjacent to the second image in the video, and the first image is arranged earlier than the second image in the video, and both the first image and the second image include a target plane; determining the position of the target plane in the second image based on multiple optical flow algorithms and the position of the target plane in the first image, wherein the multiple optical flow algorithms are used to determine motion information of the target plane between the first image and the second image, and the motion information is used to describe the positional change of the target plane from the first image to the second image.

[0006] In one possible implementation, the plurality of optical flow algorithms include a first algorithm and a second algorithm; the target plane includes a plurality of tracking points; for any tracking point, the process of determining the position of the tracking point in the second image includes: determining a position prediction result of the tracking point in the second image based on the first algorithm, the second image, and the position of the tracking point in the first image; determining a tracking result of the first algorithm at the tracking point based on the position prediction result of the tracking point in the second image, the tracking result indicating whether the tracking point has been tracked from the second image using the first algorithm; in response to the tracking result indicating that the tracking point cannot be tracked from the second image using the first algorithm, determining the position of the tracking point in the second image based on the second algorithm, the second image, and the position of the tracking point in the first image.

[0007] In one possible implementation, the predicted position of the tracking point in the second image is used to indicate the position obtained by the first algorithm performing forward tracking of the tracking point from the first image; the process of determining the tracking result includes: determining the predicted position of the tracking point in the first image based on the first algorithm, the first image, and the predicted position of the tracking point in the second image, wherein the predicted position of the tracking point in the first image is used to indicate the position obtained by the first algorithm performing reverse tracking of the tracking point from the second image; and determining the tracking result based on the distance between the predicted position of the tracking point in the first image and the position of the tracking point in the first image.

[0008] In one possible implementation, the process of determining the tracking result includes: determining the neighboring points of the tracking point in the first image based on the position of the tracking point in the first image, and determining the neighboring points of the tracking point in the second image based on the position prediction result of the tracking point in the second image; and determining the tracking result based on the difference between the neighboring points of the tracking point in the first image and the neighboring points of the tracking point in the second image.

[0009] In one possible implementation, the process of determining the position of the target plane in the second image includes: determining first motion information based on the plurality of optical flow algorithms, the second image, and the position of the target plane in the first image, wherein the first motion information describes the positional change of the target plane from the first image to the second image; solving the homography matrix between the first image and the second image based on the first motion information, wherein the homography matrix describes the positional mapping relationship of the target plane between the first image and the second image; and determining the position of the target plane in the second image based on the homography matrix and the position of the target plane in the first image.

[0010] In one possible implementation, if the first image is not the first frame of the video, the process of determining the homography matrix includes: determining second motion information based on the plurality of optical flow algorithms, the second image, and the position of the target plane in the first frame image, wherein the second motion information is used to describe the positional change of the target plane from the first frame image to the second image; and solving the homography matrix between the first image and the second image based on the first motion information and the second motion information.

[0011] In one possible implementation, the process of determining the second motion information includes: acquiring the viewpoint of the first image; adjusting the first frame image according to the viewpoint of the first image to obtain an adjusted image, wherein the viewpoint of the adjusted image is consistent with the viewpoint of the first image; and determining the second motion information according to the plurality of optical flow algorithms, the second image, and the position of the target plane in the adjusted image.

[0012] In one possible implementation, the viewpoint of the first image is determined based on the position of the target plane in the first image.

[0013] In one possible implementation, the method further includes: updating the video based on the position of the target plane in the first image and the position of the target plane in the second image, so that the content presented on the target plane described by the updated video is different from the content presented on the target plane described by the unupdated video.

[0014] In one possible implementation, the target plane is determined based on a plane selection operation triggered for the first frame of the video.

[0015] This application provides a data processing apparatus, comprising: an acquisition unit for acquiring a video, the video including a first image and a second image, wherein the first image is arranged adjacent to the second image in the video, and the first image is arranged earlier than the second image in the video, and both the first image and the second image include a target plane; and a determination unit for determining the position of the target plane in the second image based on multiple optical flow algorithms and the position of the target plane in the first image, wherein the multiple optical flow algorithms are used to determine motion information of the target plane between the first image and the second image, and the motion information is used to describe the positional change of the target plane from the first image to the second image.

[0016] This application provides an electronic device, the device comprising: a processor and a memory; the memory for storing instructions or computer programs; the processor for executing the instructions or computer programs in the memory, so that the electronic device performs the data processing method provided in this application.

[0017] This application provides a computer-readable medium storing instructions or computer programs that, when executed on a device, cause the device to perform the data processing method provided in this application.

[0018] This application provides a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the data processing method provided in this application.

[0019] Compared with related technologies, this application has at least the following advantages:

[0020] This application provides a planar tracking scheme. For any pair of adjacent images in a video, based on multiple optical flow algorithms and the position of the target plane in the image with the earlier position in the sequence, the position of the target plane in the image with the later position in the sequence is determined. This allows for automatic determination of the target plane's positional changes in the video through an iterative approach. The multiple optical flow algorithms are used to determine the motion information of the target plane between adjacent images, enabling this motion information to more accurately represent the positional changes of the target plane between adjacent images. This results in a more accurate planar position determined based on the motion information. This effectively overcomes the shortcomings of a single optical flow algorithm, which struggles to simultaneously guarantee the number and quality of tracking points, such as poor robustness. Therefore, it enables stable and sufficient tracking of all tracking points within the target plane in the video, thereby improving tracking performance, such as enhancing tracking robustness. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A schematic diagram of a data processing flow provided in an embodiment of this application;

[0023] Figure 2 This is a schematic diagram illustrating content updating for a target plane, provided as an embodiment of this application.

[0024] Figure 3 This is another schematic diagram illustrating content updating for a target plane, provided as an embodiment of this application.

[0025] Figure 4 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0026] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] Research has revealed that in some related solutions, to achieve the requirements shown in the background section, personnel can manually adjust the video frame by frame using video editing tools. However, this manual adjustment method requires a high degree of human intervention, resulting in relatively high costs.

[0028] The research also found that, in order to further reduce costs, optical flow algorithms can be used to automatically determine the positional changes of a plane in a video, thereby achieving the purpose of plane tracking.

[0029] Research has also revealed that a single optical flow algorithm struggles to simultaneously guarantee both the number of tracks tracked and the quality of tracking. For instance, the sparse optical flow (Bruce D. Lucas and Takeo Kanade, KLT) algorithm offers high overall tracking accuracy but has a limited number of tracks, making it prone to missing points during planar tracking. Conversely, dense optical flow or deep optical flow algorithms track a larger number of tracks but achieve lower tracking accuracy than the KLT algorithm.

[0030] Based on the above research, in order to better improve the plane tracking effect, this application provides a data processing method, which includes: firstly acquiring a video, the video including a first image and a second image, wherein the arrangement position of the first image in the video is adjacent to the arrangement position of the second image in the video, and the arrangement position of the first image in the video is earlier than the arrangement position of the second image in the video, and both the first image and the second image include a target plane; then determining the position of the target plane in the second image based on multiple optical flow algorithms and the position of the target plane in the first image, wherein the multiple optical flow algorithms are used to determine the motion information of the target plane between the first image and the second image, so that the motion information is used to describe the positional change of the target plane from the first image to the second image.

[0031] As can be seen, in the planar tracking scheme shown above, for any pair of adjacent images in the video, the position of the target plane in the later images is determined based on multiple optical flow algorithms and the position of the target plane in the earlier images. This iterative approach automatically determines the positional changes of the target plane in the video. The multiple optical flow algorithms are used to determine the motion information of the target plane between adjacent images, allowing this motion information to more accurately represent the positional changes of the target plane between adjacent images. This results in a more accurate plane position determined based on this motion information. This effectively overcomes the shortcomings of a single optical flow algorithm, which struggles to simultaneously guarantee both the number and quality of tracking points, such as poor robustness. Therefore, it enables stable and sufficient tracking of all tracking points within the target plane in the video, thereby improving tracking performance, such as enhancing tracking robustness.

[0032] Furthermore, this application does not limit the entity executing the data processing method. For example, the method can be applied to a terminal device or a server. Alternatively, the method can be implemented through data interaction between the terminal device and the server. The terminal device can be a smartphone, computer, personal digital assistant (PDA), tablet computer, etc. The server can be a standalone server, a cluster server, or a cloud server.

[0033] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0034] To better understand the technical solution provided in this application, the data processing method provided in this application will be explained below with reference to some accompanying drawings. For example... Figure 1 As shown, the data processing method provided in this application embodiment includes S1-S2 below.

[0035] S1: Acquire a video, which includes a first image and a second image. The first image is positioned adjacent to the second image in the video, and the first image is positioned earlier than the second image in the video. Both the first image and the second image include the target plane.

[0036] Here, the first image refers to the image that is relatively early within any pair of adjacent images in the video to be tracked in planar motion; and the second image refers to the image that is relatively late within that pair of adjacent images. For example, if the first image is the i-th frame of the video (e.g., ... Figure 2 or Figure 3 If the image shown is the i-th frame, then the second image refers to the (i+1)-th frame in the video, where i is a positive integer and i+1 ≤ the number of images in the video.

[0037] Furthermore, this application does not limit the method of acquiring the aforementioned video. For example, it can be implemented using any video acquisition method, such as the video acquisition method in an AR scene.

[0038] The target plane refers to the plane described in the video above that requires position tracking, such as a large screen or the tabletop of a dining table. Figure 2 Plane 1 is shown.

[0039] Furthermore, this application does not limit the relationship between the target plane and the video. For example, the relationship could be that the target plane exists in some or all of the images in the video, so that these images can describe the changes in the target plane (e.g., changes in position and shape), thereby enabling the video to represent the changes in the target plane. It should be noted that this application does not limit the cause of this change. For example, the change could be due to the continuous changes in the shooting state (e.g., shooting position and shooting angle) of the video shooting device during the video recording period. It should also be noted that, to better understand the plane tracking scheme provided in this application, the following explanation assumes that each frame of the video includes the target plane.

[0040] In addition, in some scenarios, such as human-computer interaction scenarios, in order to better meet the planar tracking needs of some users, the user can select a planar area as the target plane through human-computer interaction, so that the need can be met by using the target plane as the tracking target in subsequent planar tracking.

[0041] Therefore, in one possible implementation, the target plane can be determined based on a plane selection operation triggered by the first frame of the video, so that subsequent tracking processing based on the target plane can achieve automatic fitting and tracking according to the user-selected plane region. Here, the first frame refers to the image present in the video and located in the first position in the sequence. The plane selection operation is used to indicate to the user which plane region has been selected as the tracking target.

[0042] Based on the above content related to S1, it can be seen that in some scenarios, such as AR scenarios, after acquiring a video that requires planar tracking, the user uses some tools to select the plane to be tracked in the video (e.g., Figure 2 or Figure 3 Plane 1 is shown in the figure so that the position of the plane in each frame of the video can be analyzed using some optical flow algorithms to achieve plane tracking.

[0043] S2: Based on multiple optical flow algorithms and the position of the target plane in the first image, determine the position of the target plane in the second image. The multiple optical flow algorithms are used to determine the motion information of the target plane between the first image and the second image. The motion information is used to describe the positional change of the target plane from the first image to the second image.

[0044] Among them, multiple optical flow algorithms are used to determine the same plane (e.g. Figure 2 or Figure 3 The motion information of plane 1) between two frames of images is shown so that the motion information can represent the positional change of the plane between the two frames of images, thereby enabling the motion information to represent what changes the plane undergoes in one image to obtain the plane in the other image.

[0045] Furthermore, this application does not limit the implementation of the above-mentioned multiple optical flow algorithms. For example, the multiple optical flow algorithms may include a first algorithm (such as the KLT algorithm) and a second algorithm (such as the Deep optical flow algorithm) so that the motion information of the same plane between different images can be determined based on the tracking results given by the two algorithms.

[0046] Furthermore, this application does not limit the relationship between the first algorithm and the second algorithm. For example, the two algorithms can at least satisfy the following constraints: the tracking accuracy of the first algorithm is higher than that of the second algorithm, but the number of tracking points tracked by the first algorithm is less than that tracked by the second algorithm. This is so that in planar tracking, the first algorithm is used primarily to determine some tracking point information, and the second algorithm is used as an auxiliary to supplement the remaining tracking point information in areas that the first algorithm cannot track. Here, tracking accuracy indicates whether the position of the tracking point predicted by a certain optical flow algorithm is accurate. The number of tracking points indicates the number of tracking points predicted by a certain optical flow algorithm.

[0047] Therefore, in one possible implementation, the first algorithm can be the KLT algorithm, and the second algorithm can be either a dense optical flow algorithm or a deep optical flow algorithm. Since the KLT algorithm has the highest overall accuracy but a limited number of tracking points, the deep optical flow algorithm (or dense optical flow algorithm) can be used to supplement it in areas where the KLT algorithm cannot track. This allows the scheme implemented using the KLT algorithm + deep optical flow algorithm (or dense optical flow algorithm) to provide stable and sufficient tracking of tracking points within a certain plane, thus achieving a robust tracking scheme.

[0048] Furthermore, this application does not limit the implementation of the above-described S2. For example, it can specifically be: firstly, using multiple optical flow algorithms, determining the motion information of the target plane between the first image and the second image, so that the motion information can represent the changes that occur between the state of the target plane in the first image and the state of the target plane in the second image (such as position change + shape change, etc.), thereby enabling the motion information to represent what kind of changes to be made to the target plane in the first image to obtain the target plane in the second image; then, based on the motion information and the position of the target plane in the first image, determining the position of the target plane in the second image.

[0049] It should be noted that this application does not limit the method of obtaining the "position of the target plane in the first image". For example, for the first image (such as the i-th frame in a video), if the first image is the first frame of the video (e.g., i=1), the position of the target plane in the first image can be determined based on the plane selection operation triggered by the user for the first frame, so that the target plane can represent the plane selected by the user for tracking processing; however, if the first image is not the first frame of the video (e.g., i≥2), it can be determined that the video also includes the previous frame corresponding to the first image (e.g., the (i-1)-th frame). Therefore, multiple optical flow algorithms can be used to determine the motion information of the target plane between the previous frame and the first image, so that the motion information can represent the positional change of the target plane from the previous frame to the first image; then, based on the motion information and the position of the target plane in the previous frame, the position of the target plane in the first image can be determined. In this context, the previous frame image is positioned adjacent to the first image in the video, and the previous frame image is positioned earlier than the first image in the video.

[0050] It should also be noted that this application does not limit the implementation of the above-described "motion information of the target plane between the first image and the second image". For example, when the target plane includes multiple tracking points, the "motion information of the target plane between the first image and the second image" may include the motion information of each tracking point between the first image and the second image, such as the position offset in each direction (e.g., the offset of UV coordinates). Here, the tracking point refers to a pixel point existing within the target plane that has tracking value. Furthermore, this application does not limit the implementation of these tracking points; for example, they may include some or all of the pixels within the target plane.

[0051] It should also be noted that this application does not limit the implementation of the above-mentioned step of "determining the position of the target plane in the second image based on the motion information and the position of the target plane in the first image". For example, it can adopt any method that can solve the position of the next frame based on motion information and the position of the previous frame, such as by using a pre-built machine learning model with position solving function.

[0052] It should also be noted that this application does not limit the implementation of the above-mentioned step of "using multiple optical flow algorithms to determine the motion information of the target plane between the first image and the second image". For example, it can be: firstly, using each algorithm to process the first image and the second image to obtain the processing results corresponding to each algorithm, so that these processing results can respectively represent the tracking point information obtained by each algorithm; then integrating these processing results (such as calculating the average value) to obtain the motion information of the target plane between the first image and the second image.

[0053] Based on the above S1 to S2, it can be seen that in the planar tracking scheme provided in this application, for any pair of adjacent images in the video, the position of the target plane in the later images is determined based on multiple optical flow algorithms and the position of the target plane in the earlier images. This allows for automatic determination of the target plane's positional changes in the video through an iterative process. The multiple optical flow algorithms are used to determine the motion information of the target plane between adjacent images, enabling this motion information to more accurately represent the positional changes of the target plane between adjacent images. This makes the planar position determined based on the motion information more accurate, effectively overcoming the shortcomings caused by a single optical flow algorithm's difficulty in simultaneously guaranteeing the tracking quantity and quality, such as poor robustness. This allows for stable and sufficient tracking of all tracking points within the target plane in the video, thereby improving tracking performance, such as enhancing tracking robustness.

[0054] In addition, in order to improve the tracking effect, this application also provides a possible implementation of the above S2. In this way, when the above multiple optical flow algorithms include the first algorithm and the second algorithm, and the target plane includes N tracking points, where N is a positive integer, the position of the target plane in the second image may include the position of each tracking point in the second image, and the S2 may include some or all of the steps in steps 11-14 below.

[0055] Step 11: Based on the first algorithm (such as the KLT algorithm), the second image, and the position of the j-th tracking point in the first image, determine the predicted position of the j-th tracking point in the second image. This predicted position is used to indicate the position obtained by the first algorithm through forward tracking of the j-th tracking point from the first image, where j is a positive integer and j≤N. It should be noted that forward tracking refers to planar tracking along the timeline of the video, ensuring that the direction of the planar tracking is the same as the timeline's direction of change.

[0056] The position prediction result of the j-th tracking point in the second image refers to the position determined by the first algorithm for the j-th tracking point from the first image to the second image, so that the position prediction result can be used to describe the predicted position of the j-th tracking point in the second image.

[0057] Furthermore, this application does not limit the implementation of step 11 above. For example, it can be: firstly, using a first algorithm, determining the motion information of the j-th tracking point between the first image and the second image, so that the motion information can represent the predicted positional change that occurs when the j-th tracking point moves from its position in the first image to its position in the second image, thereby enabling the motion information to represent the predicted positional change of the j-th tracking point in the first image to obtain the j-th tracking point in the second image; then, based on the motion information and the position of the j-th tracking point in the first image, determining the position prediction result of the j-th tracking point in the second image, so that the position prediction result can represent the information used by the first algorithm to describe the predicted position of the j-th tracking point in the second image.

[0058] Step 12: Based on the position prediction result of the j-th tracking point in the second image, determine the tracking result of the first algorithm at the j-th tracking point. This tracking result is used to indicate whether the first algorithm has tracked the j-th tracking point from the second image, where j is a positive integer and j≤N.

[0059] The tracking result of the first algorithm at the j-th tracking point is used to describe the tracking performance of the first algorithm at the j-th tracking point, so that the tracking result can indicate whether the first algorithm can accurately track the j-th tracking point from the second image, and thus whether the first algorithm has successfully tracked at the j-th tracking point.

[0060] Furthermore, this application does not limit the implementation of step 12 above. For example, it can be implemented using any method that can evaluate the tracking performance of an optical flow algorithm at each tracking point, such as a method implemented by a machine learning model with tracking performance evaluation function.

[0061] Research has shown that in some scenarios, the tracking quality of an optical flow algorithm can be tested by using the same algorithm for both forward and reverse tracking inference.

[0062] Based on the above research, in order to better improve the tracking effect, step 12 may include steps 121-122 below.

[0063] Step 121: Based on the first algorithm, the first image, and the position prediction result of the j-th tracking point in the second image, determine the position prediction result of the j-th tracking point in the first image, so that the position prediction result can be used to indicate the position obtained by the first algorithm from the second image through reverse tracking of the tracking point. It should be noted that reverse tracking refers to planar tracking against the timeline change direction of the video, so that the direction of the planar tracking is opposite to the timeline change direction.

[0064] The predicted position of the j-th tracking point in the first image refers to the result obtained by the first algorithm using the forward tracking result as a starting point to perform reverse tracking after the position of the j-th tracking point in the second image is derived by the first algorithm through forward tracking. This result describes the predicted position of the j-th tracking point in the first image.

[0065] Furthermore, this application does not limit the implementation of step 121; for example, it may be similar to the implementation of step 11 above.

[0066] Step 122: Based on the distance between the predicted position of the j-th tracking point in the first image and the position of the j-th tracking point in the first image, determine the tracking result of the first algorithm at the j-th tracking point.

[0067] In this application, after deriving the predicted position of the j-th tracking point in the first image using a reverse tracking method with the first algorithm, the distance between the predicted position and the position of the j-th tracking point in the first image is first calculated. This distance is used to represent the tracking accuracy of the first algorithm at the j-th tracking point to a certain extent, so that the smaller the distance, the higher the tracking accuracy. Then, it is determined whether the distance exceeds a preset first threshold. If it does, it can be determined that the tracking accuracy of the first algorithm at the j-th tracking point is relatively poor. Therefore, the preset threshold can be used to determine the accuracy of the tracking point. The first data (such as the string "tracking failed") is determined as the tracking result of the first algorithm at the j-th tracking point, so that the tracking result can indicate that the first algorithm cannot track the j-th tracking point from the second image; if it does not exceed the limit, it can be determined that the tracking accuracy of the first algorithm at the j-th tracking point is relatively good, so the pre-set second data (such as the string "tracking successful") can be determined as the tracking result of the first algorithm at the j-th tracking point, so that the tracking result can indicate that the first algorithm can track the j-th tracking point from the second image.

[0068] Based on the relevant content of steps 121 to 122 above, it can be seen that in some scenarios, the same optical flow algorithm (such as the KLT algorithm) can be used to forward infer the tracking point information between two frames of images and backward infer the tracking point information between two frames of images in order to detect whether the optical flow algorithm can accurately predict the motion information of a certain tracking point between the two frames of images and obtain the tracking result of the optical flow algorithm at the tracking point.

[0069] Research has shown that for the same tracking point, the luminance of the tracking point in different images hardly changes, so that the luminance difference between the tracking point in different images is relatively small. Therefore, the tracking effect of a certain optical flow algorithm on the tracking point can be detected by using the luminance difference.

[0070] The study also found that, for a given tracking point, the luminance of that tracking point in an image can be determined by using the pixels in the image that are relatively close to the tracking point (hereinafter referred to as neighboring points).

[0071] Based on the above research, in order to better improve the tracking effect, step 12 above may include steps 123-124 below.

[0072] Step 123: Based on the position of the j-th tracking point in the first image, determine the neighboring points corresponding to the j-th tracking point in the first image, and based on the position prediction result of the j-th tracking point in the second image, determine the neighboring points corresponding to the j-th tracking point in the second image.

[0073] In this context, the neighboring points of the j-th tracking point in the first image refer to the pixels in the first image that are relatively close to the j-th tracking point, such as the pixels within a small patch area centered on the j-th tracking point in the first image.

[0074] It is evident that the neighboring points corresponding to the j-th tracking point in the first image can at least satisfy the following constraints: the first image includes the neighboring point, and the distance between the position of the neighboring point in the first image and the position of the j-th tracking point in the first image (that is, the position of the j-th tracking point in the first image mentioned above) is less than a preset second threshold. The second threshold can be determined based on the actual application scenario.

[0075] The neighboring points of the j-th tracking point in the second image refer to the pixels in the second image that are relatively close to the j-th tracking point, such as the pixels within a small patch area centered on the j-th tracking point in the second image.

[0076] It can be seen that the neighboring point corresponding to the j-th tracking point in the second image can at least satisfy the following constraints: the second image includes the neighboring point, and the distance between the position of the neighboring point in the second image and the position of the j-th tracking point in the second image (that is, the position described by the above "position prediction result of the j-th tracking point in the second image") is less than a preset second threshold.

[0077] Step 124: Based on the difference between the neighboring points of the j-th tracking point in the first image and the neighboring points of the j-th tracking point in the second image, determine the tracking result of the first algorithm at the j-th tracking point.

[0078] It should be noted that this application does not limit the implementation of step 124 above. For example, it can specifically be: First, based on the neighboring points corresponding to the j-th tracking point in the first image, calculate the luminance (e.g., SSD score) of the j-th tracking point in the first image, and based on the neighboring points corresponding to the j-th tracking point in the second image, calculate the luminance (e.g., SSD score) of the j-th tracking point in the second image; then, calculate the difference between the luminance of the j-th tracking point in the second image and the luminance of the j-th tracking point in the first image, as the luminance difference, so that the luminance difference can, to a certain extent, represent the difference between the neighboring points corresponding to the j-th tracking point in the first image and the neighboring points corresponding to the j-th tracking point in the second image; finally, based on the luminance difference, determine the tracking result of the first algorithm at the j-th tracking point.

[0079] As can be seen, in this application, after obtaining the difference (such as the above-mentioned photometric difference) between the neighboring points corresponding to the j-th tracking point in the first image and the neighboring points corresponding to the j-th tracking point in the second image, it is determined whether the difference exceeds a preset third threshold. If it exceeds the threshold, it can be determined that the tracking accuracy of the first algorithm at the j-th tracking point is relatively poor, so the above-mentioned first data (such as the string "tracking failed") can be determined as the tracking result of the first algorithm at the j-th tracking point; if it does not exceed the threshold, it can be determined that the tracking accuracy of the first algorithm at the j-th tracking point is relatively good, so the above-mentioned second data (such as the string "tracking successful") can be determined as the tracking result of the first algorithm at the j-th tracking point.

[0080] Based on the relevant content of steps 123 to 124 above, it can be seen that in some scenarios, the optical flow algorithm can accurately predict the motion information of a tracking point between two frames of images by calculating the photometric difference between the same tracking point in different images, so as to obtain the tracking result of the optical flow algorithm at the tracking point.

[0081] Furthermore, to improve the results, the tracking result of the optical flow algorithm at the tracking point can be determined jointly by the reasoning methods shown in steps 121 to 122 and steps 123 to 124. Based on this, in one possible implementation, step 12 can specifically be:

[0082] First, based on the first algorithm, the first image, and the position prediction result of the j-th tracking point in the second image, the position prediction result of the j-th tracking point in the first image is determined. Based on the position of the j-th tracking point in the first image, the corresponding neighboring points of the j-th tracking point in the first image are determined. Based on the position prediction result of the j-th tracking point in the second image, the corresponding neighboring points of the j-th tracking point in the second image are also determined. Then, based on the distance between the position prediction result of the j-th tracking point in the first image and the position of the j-th tracking point in the first image, and the difference between the corresponding neighboring points of the j-th tracking point in the first image and the corresponding neighboring points of the j-th tracking point in the second image, the tracking result of the first algorithm at the j-th tracking point is determined.

[0083] As can be seen, in this application, after obtaining the distance between the predicted position of the j-th tracking point in the first image and the position of the j-th tracking point in the first image, and the difference between the neighboring points of the j-th tracking point in the first image and the neighboring points of the j-th tracking point in the second image, if the distance does not exceed the first threshold and the difference does not exceed the third threshold, it can be determined that the tracking accuracy of the first algorithm at the j-th tracking point is relatively good, so the second data (such as the string "tracking successful") can be determined as the tracking result of the first algorithm at the j-th tracking point; conversely, it can be determined that the tracking accuracy of the first algorithm at the j-th tracking point is relatively poor, so the first data (such as the string "tracking failed") can be determined as the tracking result of the first algorithm at the j-th tracking point.

[0084] Based on the relevant content of step 12 above, after obtaining the position prediction result of the j-th tracking point in the second image by using the first algorithm to perform forward tracking from the first image, the tracking result of the first algorithm at the j-th tracking point can be determined based on the position prediction result, so that the tracking result can indicate whether the first algorithm can accurately track the j-th tracking point from the second image.

[0085] Step 13: In response to the above tracking result indicating that the first algorithm cannot track the j-th tracking point from the second image, the position of the j-th tracking point in the second image is determined based on the second algorithm (such as the Deep optical flow algorithm), the second image, and the position of the j-th tracking point in the first image, where j is a positive integer and j≤N.

[0086] In this application, after obtaining the tracking result of the first algorithm at the j-th tracking point, if the tracking result is used to indicate that the j-th tracking point cannot be tracked from the second image using the first algorithm, it can be determined that the position prediction result of the j-th tracking point determined by the first algorithm in the second image is inaccurate. Therefore, other optical flow algorithms (such as Deep) can be used to supplement the reasoning of the position of the j-th tracking point in the second image.

[0087] It should be noted that this application does not limit the implementation of step 13 above. For example, it can specifically be: first, using the second algorithm to determine the motion information of the j-th tracking point between the first image and the second image; then, based on the motion information and the position of the j-th tracking point in the first image, determining the position of the j-th tracking point in the second image.

[0088] Step 14: In response to the above tracking result indicating that the j-th tracking point has been tracked from the second image using the first algorithm, the predicted position of the j-th tracking point in the second image is determined as the position of the j-th tracking point in the second image, where j is a positive integer and j≤N.

[0089] In this application, after obtaining the tracking result of the first algorithm at the j-th tracking point, if the tracking result is used to indicate that the first algorithm can accurately track the j-th tracking point from the second image, then it can be determined that the position prediction result of the j-th tracking point determined by the first algorithm in the second image is relatively accurate. Therefore, the position prediction result can be directly used as the position of the j-th tracking point in the second image.

[0090] Based on the relevant content of steps 11 to 14 above, it can be seen that this application can use a combination of multiple optical flow algorithms to perform planar tracking, so as to obtain the positions of all tracking points in the same plane in each frame of the video, thus achieving a stable and sufficient tracking effect.

[0091] Research revealed that for the motion information of the target plane derived using multiple optical flow algorithms between the first and second images, the large number of tracking points within the target plane results in a substantial amount of motion data, making the planar position calculation based on this motion information complex and costly. Furthermore, the motion information predicted for individual tracking points may contain errors, leading to more errors in the planar position calculation based on these tracking points, thus slightly reducing the accuracy of the final result.

[0092] Based on the above research, in order to better solve the problems shown in the previous paragraph, this application also provides a possible implementation of the above S2, in which S2 may include steps 21-23 below.

[0093] Step 21: Based on multiple optical flow algorithms, the second image, and the position of the target plane in the first image, determine the first motion information so that the first motion information is the motion information of the target plane between the first image and the second image, thereby making the first motion information used to describe the positional change of the target plane from the first image to the second image.

[0094] It should be noted that this application does not limit the implementation of the first motion information. For example, when the target plane includes multiple tracking points, the first motion information may include the motion information of each tracking point between the first image and the second image. The motion information of the j-th tracking point between the first image and the second image is used to describe the positional change of the j-th tracking point from the first image to the second image, where j is a positive integer and j≤N.

[0095] It should also be noted that the implementation method of step 21 above can be found in the relevant content above.

[0096] Step 22: Based on the first motion information, solve for the homography matrix between the first image and the second image. This homography matrix is ​​used to describe the positional mapping relationship of the target plane between the first image and the second image.

[0097] The homography matrix is ​​used to describe the relationship between the pixel positions of points in the real world in different images, such as transformation relationships, so that the homography matrix can describe the position transformation relationship of the same point (such as each tracking point) between two frames of images.

[0098] Furthermore, this application does not limit the implementation of step 22 above. For example, it can be implemented using any method that can solve the homography matrix based on the motion information of multiple points, such as by using some pre-set derivation formulas or a pre-built machine learning model with homography matrix solving function.

[0099] Step 23: Based on the homography matrix and the position of the target plane in the first image, determine the position of the target plane in the second image.

[0100] It should be noted that this application does not limit the implementation of step 23.

[0101] Based on steps 21 to 23 above, after inferring the motion information of each tracking point in the target plane between the first and second images using multiple optical flow algorithms, a homography matrix with a smaller data volume and higher accuracy can be calculated based on this motion information. Then, the position of the target plane in the second image can be calculated based on this homography matrix. Since the data volume of the homography matrix is ​​much smaller than that of the motion information, this overcomes the shortcomings caused by the large data volume of the motion information. Furthermore, since the homography matrix is ​​obtained by fitting the motion information of multiple tracking points, the positional changes described by the homography matrix are more accurate, thus making the plane tracking based on the homography matrix more accurate, which is beneficial for improving the plane tracking effect.

[0102] Research has revealed that when performing planar tracking on a video, for any pair of adjacent images in the video, the position of the plane in the later image is inferred based on its position in the earlier image. This means that planar tracking for the video is obtained through iterative inference of the planar position. Consequently, when performing planar position inference on images that are later in the video, the accumulated errors may cause the tracking area to shift, thus affecting the tracking performance.

[0103] Based on the above research, in order to better improve the tracking effect, this application also provides a possible implementation of step 22 above. In this way, when the first image is not the first frame image of the video, step 22 may include steps 221-222 below.

[0104] Step 221: Based on multiple optical flow algorithms, the second image, and the position of the target plane in the first frame image, determine the second motion information so that the second motion information is the motion information of the target plane between the first frame image and the second image, thereby making the second motion information used to describe the positional change of the target plane from the first frame image to the second image.

[0105] It should be noted that this application does not limit the implementation of the second motion information. For example, when the target plane includes multiple tracking points, the second motion information may include the motion information of each tracking point between the first frame image and the second image. Specifically, the motion information of the j-th tracking point between the first frame image and the second image describes the positional change of the j-th tracking point from the first frame image to the second image, where j is a positive integer and j≤N.

[0106] It should also be noted that the process of determining the second motion information mentioned above is similar to the process of determining the first motion information mentioned above, and will not be repeated here for the sake of brevity.

[0107] Step 222: Based on the first motion information and the second motion information, solve for the homography matrix between the first image and the second image.

[0108] It should be noted that this application does not limit the implementation of step 222 above. For example, it can specifically be: first, perform homography matrix solving based on the first motion information to obtain a first matrix, so that the first matrix can describe the positional mapping relationship of the target plane between the first image and the second image; and perform homography matrix solving based on the second motion information to obtain a second matrix, so that the second matrix can describe the positional mapping relationship of the target plane between the first image and the second image; then integrate the first matrix and the second matrix (e.g., by weighted summation) to obtain the homography matrix between the first image and the second image.

[0109] It should be noted that this application does not limit the solution process of the second matrix. For example, it can specifically be as follows: first, perform homography matrix solving based on the second motion information to obtain the homography matrix between the first frame image and the second image, so that the homography matrix can describe the positional mapping relationship of the target plane between the first frame image and the second image; then, based on the homography matrix between the first frame image and the second image, and the difference between the homography matrix between the first frame image and the first image, obtain the second matrix, so that the second matrix can describe the positional mapping relationship of the target plane between the first image and the second image. Here, the "homography matrix between the first frame image and the first image" is used to describe the positional mapping relationship of the target plane between the first frame image and the first image; moreover, this application does not limit the method of obtaining the "homography matrix between the first frame image and the first image," for example, the "homography matrix between the first frame image and the first image" can be obtained during the planar tracking process for the first image. For example, the "homography matrix between the first frame image and the first image" can be obtained by solving the homography matrix for the motion information of the target plane between the first frame image and the first image. Moreover, the process of determining the "motion information of the target plane between the first frame image and the first image" is similar to the process of determining the first motion information mentioned above. For the sake of brevity, it will not be described again here.

[0110] For example, step 222 above can specifically be: solving the homography matrix based on the set between the first motion information and the second motion information to obtain the homography matrix between the first image and the second image.

[0111] Based on the relevant content of steps 221 to 222 above, in some scenarios, in order to solve the problem of tracking area offset after long-term tracking, the homography matrix between the first image and the second image can be solved based on the position of the target plane in the first frame of the video and the motion information of the target plane between the first image and the second image. This homography matrix can satisfy the characteristics of the target plane described by the first frame of the video, such as the relationship between different tracking points in the target plane. This allows the offset to be corrected as much as possible with the help of the first frame, thus effectively improving the tracking effect.

[0112] Research has found that for two frames of images that are far apart in a video, the large difference between the shooting conditions (such as shooting angle) of these two frames can cause them to exhibit significant differences in some dimensions (such as shooting angle), which can easily affect the solution of the homography matrix.

[0113] Based on the above research, in order to improve the effect, this application also provides a process for determining the second motion information, which may specifically include steps 31 to 33 below.

[0114] Step 31: Obtain the viewpoint of the first image so that the viewpoint is used to describe the angle at which the target plane is presented in the first image.

[0115] It should be noted that this application does not limit the method of obtaining the perspective of the first image. For example, it can be implemented using any method that can obtain the perspective of an image, such as directly reading the shooting perspective from the image capturing device.

[0116] Furthermore, to improve accuracy, the viewpoint of the first image can be determined based on the position of the target plane within the first image, so that the viewpoint can more accurately describe the angle at which the target plane is presented in the first image. It should be noted that this application does not limit the implementation method to this determination. For example, it can employ any method capable of inferring the shooting viewpoint of a plane based on its position, such as using a pre-built machine learning model with shooting viewpoint prediction capabilities or using some pre-defined inference formulas.

[0117] Step 32: Adjust the first frame image according to the perspective of the first image to obtain the adjusted image so that the perspective of the adjusted image is consistent with the perspective of the first image.

[0118] It should be noted that this application does not limit the implementation of "maintaining consistency". For example, it can refer to being the same or highly similar (such as having a similarity greater than a preset similarity threshold).

[0119] It should also be noted that this application does not limit the implementation of step 32 above. For example, it can be implemented using any method that can adjust the image according to a certain perspective, such as by using a pre-built machine learning model with image adjustment function, or by using some pre-set inference formulas, or any method for implementing image warping.

[0120] As can be seen, in one possible implementation, step 32 can specifically be as follows: First, based on the difference between the viewpoint of the first image and the viewpoint of the first frame image, a transformation matrix is ​​determined so that the transformation matrix can describe the relationship between the image presented under the viewpoint of the first image and the image presented under the viewpoint of the first frame image, such as the transformation relationship between pixel positions; then, the first frame image is adjusted according to the transformation matrix to obtain an adjusted image so that the viewpoint of the adjusted image is consistent with the viewpoint of the first image, thereby making the viewpoint of the adjusted image as close as possible to the viewpoint of the second image. This can effectively overcome the defect of poor homography matrix solution caused by the large viewpoint difference between the two frames.

[0121] Step 33: Based on multiple optical flow algorithms, the second image, and the position of the target plane in the adjusted image, determine the second motion information so that the second motion information can be used to describe the positional change of the target plane from the adjusted image to the second image.

[0122] It should be noted that this application does not limit the method of determining the "position of the target plane in the adjusted image". For example, if the adjusted image is obtained by adjusting the first frame image according to the above transformation matrix, the position of the target plane in the adjusted image can be obtained by adjusting the position of the target plane in the first frame image according to the transformation matrix.

[0123] It should also be noted that this application does not limit the implementation of step 33 above. For example, the implementation of step 33 is similar to the implementation of step 221 above.

[0124] Based on the relevant content of steps 31 to 33 above, in some scenarios, in order to solve the defects caused by the large differences in some dimensions (such as shooting angle) between two frames that are too far apart, the state (such as the shooting angle) of the first frame image in the video can be adjusted first to obtain an adjusted image. This adjusted image can describe the content described by the first frame image, such as the characteristics of the target plane, in a state that is closer to the second image (such as the state of the first image). Then, based on the position of the target plane in the adjusted image and the motion information of the target plane between the first and second images, the homography matrix between the first and second images is solved. This homography matrix can more accurately describe the positional mapping relationship of the target plane between the first and second images, thereby making the plane tracking based on the homography matrix more accurate.

[0125] In addition, in some scenarios, in order to better meet some user needs (such as the need to embed content on a certain plane), the above data processing method may also include step 41 below.

[0126] Step 41: Update the video based on the position of the target plane in the first image and its position in the second image (e.g., ...). Figure 2 or Figure 3 (The update shown) is to make the content presented on the target plane described by the updated video different from the content presented on the target plane described by the unupdated video.

[0127] It should be noted that this application does not limit the implementation method of the above-mentioned "updating video" step. For example, it can specifically be: for any frame of the video, after obtaining the position of the target plane in the image, the content within the planar region defined by that position can be updated in the image, such as adding new content to the planar region (e.g., Figure 2 (as shown in the image) or directly replace the existing content within the planar area with other content (such as...) Figure 3 The update process (such as the content replacement process shown) is designed to ensure that the updated content fits perfectly into the planar area, thereby improving the content update effect, such as the content embedding effect.

[0128] Based on the above data processing methods, the planar tracking scheme provided in this application has the following advantages: Firstly, this application achieves a stable and sufficient number of tracking schemes by using a combination of multiple optical flow algorithms, which improves tracking robustness. Secondly, this application simplifies the planar position calculation process by using a homography matrix, reducing the complexity of planar position calculation. Thirdly, this application achieves long-distance stable tracking by fusing neighbor information (such as information described by the first image) and prior information (such as information described by the first frame image), which improves the long-term stability of planar tracking and thus expands the application scope of this planar tracking scheme.

[0129] Based on the data processing method provided in the embodiments of this application, the embodiments of this application also provide a data processing apparatus, which will be described below in conjunction with... Figure 4 Explanation and clarification will be provided. Among them, Figure 4 This is a schematic diagram of a data processing apparatus provided in an embodiment of this application. It should be noted that for technical details of the data processing apparatus provided in this application embodiment, please refer to the relevant content of the data processing method above.

[0130] like Figure 4 As shown, the data processing apparatus 400 provided in this embodiment includes:

[0131] Acquisition unit 401 is used to acquire video, the video including a first image and a second image, the first image being arranged in a position adjacent to the second image in the video, the first image being arranged in a position earlier than the second image in the video, and both the first image and the second image including a target plane;

[0132] The determining unit 402 is used to determine the position of the target plane in the second image based on multiple optical flow algorithms and the position of the target plane in the first image. The multiple optical flow algorithms are used to determine the motion information of the target plane between the first image and the second image. The motion information is used to describe the positional change of the target plane from the first image to the second image.

[0133] In one possible implementation, the plurality of optical flow algorithms include a first algorithm and a second algorithm; the target plane includes a plurality of tracking points; the determining unit 402 is specifically configured to: for any tracking point, determine a position prediction result of the tracking point in the second image based on the first algorithm, the second image, and the position of the tracking point in the first image; determine a tracking result of the first algorithm at the tracking point based on the position prediction result of the tracking point in the second image, wherein the tracking result indicates whether the tracking point has been tracked from the second image using the first algorithm; and, in response to the tracking result indicating that the tracking point cannot be tracked from the second image using the first algorithm, determine the position of the tracking point in the second image based on the second algorithm, the second image, and the position of the tracking point in the first image.

[0134] In one possible implementation, the predicted position of the tracking point in the second image is used to indicate the position obtained by the first algorithm performing forward tracking of the tracking point from the first image; the determining unit 402 is specifically used to: determine the predicted position of the tracking point in the first image based on the first algorithm, the first image, and the predicted position of the tracking point in the second image, wherein the predicted position of the tracking point in the first image is used to indicate the position obtained by the first algorithm performing reverse tracking of the tracking point from the second image; and determine the tracking result based on the distance between the predicted position of the tracking point in the first image and the position of the tracking point in the first image.

[0135] In one possible implementation, the determining unit 402 is specifically configured to: determine the neighboring points of the tracking point in the first image based on the position of the tracking point in the first image, and determine the neighboring points of the tracking point in the second image based on the position prediction result of the tracking point in the second image; and determine the tracking result based on the difference between the neighboring points of the tracking point in the first image and the neighboring points of the tracking point in the second image.

[0136] In one possible implementation, the determining unit 402 is specifically configured to: determine first motion information based on the plurality of optical flow algorithms, the second image, and the position of the target plane in the first image, wherein the first motion information describes the positional change of the target plane from the first image to the second image; solve for the homography matrix between the first image and the second image based on the first motion information, wherein the homography matrix describes the positional mapping relationship of the target plane between the first image and the second image; and determine the position of the target plane in the second image based on the homography matrix and the position of the target plane in the first image.

[0137] In one possible implementation, the determining unit 402 is specifically configured to: if the first image is not the first frame of the video, determine second motion information based on the plurality of optical flow algorithms, the second image, and the position of the target plane in the first frame image, wherein the second motion information is used to describe the positional change of the target plane from the first frame image to the second image; and solve the homography matrix between the first image and the second image based on the first motion information and the second motion information.

[0138] In one possible implementation, the determining unit 402 is specifically configured to: acquire the viewpoint of the first image; adjust the first frame image according to the viewpoint of the first image to obtain an adjusted image, wherein the viewpoint of the adjusted image is consistent with the viewpoint of the first image; and determine the second motion information according to the plurality of optical flow algorithms, the second image, and the position of the target plane in the adjusted image.

[0139] In one possible implementation, the viewpoint of the first image is determined based on the position of the target plane in the first image.

[0140] In one possible implementation, the data processing apparatus 400 further includes:

[0141] An update unit is configured to update the video based on the position of the target plane in the first image and the position of the target plane in the second image, so that the content presented on the target plane described in the updated video is different from the content presented on the target plane described in the unupdated video.

[0142] In one possible implementation, the target plane is determined based on a plane selection operation triggered for the first frame of the video.

[0143] Based on the aforementioned data processing device 400, its working principle includes: for any pair of adjacent images in a video, determining the position of the target plane in the subsequent images based on multiple optical flow algorithms and the position of the target plane in the preceding images. This allows for automatic determination of the target plane's positional changes in the video through an iterative process. The multiple optical flow algorithms are used to determine the motion information of the target plane between adjacent images, enabling this motion information to more accurately represent the positional changes of the target plane between adjacent images. This results in a more accurate plane position determined based on the motion information, effectively overcoming the shortcomings of a single optical flow algorithm that struggles to simultaneously guarantee both tracking quantity and quality, such as poor robustness. This allows for stable and sufficient tracking of all tracking points within the target plane in the video, thereby improving tracking performance, such as enhancing tracking robustness.

[0144] In addition, this application also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory so that the electronic device performs any implementation of the data processing method provided in this application.

[0145] See Figure 5 The diagram illustrates a structural schematic of an electronic device 500 suitable for implementing embodiments of the present disclosure. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0146] like Figure 5As shown, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0147] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0148] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0149] The electronic device provided in this embodiment belongs to the same inventive concept as the method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0150] This application also provides a computer-readable medium storing instructions or a computer program that, when executed on a device, causes the device to perform any implementation of the data processing method provided in this application.

[0151] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0152] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0153] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0154] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, enable the electronic device to perform the aforementioned methods.

[0155] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0157] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units / modules do not necessarily limit the specific unit itself.

[0158] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0159] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0160] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0161] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0162] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0163] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0164] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, characterized in that, The method includes: Acquire a video, the video including a first image and a second image, the first image being arranged adjacent to the second image in the video, the first image being arranged earlier than the second image in the video, and both the first image and the second image including a target plane; Based on multiple optical flow algorithms and the position of the target plane in the first image, the position of the target plane in the second image is determined. The multiple optical flow algorithms are used to determine the motion information of the target plane between the first image and the second image. The motion information is used to describe the positional change of the target plane from the first image to the second image.

2. The method according to claim 1, characterized in that, The plurality of optical flow algorithms include a first algorithm and a second algorithm; The target plane includes multiple tracking points; For any of the tracking points, the process of determining the position of the tracking point in the second image includes: Based on the first algorithm, the second image, and the position of the tracking point in the first image, the predicted position of the tracking point in the second image is determined; Based on the position prediction result of the tracking point in the second image, the tracking result of the first algorithm in the tracking point is determined, and the tracking result is used to indicate whether the tracking point has been tracked from the second image using the first algorithm; In response to the tracking result indicating that the tracking point cannot be tracked from the second image using the first algorithm, the position of the tracking point in the second image is determined based on the second algorithm, the second image, and the position of the tracking point in the first image.

3. The method according to claim 2, characterized in that, The predicted position of the tracking point in the second image is used to indicate the position obtained by the first algorithm from the first image through forward tracking of the tracking point; The process of determining the tracking result includes: Based on the first algorithm, the first image, and the position prediction result of the tracking point in the second image, the position prediction result of the tracking point in the first image is determined. The position prediction result of the tracking point in the first image is used to indicate the position obtained by the first algorithm from the second image through reverse tracking of the tracking point. The tracking result is determined based on the distance between the predicted position of the tracking point in the first image and the position of the tracking point in the first image.

4. The method according to claim 2, characterized in that, The process of determining the tracking result includes: Based on the position of the tracking point in the first image, determine the neighboring points of the tracking point in the first image; and based on the position prediction result of the tracking point in the second image, determine the neighboring points of the tracking point in the second image. The tracking result is determined based on the difference between the neighboring points of the tracking point in the first image and the neighboring points of the tracking point in the second image.

5. The method according to claim 1, characterized in that, The process of determining the position of the target plane in the second image includes: Based on the multiple optical flow algorithms, the second image, and the position of the target plane in the first image, first motion information is determined. The first motion information is used to describe the positional change of the target plane from the first image to the second image. Based on the first motion information, the homography matrix between the first image and the second image is solved. The homography matrix is ​​used to describe the positional mapping relationship of the target plane between the first image and the second image. The position of the target plane in the second image is determined based on the homography matrix and the position of the target plane in the first image.

6. The method according to claim 5, characterized in that, If the first image is not the first frame of the video, the process of determining the homography matrix includes: Based on the multiple optical flow algorithms, the second image, and the position of the target plane in the first frame image, second motion information is determined. The second motion information is used to describe the positional change of the target plane from the first frame image to the second image. Based on the first motion information and the second motion information, the homography matrix between the first image and the second image is solved.

7. The method according to claim 6, characterized in that, The process of determining the second motion information includes: Obtain the viewpoint of the first image; Based on the perspective of the first image, the first frame image is adjusted to obtain an adjusted image, wherein the perspective of the adjusted image is consistent with that of the first image. The second motion information is determined based on the multiple optical flow algorithms, the second image, and the position of the target plane in the adjusted image.

8. The method according to claim 7, characterized in that, The viewpoint of the first image is determined based on the position of the target plane in the first image.

9. The method according to claim 1, characterized in that, The method further includes: The video is updated based on the position of the target plane in the first image and the position of the target plane in the second image, so that the content presented on the target plane in the updated video is different from the content presented on the target plane in the video before the update.

10. The method according to any one of claims 1-9, characterized in that, The target plane is determined based on a plane selection operation triggered for the first frame of the video.

11. A data processing apparatus, characterized in that, include: An acquisition unit is used to acquire a video, the video including a first image and a second image, wherein the first image is arranged adjacent to the second image in the video, the first image is arranged before the second image in the video, and both the first image and the second image include a target plane; The determining unit is used to determine the position of the target plane in the second image based on multiple optical flow algorithms and the position of the target plane in the first image. The multiple optical flow algorithms are used to determine the motion information of the target plane between the first image and the second image. The motion information is used to describe the positional change of the target plane from the first image to the second image.

12. An electronic device, characterized in that, The device includes: a processor and a memory; The memory is used to store instructions or computer programs; The processor is configured to execute the instructions or computer program in the memory to cause the electronic device to perform the method according to any one of claims 1-10.

13. A computer-readable medium, characterized in that, The computer-readable medium stores instructions or computer programs that, when executed on the device, cause the device to perform the method according to any one of claims 1-10.

14. A computer program product, characterized in that, It includes a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the method according to any one of claims 1-10.

Images