Image processing apparatus and method for view angle conversion

Abstract

The invention discloses an image processing device and method for view angle conversion. The image processing device for view angle conversion comprises an image acquisition circuit, a memory and a processor. The image acquisition circuit is configured to capture a previous image and a current image. The processor is configured to process a plurality of instructions to perform the following operations: feature extraction is performed on a previous image and a current image to calculate an optical flow information matrix from a first feature set of the previous image and a second feature set of the current image; calculating the position of a second key point set in the previous image in the current image by using the optical flow information matrix, and storing the position as a third key point set; calculating a current homography matrix between the first key point set and the third key point set; and projecting the target image to a corresponding position on the current image by using the current homography matrix.

Description

Technical Field

[0001] The present disclosure relates to an image processing device and method. Specifically, the present invention relates to an image processing device and method for perspective conversion. Background Art

[0002] In general image processing for perspective conversion, in order to find the correspondence between images of different perspectives, there are two current practices: manual and automatic methods.

[0003] However, the manual method requires manual pre-processing of different images, but this method may sometimes be inaccurate due to human errors (eg, hand shaking of the user), and the manual method is also very time-consuming and labor-intensive.

[0004] In addition, although the automatic method can currently use computer vision and deep learning to obtain the homography matrix. However, the current method is very time-consuming to process an image and consumes a lot of computing resources. Because it uses deep learning, it will not be possible to fine-tune the internal parameters, which may result in an incorrect homography matrix when the light and shadow scenery of the application site is not similar to the training data.

[0005] Furthermore, the above existing methods often need to recalculate the perspective conversion relationship (ie, a new homography matrix) from the new image when the shooting position may change, which will consume more computing resources.

[0006] Therefore, how to greatly reduce the amount of calculation when the shooting position may change and solve the problem of rapid changes in viewing angle is an urgent problem to be solved by those skilled in the art. Summary of the invention

[0007] The present disclosure proposes an image processing device for perspective conversion, including an image acquisition circuit, a memory, and a processor. The image acquisition circuit is used to capture a previous image and a current image; the memory is used to store multiple instructions, a target image in a field template, and a first key point set in the field template; and the processor is used to process multiple instructions to perform the following operations: perform feature extraction operations on the previous image and the current image to calculate an optical flow information matrix from the first feature set of the previous image and the second feature set of the current image; use the optical flow information matrix to infer the position of the second key point set in the previous image in the current image to store it as a third key point set; calculate the current homography matrix between the first key point set and the third key point set; and use the current homography matrix to project the target image to a corresponding position on the current image.

[0008] The present disclosure also proposes an image method for perspective conversion, which includes: capturing a previous image and a current image, and storing a target image in a field template and a first key point set in the field template; performing feature extraction operations on the previous image and the current image to calculate an optical flow information matrix from the first feature set of the previous image and the second feature set of the current image; using the optical flow information matrix to infer the position of the second key point set in the previous image in the current image to store it as a third key point set; calculating a current homography matrix between the first key point set and the third key point set; and using the current homography matrix to project the target image to a corresponding position on the current image.

[0009] Based on the above, the image device and method for perspective conversion provided by the present disclosure can use the optical flow method to update the homography matrix, so as to greatly reduce the amount of calculation for regenerating the homography matrix. In addition, the homography matrix can be updated in real time between different frames, and this approach greatly increases the sense of smoothness after projecting to continuous images. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A schematic diagram showing an image processing device in some embodiments of the present disclosure;

[0011] Figure 2 A flowchart showing an image processing method in some embodiments of the present disclosure;

[0012] Figure 3 A schematic diagram showing the generation of a binary image in some embodiments of the present disclosure;

[0013] Figure 4 A schematic diagram showing the generation of multiple line segments from a field image in some embodiments of the present disclosure;

[0014] Figure 5 A schematic diagram showing the angles of line segments in some embodiments of the present disclosure;

[0015] Figure 6 A schematic diagram showing the maximum distance in some embodiments of the present disclosure;

[0016] Figure 7 A schematic diagram showing a plurality of fusion lines in a field image in some embodiments of the present disclosure;

[0017] Figure 8 A schematic diagram showing the sorting of the first group in some embodiments of the present disclosure;

[0018] Fig. 9 A schematic diagram showing the sorting of the second group in some embodiments of the present disclosure;

[0019] Fig.10 A schematic diagram showing a field template in some embodiments of the present disclosure;

[0020] Fig.11 A schematic diagram showing selection of multiple fusion lines in some embodiments of the present disclosure;

[0021] Fig.12 A schematic diagram showing selection of two horizontal site lines and two vertical site lines in some embodiments of the present disclosure;

[0022] Fig.13 A schematic diagram showing a plurality of conversion lines on a field template in some embodiments of the present disclosure;

[0023] Fig.14 A schematic diagram showing selection of multiple fusion lines in some embodiments of the present disclosure;

[0024] Fig.15 A schematic diagram showing selection of two horizontal site lines and two vertical site lines in some embodiments of the present disclosure;

[0025] Fig.16 A schematic diagram showing a plurality of conversion lines on a field template in some embodiments of the present disclosure;

[0026] Fig.17 A schematic diagram showing a plurality of pixel points of a field line on a field template in some embodiments of the present disclosure;

[0027] Fig.18 A schematic diagram showing a plurality of conversion points on a field image in some embodiments of the present disclosure;

[0028] Fig.19 A schematic diagram showing a stacking template in some embodiments of the present disclosure;

[0029] Fig. 20 A schematic diagram showing superimposed images in some embodiments of the present disclosure;

[0030] Fig.21 A flowchart showing an image processing method for perspective conversion in some embodiments of the present disclosure;

[0031] Fig. 22 A schematic diagram showing the generation of an optical flow map in some embodiments of the present disclosure;

[0032] Fig.23 A schematic diagram showing a field template in some other embodiments of the present disclosure;

[0033] Fig.24 A schematic diagram showing a second key point set and a third key point set in some embodiments of the present disclosure;

[0034] Fig.25 A schematic diagram showing the perspective conversion relationship in some embodiments of the present disclosure;

[0035] Fig.26 A schematic diagram showing an updated third keypoint set in some embodiments of the present disclosure.

[0036] Description of Reference Numerals

[0037] 100: Image processing device

[0038] 110: Image acquisition circuit

[0039] 120: Memory

[0040] 130: Processor

[0041] IMG1: Field Image

[0042] S210~S250, S2110~S2150: Steps

[0043] BIMG: Binary Image

[0044] LS1~LS12:Line segment

[0045] L1, L2: Line segment

[0046] EL: Extension line

[0047] θ: line segment angle

[0048] d: maximum distance

[0049] IL1~IL10, VL1~VL5, HL1~HL5: Fusion line

[0050] IMG2, IMG3: images

[0051] LO, UO: Reference point

[0052] LD1~LD5, UD1~UD5: minimum distance

[0053] TEM1, TEM: Field Template

[0054] TO: origin

[0055] TL1~TL4: Horizontal site line

[0056] TL5~TL9: vertical site line

[0057] IP1~IP4、TP1~TP4:intersection point

[0058] TSL1~TSL10: Conversion line

[0059] P: Pixels

[0060] TSP: Transition Point

[0061] OG1~OG3: site line segment

[0062] TEM2: Template to be superimposed

[0063] AD: Advertising marker

[0064] SIMG: Superimposed Images

[0065] C_IMG: current image

[0066] P_IMG: Previous image

[0067] OF_MAP: Optical flow map

[0068] KP: Key Point

[0069] IMG_O, IMG_O': Image

[0070] O, O': Optical center

[0071] x, x': pixel points

[0072] X: Field point

[0073] r: ray

[0074] l, l': epipolar lines

[0075] e, e': antipodes

[0076] l, l': epipolar lines

[0077] FKP: Filter Key Points DETAILED DESCRIPTION

[0078] In view of the current general image processing for perspective transformation, it may require a lot of manpower for pre-processing, resulting in a lot of manpower consumption or the inability to fine-tune the training parameters in real time. In addition, since it relies on human judgment, it often causes errors in the image after perspective transformation processing.

[0079] To solve the above problems, the present disclosure proposes an image processing device and method, which greatly increases the accuracy of the calculated homography matrix and can make fine adjustments to the changes in the venue in real time through color gamut conversion, line finding processing, line segment fusion, angle difference and distance difference methods, and methods for calculating the number of overlaps and non-overlaps. Therefore, this high-precision homography matrix can be used in image processing with various perspective changes. The above technology of the present disclosure is specifically described by taking the following embodiments as examples.

[0080] Reference Figure 1 , Figure 1 FIG. 1 is a schematic diagram showing an image processing device 100 in some embodiments of the present disclosure. Figure 1As shown, the image processing device 100 includes an image acquisition circuit 110 , a memory 120 , and a processor 130 . The processor 130 is connected between the image acquisition circuit 110 and the memory 120 .

[0081] In some embodiments, the image processing device 100 may be any device having an image processor, or a device for providing image processing functions, such as a computer, a server, or a processing center, etc., which is not limited here.

[0082] In this embodiment, the image acquisition circuit 110 is used to capture a field image IMG1 (for example, an image of a tennis court or a football field). In some embodiments, the image acquisition circuit 110 can be implemented by various image acquirers, such as a photographic circuit or a photosensitive circuit, etc., to capture an image of any field (for example, a tennis court, a football field, etc.) within a preset shooting range (field of view, FoV). In some embodiments, the field image IMG1 can be an image in an RGB color space. In some embodiments, the image acquisition circuit 110 can capture a field (for example, a tennis court or a football field) from a specific shooting angle to generate a field image IMG1.

[0083] In addition, in the present embodiment, the memory 120 is used to store a plurality of instructions, which are used to execute the detailed steps described in the subsequent paragraphs. In some embodiments, these instructions can be corresponding software or firmware instruction programs. In some embodiments, the memory 120 can be implemented by a storage unit, a flash memory, a read-only memory, a hard disk, or any equivalent storage component.

[0084] In addition, in this embodiment, the processor 130 is used to process these instructions and execute the detailed steps described in the following subsequent paragraphs. In some embodiments, the processor 130 can be implemented by a processing unit, a central processing unit or a computing unit.

[0085] Refer to Figure 2 , Figure 2 A flowchart showing an image processing method in some embodiments of the present disclosure is shown. The image processing method is applicable to an electronic device, which may be Figure 1 The image processing device 100, Figure 1 The components in the image processing device 100 are used to perform steps S210 to S250 in the image processing method. Figure 2 As shown, first, in step S210 , a plurality of line segments are found from the field image IMG1 .

[0086] In some embodiments, the field image IMG1 may be converted from the RGB color space to an image in the HSV color space, and the image in the HSV color space may be converted into a binary image according to a plurality of color intervals. Then, a line detection operation may be performed on the binary image to generate a plurality of line segments. In some embodiments, the line detection operation may be a Hough transform or a fast line detection (i.e., a HoughLines function or a FastLineDetector function in an openCV library).

[0087] In some embodiments, the plurality of color intervals include a hue interval, a saturation interval, and a brightness interval. In some embodiments, the binary image may be a black and white image. In some embodiments, a line detection operation may be performed on the binary image to generate a plurality of line segments' respective starting point coordinates and end point coordinates. In some embodiments, the plurality of line segments may be marked on the field image IMG1 according to the plurality of line segments' respective starting point coordinates and end point coordinates.

[0088] The following is an actual example of how to generate a binary image. Figure 3 , Figure 3 Schematic diagram showing the generation of a binary image BIMG in some embodiments of the present disclosure. Figure 3 As shown, assuming that the hue range is 0-180, the saturation range is 0-100, and the brightness range is 180-255, the field image IMG1 can be first converted from the RGB color space to the HSV color space.

[0089] When the hue of a pixel in an image in the HSV color space is between 0 and 180, the saturation of a pixel in the HSV color space is between 0 and 100, and the lightness of a pixel in the HSV color space is between 180 and 255, the pixel in the image in the HSV color space may be set to white.

[0090] On the contrary, when the hue of a pixel in the image of the HSV color space is not in the range of 0 to 180, or the saturation of a pixel in the image of the HSV color space is not in the range of 0 to 100, or the lightness of a pixel in the image of the HSV color space is not in the range of 180 to 255, the pixel in the image of the HSV color space can be set to black. Thus, the image in the HSV color space can be converted into a binary image BIMG (i.e., a black and white image).

[0091] The following is an actual example to illustrate the line segments generated on the field image IMG1. Figure 4 , Figure 4 FIG. 4 is a schematic diagram showing a method of generating a plurality of line segments LS1 to LS12 from a field image IMG1 in some embodiments of the present disclosure. Figure 4As shown, line segments LS1 - LS12 are generated from the field image IMG1 through the above-mentioned line detection operation, and the line segments LS1 - LS12 are marked on the field image IMG1 .

[0092] Furthermore, if Figure 2 As shown, in step S220 , a fusion operation is performed according to the plurality of line segments to generate a plurality of fused lines.

[0093] In some embodiments, the fusion operation may include: calculating the line segment angle and the maximum distance between one of the plurality of line segments and another of the plurality of line segments. Then, it may be determined whether to fuse one of the plurality of line segments and another of the plurality of line segments according to the line segment angle and the maximum distance.

[0094] In some embodiments, it may be determined whether the line segment angle is less than an angle threshold and the maximum distance is less than a distance threshold. When it is determined that the line segment angle is less than the angle threshold and the maximum distance is less than the distance threshold, one of the multiple line segments and another of the multiple line segments may be fused into one of the multiple fused lines.

[0095] Specifically, two of the plurality of line segments may be randomly selected, and it is determined whether the two selected line segments can be fused. If they can be fused, a fusion operation may be performed to calculate the linear equation of one of the fused lines. Figure 4 For example, two segments LS1 to LS12 (e.g., segments LS1 and LS3) can be selected at random to determine whether the two selected segments can be merged to calculate a straight line equation. That is, the segments LS1 to LS12 can be selected 66 times to determine whether they can be merged to calculate multiple straight line equations.

[0096] In some embodiments, one of the plurality of line segments and another of the plurality of line segments may be fused using any line segment fusion process. In some embodiments, the line segment fusion process may be a perceptually accurate line segment merging algorithm, a linear regression algorithm, or a line parameter averaging algorithm (e.g., calculating the average of the start coordinates (or end coordinates) of two line segments and the average of the slopes to generate a line equation according to the average of the start coordinates (or end coordinates) and the average of the slopes), etc.

[0097] The following is an actual example to illustrate the line segment angle and maximum distance. Figure 5 , Figure 5 Schematic diagram showing the line segment angle θ in some embodiments of the present disclosure. Figure 5As shown, assuming that line segments L1-L2 are selected, there is a line segment angle θ between the extended line EL of line segment L1 and line segment L2. Therefore, it can be determined whether the line segment angle θ is less than the angle threshold.

[0098] In addition, to understand how to calculate the maximum distance disclosed in this disclosure, please refer to Figure 6 . Figure 6 A schematic diagram showing the maximum distance d in some embodiments of the present disclosure is shown. It should be noted that any point on the line segment L2 corresponds to a vertical distance from the line segment L1, wherein the vertical distance with the largest distance value is defined as the maximum distance d.

[0099] In this example, it is assumed that the processor 130 selects line segments L1-L2, such as Figure 6 As shown, there is a maximum distance d between the line segment L1 and the line segment L2. Therefore, the processor 130 can determine whether the maximum distance d between the line segment L1 and the line segment L2 is less than a distance threshold.

[0100] The following is an actual example of multiple fusion lines. Figure 7 , Figure 7 FIG. 4 is a schematic diagram showing a plurality of fusion lines IL1 ˜ IL10 in a field image IMG1 in some embodiments of the present disclosure. Figure 7 As shown, the fusion lines IL1 ˜ IL10 may be marked on the field image IMG1 according to their linear equations.

[0101] Furthermore, if Figure 2 As shown, in step S230, a first group and a second group are distinguished by a plurality of fusion lines.

[0102] In some embodiments, a plurality of fused lines may be distinguished into a first group and a second group according to a slope threshold. In some embodiments, fused lines with a slope greater than the slope threshold may be used as fused lines of the first group, and fused lines with a slope less than the slope threshold may be used as fused lines of the second group.

[0103] In some embodiments, the minimum distance of the plurality of fused lines of the first group may be calculated based on the leftmost middle point of the field image IMG1, and the plurality of fused lines of the first group may be sorted according to the minimum distance of the plurality of fused lines of the first group. In some embodiments, the minimum distance of the plurality of fused lines of the second group may be calculated based on the uppermost middle point of the field image IMG1, and the plurality of fused lines of the second group may be sorted according to the minimum distance of the plurality of fused lines of the second group. Figure 7 For example, Figure 7As shown, the merged lines IL1-IL5 with a slope greater than the slope threshold and the merged lines IL6-IL10 with a slope less than the slope threshold can be identified from the merged lines IL1-IL10 according to the slope threshold. Thus, the merged lines IL1-IL5 can be grouped as the first group, and the merged lines IL6-IL10 can be grouped as the second group.

[0104] The following is an actual example to illustrate the sorting of the first group and the sorting of the second group. Figure 8 , Figure 8 A schematic diagram showing the sorting of the first group in some embodiments of the present disclosure. Figure 8 As shown, the leftmost middle point of the image IMG2 is used as the reference point LO, and the minimum distances LD1-LD5 between the first group of fused lines VL1-VL5 and the reference point LO are calculated. Thus, the fused lines VL1 to VL5 can be arranged in sequence according to the minimum distances LD1-LD5.

[0105] In addition, refer to Fig. 9 , Fig. 9 A schematic diagram showing the sorting of the second group in some embodiments of the present disclosure. Fig. 9 As shown, the uppermost middle point of the image IMG3 is used as the reference point UO, and the minimum distances UD1-UD5 between the second group of fused lines HL1-HL5 and the reference point UO are calculated. Thus, the fused lines HL1 to HL5 can be arranged in sequence according to the minimum distances UD1-UD5.

[0106] Furthermore, if Figure 2 As shown, in step S240, at least one first intersection point set is selected from the plurality of intersection points between the fusion lines in the first group and the second group, and at least one second intersection point set is selected from the field template. In some embodiments, the field template may be a field map overlooking the field, or an image generated by photographing the field at a different perspective from the field image IMG1.

[0107] In some embodiments, the step of selecting the first intersection point set and the second intersection point set includes: randomly selecting two of the first group and two of the second group, and using four intersection points between the two of the first group and the two of the second group as one of at least one first intersection point set. Then, two of the plurality of horizontal site lines and two of the plurality of vertical site lines may be randomly selected from the site template, and using four intersection points between the two of the plurality of horizontal site lines and the two of the plurality of vertical site lines as one of at least one second intersection point set.

[0108] In some embodiments, when one of the two in the first group is selected first, one of the vertical field lines can be selected from the field template according to the ranking of the one. When one of the two in the second group is selected first, one of the horizontal field lines can be selected from the field template according to the ranking of the one. Thus, the number of subsequent selections of the first intersection point and the second intersection point set can be greatly reduced to reduce the amount of calculation.

[0109] For example, assuming that one of the fusion lines in the first group is selected first, and the fusion line is ranked earlier (i.e., closer to the leftmost middle point of the field image IMG1), then the vertical field line close to the leftmost side of the field template will be selected, and the vertical field line close to the rightmost side of the field template will not be selected. Conversely, assuming that another fusion line in the first group is selected first, and the fusion line is ranked later (i.e., farther from the leftmost middle point of the field image IMG1), then the vertical field line farther from the leftmost side of the field template will be selected, and the vertical field line close to the leftmost side of the field template will not be selected. In this way, the number of selections will be greatly reduced.

[0110] The following is an actual example of the horizontal and vertical site lines in the site template. Fig.10 , Fig.10 Schematic diagram of a field template TEM1 in some embodiments of the present disclosure is shown. Fig.10 As shown, the field template TEM1 is a field map of a tennis court, and a plurality of field lines are shown on the field map. Then, the linear equations of a plurality of horizontal field lines TL1-TL4 and a plurality of vertical field lines TL5-TL9 can be calculated from the plurality of field lines in the field template TEM1, and the origin TO of the field template TEM1 can be set at the upper left corner of the field template TEM1, the horizontal field lines TL1-TL4 are all the horizontal lines in the plurality of field lines, and the vertical field lines TL5-TL9 are all the vertical lines in the plurality of field lines.

[0111] The following is an actual example to illustrate the selection of two fusion lines in the first group and two fusion lines in the second group. Fig.11 , Fig.11 FIG. 2 is a schematic diagram showing the selection of multiple fusion lines IL1, IL5, IL6, and IL9 in some embodiments of the present disclosure. Fig.11 As stated, continued Figure 7 Examples can be found in Figure 7The merged lines IL1 and IL5 in the first group are randomly selected, and the merged lines IL6 and IL9 are randomly selected from the merged lines IL6 to IL10 in the second group. Then, the four intersection points IP1 to IP4 between the merged lines IL1 and IL5 and the merged lines IL6 and IL9 are used as one of the first intersection points.

[0112] The following is a practical example to illustrate the selection of two horizontal site lines and two vertical site lines. Fig.12 , Fig.12 FIG. 2 is a schematic diagram showing the selection of two horizontal site lines TL1, TL4 and two vertical site lines TL5, TL6 in some embodiments of the present disclosure. Fig.12 As stated, continued Fig.10 Examples can be found in Fig.10 The horizontal site lines TL1 and IL4 are randomly selected from the horizontal site lines TL1 to TL4 in the field template TEM1, and the vertical site lines TL5 and TL6 are randomly selected from the vertical site lines TL5 to TL9. Then, the four intersection points TP1 to TP4 between the horizontal site lines TL1 and IL4 and the vertical site lines TL5 and TL6 can be used as one of the second intersection points.

[0113] In other words, by the above method of randomly selecting two of the first group and two of the second group and randomly selecting two of the plurality of horizontal field lines and two of the plurality of vertical field lines from the field template, Figure 7 as well as Fig.10 For example, 6000 selection results may be generated.

[0114] Furthermore, if Figure 2 As shown, in step S250 , at least one homography matrix between at least one first intersection point set and at least one second intersection point set is calculated.

[0115] In some embodiments, at least one homography matrix is ​​a homography matrix used to transform the coordinates of the field template to the coordinates of the field image IMG1. For example, the field image IMG1 may be an image of a tennis court taken from a television broadcast perspective, the field template may be a field map overlooking the tennis court, and the homography matrix may be a transformation matrix between this image and the field map. In addition, the homography matrix can be calculated by taking out at least four points from the image (i.e., one of the first intersection points mentioned above) and at least four corresponding points from the field map (i.e., one of the corresponding second intersection points mentioned above). For example, Fig.11 as well as Fig.12For example, the homography matrix between the field image IMG1 and the field template TEM1 may be calculated according to the intersection points IP1-IP4 in the field image IMG1 and the intersection points TP1-TP4 in the field template TEM1.

[0116] It is worth noting that the homography matrix is ​​a transformation matrix between two planar images generated by photographing the same scene from two different perspectives in the art. Therefore, the calculation of the homography matrix will not be described in detail.

[0117] In some embodiments, after calculating the homography matrix, an accuracy operation may be performed on the homography matrix to generate a corresponding conversion score. Then, the template to be superimposed may be superimposed on the field image IMG1 using the homography matrix with the highest conversion score to generate a superimposed image. In some embodiments, the template to be superimposed may also be superimposed on the field image IMG1 using multiple homography matrices with the highest conversion scores (e.g., homography matrices with the highest 10 conversion scores) to generate multiple superimposed images.

[0118] In some embodiments, the accuracy operation may include: for each of the homography matrices, performing the following steps: a perspective transformation may be performed on the first group and the second group. Then, a first angle difference and a first distance difference between each of the transformed first group and the corresponding vertical field line in the field template may be calculated, and a first score for each of the transformed first group may be calculated based on the plurality of first angle differences and the plurality of first distance differences. Then, a second angle difference and a second distance difference between each of the transformed second group and the corresponding horizontal field line in the field template may be calculated, and a second score for each of the transformed second group may be calculated based on the plurality of second angle differences and the plurality of second distance differences. Then, one of a plurality of transformation scores may be calculated based on the plurality of first scores and the plurality of second scores.

[0119] In some embodiments, a quotient between one of the first angle differences and a preset angle may be calculated, and a hyperbolic tangent function of the quotient of one of the first angle differences may be calculated. Next, a quotient between one of the first distance differences and a preset distance may be calculated, and a hyperbolic tangent function of the quotient of one of the first distance differences may be calculated. Next, a first score of one of the first groups of conversions may be calculated based on the angle weight, the distance weight, the hyperbolic tangent function of the quotient of one of the first angle differences, and the hyperbolic tangent functions of the multiple quotients of one of the first distance differences.

[0120] In some embodiments, a quotient between one of the second angle differences and a preset angle may be calculated, and a hyperbolic tangent function of the quotient of one of the second angle differences may be calculated. Then, multiple quotients between one of the second distance differences and a preset distance may be calculated, and a hyperbolic tangent function of the quotient of one of the second distance differences may be calculated. Then, a second score of one of the converted second groups is calculated based on the angle weight, the distance weight, the hyperbolic tangent function of the multiple quotients of one of the second angle differences, and the hyperbolic tangent function of the multiple quotients of one of the second distance differences.

[0121] The following is an actual example to illustrate the calculation of conversion score. Fig.13 , Fig.13 FIG. 4 is a schematic diagram showing a plurality of conversion lines TSL1 to TSL10 on a field template TEM1 in some embodiments of the present disclosure. Figure 7 , Fig.10 as well as Fig.13 As shown, continued Fig.11 and Fig.12 For example, assuming that Fig.11 and Fig.12 The selection of can calculate the homography matrix between the field image IMG1 and the field template TEM1, and this homography matrix can be used to convert the fusion lines IL1~IL10 in the field image IMG1 into the straight line equations of the conversion lines TSL1~TSL10 on the field template TEM1 (that is, perform the above-mentioned perspective conversion), wherein the fusion lines IL6~IL10 are the fusion lines of the first group, the fusion lines IL1~IL5 are the fusion lines of the second group, the conversion lines TSL6~TSL10 are the conversion lines of the converted first group, and the conversion lines TSL1~TSL5 are the conversion lines of the converted second group.

[0122] Furthermore, since the conversion line TSL1 is converted from the fusion line IL1 in the second group, the corresponding horizontal field line can be found from the horizontal field lines TL1 - TL4 in the field template TEM1 according to the conversion line TSL1 .

[0123] Specifically, the minimum distances between the horizontal site lines TL1 to TL4 and the origin TO may be calculated as the plurality of site line minimum distances, and the minimum distance between the conversion line TSL1 and the origin TO may be calculated as the conversion line minimum distance of the conversion line TSL1. Next, a plurality of distance differences between the plurality of site line minimum distances and the conversion line minimum distance of the conversion line TSL1 may be calculated, and the horizontal site line corresponding to the minimum distance difference may be selected as the corresponding horizontal site line.

[0124] In this embodiment, for the conversion line TSL1, the corresponding horizontal site line can be found to be the horizontal site line TL1 through the above calculation. Similarly, through the above calculation, it can also be identified that the conversion lines TSL2 and TSL4 to TSL10 correspond to the horizontal site lines TL2 to TL4 and the vertical site lines TL5 to TL9, respectively, and it can be identified that the conversion line TSL3 corresponds to the horizontal site line TL2.

[0125] Furthermore, the angle difference and the minimum distance of the inclination angle between the conversion line TSL1 and the horizontal site line TL1 can be calculated, and the calculated angle difference and the minimum distance of the inclination angle are used as the second angle difference and the second distance difference of the conversion line TSL1. Similarly, through the above calculation, the second angle difference and the second distance difference of the conversion lines TSL2-TSL5 can also be calculated, and the first angle difference and the first distance difference of the conversion lines TSL6-TSL10 can be calculated.

[0126] Furthermore, the quotient between the first angle difference of conversion line TSL6 and a preset angle can be calculated, and the hyperbolic tangent function of the quotient of the first angle difference of conversion line TSL6 can be calculated. Then, the quotient between the first distance difference of conversion line TSL6 and a preset distance can be calculated, and the hyperbolic tangent function of the quotient of the first distance difference of conversion line TSL6 can be calculated. Then, based on the angle weight, the distance weight, the hyperbolic tangent function of the quotient of the first angle difference of conversion line TSL6, and the hyperbolic tangent functions of multiple quotients of the first distance differences of conversion line TSL6, the first score of conversion line TSL6 is calculated. Similarly, the first scores of conversion lines TSL7 to TSL10 can also be calculated by the same calculation.

[0127] Furthermore, the quotient between the second angle difference of conversion line TSL1 and a preset angle can be calculated, and the hyperbolic tangent function of the quotient of the second angle difference of conversion line TSL1 can be calculated. Then, the quotient between the second distance difference of conversion line TSL1 and a preset distance can be calculated, and the hyperbolic tangent function of the quotient of the second distance difference of conversion line TSL1 can be calculated. Then, the second score of conversion line TSL1 is calculated based on the angle weight, the distance weight, the hyperbolic tangent function of the quotient of the second angle difference of conversion line TSL1, and the hyperbolic tangent functions of multiple quotients of the second distance differences of conversion line TSL1. Similarly, the second scores of conversion lines TSL2 to TSL5 can also be calculated through the same calculation.

[0128] In some embodiments, the hyperbolic tangent function of the quotient of the above-mentioned angle difference (ie, the first angle difference or the second angle difference) is as shown in the following formula (1).

[0129] angleDiff=tanh(|a1-a2| / ā)…Formula (1)

[0130] Wherein angleDiff is the hyperbolic tangent function of the quotient of the angle difference, a1 is the inclination of one of the fusion lines, a2 is the inclination of the field line (i.e., vertical field line or horizontal field line) corresponding to the fusion line in the field template TEM1, and ā is a preset angle (e.g., set by the user).

[0131] In some embodiments, the hyperbolic tangent function of the quotient of the above distance difference (ie, the first distance difference or the second distance difference) is as shown in the following formula (2).

[0132]

[0133] Wherein originDiff is the hyperbolic tangent function of the quotient between the distance difference and the preset distance, d1 is the minimum distance between one of the fusion lines and the origin of the field template TEM1 (i.e., the origin TO), d2 is the minimum distance between the field line corresponding to the fusion line in the field template (i.e., the vertical field line or the horizontal field line) and the origin of the field template, is a preset distance (eg, set by the user).

[0134] In some embodiments, the first score is as shown in the following formula (3).

[0135] SC1=1-(W1×angleDiff1+W2×originDiff1)…Formula (3)

[0136] SC1 is the first score, W1 is the angle weight, angleDiff1 is the first angle difference of one of the transformed lines in the first group of transformations, W2 is the distance weight, and originDiff1 is the first distance difference of the transformed line.

[0137] In some embodiments, the second score is as shown in the following formula (4).

[0138] SC2=1-(W1×angleDiff2+W2×originDiff2)…Formula (4)

[0139] SC2 is a second fraction, angleDiff1 is a second angle difference of a conversion line in the second group of conversions, and originDiff1 is a second distance difference of the conversion line.

[0140] Furthermore, one of a plurality of conversion scores is calculated according to the plurality of first scores and the plurality of second scores calculated above. In some embodiments, the conversion score is as shown in the following formula (5).

[0141] warpingScore=SUM_SC1+SUM_SC2)...Formula (5)

[0142] Where warpingScore is the transformation score corresponding to this homography matrix, SUM_SC1 is the sum of the first scores of all transformation lines in the first group of transformations, and SUM_SC2 is the sum of the second scores of all transformation lines in the second group of transformations.

[0143] It is worth noting that Fig.11 as well as Fig.12 As shown in FIG. 1 , since the fusion lines IL1, IL5, IL6, and IL9 selected from the field image IMG1 do not completely correspond to the field lines TL1, TL4, TL5, and TL6 selected from the field template TEM1 (i.e., the fusion line IL9 is on the right line of the singles on the tennis ball, and the field line TL6 is on the left line of the singles on the tennis ball, and they do not correspond), the conversion effect of the calculated homography matrix is ​​not good (e.g., Fig.13 As shown, only the conversion lines TSL1, TSL5, and TSL6 correspond to the correct field lines).

[0144] The following is an example of a homography matrix with a higher conversion score. Fig.14 , Fig.14 FIG. 2 is a schematic diagram showing the selection of multiple fusion lines IL1, IL5, IL6, and IL10 in some embodiments of the present disclosure. Fig.14 As stated, continued Figure 7 Examples can be found in Figure 7 The merged lines IL1 and IL5 in the first group are randomly selected, and the merged lines IL6 and IL10 in the second group are randomly selected. Then, the four intersection points IP1 to IP4 between the merged lines IL1 and IL5 and the merged lines IL6 and IL10 can be used as one of the first intersection points.

[0145] Refer to Fig.15 , Fig.15 FIG. 2 is a schematic diagram showing the selection of two horizontal site lines IL1, IL5 and two vertical site lines IL6, IL9 in some embodiments of the present disclosure. Fig.15 As stated, continued Fig.10 Examples can be found in Fig.10 The horizontal site lines TL1 and IL5 are randomly selected from the horizontal site lines TL1 to TL4 in the field template TEM1, and the vertical site lines TL6 and TL9 are randomly selected from the vertical site lines TL5 to TL9. Then, the four intersection points TP1 to TP4 between the horizontal site lines TL1 and IL5 and the vertical site lines TL6 and TL9 can be used as one of the second intersection points.

[0146] Refer to Fig.16 , Fig.16FIG. 4 is a schematic diagram showing a plurality of conversion lines TSL1 to TSL10 on a field template TEM1 in some embodiments of the present disclosure. Figure 7 , Fig.10 as well as Fig.13 As shown, continued Fig.14 and Fig.15 For example, assuming that Fig.14 and Fig.15 The selection of can calculate the homography matrix between the field image IMG1 and the field template TEM1, and this homography matrix can be used to convert the fusion lines IL1~IL10 in the field image IMG1 into the straight line equations of the conversion lines TSL1~TSL10 on the field template TEM1 (that is, perform the above-mentioned perspective conversion), wherein the fusion lines IL6~IL10 are the fusion lines of the first group, the fusion lines IL1~IL5 are the fusion lines of the second group, the conversion lines TSL6~TSL10 are the conversion lines of the converted first group, and the conversion lines TSL1~TSL5 are the conversion lines of the converted second group.

[0147] Furthermore, by Fig.16 It can be seen that the conversion effect of this homography conversion matrix is ​​better (because all conversion lines TSL1-TSL2, TSL4-TSL10 are corresponding to the correct site line, only the conversion line TSL3 cannot be corresponding to the site line). Figures 11 to 13 The calculation method of the example calculates a homography matrix and the conversion score of this homography matrix. After actual calculation, it can be found that the conversion score of this homography matrix is ​​much larger than Figures 11 to 13 Based on the above steps, the present disclosure can select the first intersection set from the first group and the second group multiple times (which can be a preset number of times or all possible selected numbers), and select the second intersection set from the field template multiple times, and then calculate multiple homography matrices. Thus, the homography matrix with a higher conversion score can be found as the best homography matrix.

[0148] In some embodiments, the accuracy calculation may also include: for each of the at least one homography matrix, performing the following steps: a perspective transformation may be performed on a plurality of vertical lines of site, and a number of a plurality of first overlapping pixels and a number of a plurality of first non-overlapping pixels on the plurality of transformed vertical lines of site may be calculated. Then, a perspective transformation may be performed on a plurality of horizontal lines of site, and a number of a plurality of second overlapping pixels and a number of a plurality of second non-overlapping pixels on the plurality of transformed horizontal lines of site may be calculated. Then, one of a plurality of transformation scores may be calculated based on the number of the plurality of first overlapping pixels, the number of the plurality of first non-overlapping pixels, the number of the plurality of second overlapping pixels, and the number of the plurality of second non-overlapping pixels.

[0149] In some embodiments, multiple homography matrices with higher conversion scores (for example, the top ten homography matrices with higher conversion scores) can be selected from at least one homography matrix calculated by the above-mentioned method using angle difference and distance difference to execute the above-mentioned method of calculating the number of overlaps and misalignments to calculate a new conversion score.

[0150] The following is an actual example of overlapping pixels and overlapping pixels. Fig.17 , Fig.17 A schematic diagram showing a plurality of pixel points P of field lines (including horizontal field lines and vertical field lines) on a field template TEM1 in some embodiments of the present disclosure is shown. Fig.17 As shown, the coordinates of multiple pixel points P can be acquired from the field lines in the field template TEM1. In some embodiments, acquisition (or sampling) can be performed every two pixels on the field lines in the field template TEM1.

[0151] Next, refer to Fig.18 , Fig.18 FIG. 4 is a schematic diagram showing a plurality of transition points TSP on a field image IMG1 in some embodiments of the present disclosure. Fig.18 As shown, continued Fig.17 For example, the coordinates of multiple pixel points P of the field lines on the field template TEM1 can be converted into the coordinates of multiple conversion points TSP on the field image IMG1 using the homography matrix calculated above, and the coordinates of these conversion points TSP are marked on the field image IMG1. Then, it can be determined whether these conversion points TSP are located on the white pixels of the black and white image generated above. When the conversion point TSP is located on the white pixel of the black and white image generated above, it can be identified as a superimposed pixel, wherein the superimposed pixel converted by the vertical field line is the first superimposed pixel, and the superimposed pixel converted by the horizontal field line is the second superimposed pixel.

[0152] On the contrary, non-overlapping pixels (i.e., pixels on the plurality of site line segments OG1-OG3) may be identified, wherein the non-overlapping pixels converted by the vertical site lines are first non-overlapping pixels, and the non-overlapping pixels converted by the horizontal site lines are second non-overlapping pixels. Finally, one of a plurality of conversion scores may be calculated according to the number of the first overlapping pixels, the number of the first non-overlapping pixels, the number of the second overlapping pixels, and the number of the second non-overlapping pixels.

[0153] In some embodiments, the above conversion score is shown in the following formula (6).

[0154] projectScore=sum(numSpoint)-W3×sum(numUSpoint)…Formula (6)

[0155] Wherein projectScore is the conversion score, sum(,) is the sum calculation function, numSpoint is the sum of the number of all first overlapping pixels and the number of all second overlapping pixels, W3 is the non-overlapping weight, and numUSpoint is the sum of the number of all first non-overlapping pixels and the number of all second non-overlapping pixels.

[0156] It is worth noting that the above method for calculating the number of overlaps and misalignments can be performed after the method for calculating the angle difference and the distance difference, or can be performed directly after the above step S250 is performed, and there is no particular limitation. In addition, such a method for calculating the number of overlaps and misalignments can also correctly track the boundary line on the field of the field image (for example, used to determine whether the ball on the field is out of bounds).

[0157] In some embodiments, after generating at least one homography matrix, the template to be superimposed may be converted into at least one conversion map using the at least one homography matrix, and the at least one conversion map may be superimposed on the field image IMG1. In some embodiments, when a specific image (e.g., an advertisement or graphic provided by a manufacturer) needs to be superimposed on the field image IMG1, the specific image may be overlaid on the field template TEM1 to generate the template to be superimposed.

[0158] The following is an actual example of how to overlay the conversion graph. Fig.19 , Fig.19 Schematic diagram of the template TEM2 to be superimposed in some embodiments of the present disclosure is shown. Fig.19 As shown, the template TEM2 to be superimposed has an advertisement mark AD. Therefore, the template TEM2 to be superimposed can be transformed using the above homography matrix to overlay the generated transformation image on the field image IMG1. Fig. 20 , Fig. 20 Schematic diagram of a superimposed image SIMG in some embodiments of the present disclosure is shown. Fig. 20 As shown, when the conversion image is overlaid on the field image to generate the superimposed image SIMG, a deformed advertising mark AD can be generated on the superimposed image SIMG, and the deformed advertising mark AD can completely follow the deformation of the captured field image. As a result, when the user views the field image, the shape of the advertising mark AD will not be felt to be strange. In other words, the image processing device and method disclosed in the present invention can place the advertising mark AD in an appropriate position, reduce the abruptness of the advertising mark AD, and avoid deformation of the advertising mark AD.

[0159] It is worth noting that, at the application level, the image processing device and method of the disclosed document can be used in various fields with machine vision, image classification or data classification in addition to the above-mentioned application of advertising. For example, this image processing device and method can also be used to place virtual objects on the field image. On the other hand, this image processing device and method can also be used to determine whether the ball on the sports field is out of bounds. In addition, this image processing device and method can also be used in the shooting of panoramic images or the application of virtual reality (augmented reality).

[0160] It is worth noting that when shooting a scene, the shooting position may change (i.e., move from a first shooting position to a second shooting position). Since the above-calculated homography matrix can only correspond to the first shooting position, this will cause the above-calculated homography matrix to lose accuracy. Therefore, it is necessary to further recalculate the homography matrix corresponding to the second shooting position using the above-calculation method.

[0161] In view of this, a method for saving the amount of calculation for recalculating the homography matrix corresponding to the second shooting position based on the new image is proposed below. The present disclosure further proposes an image processing device and method for perspective conversion, which generates optical flow maps from different frames of different shooting positions, and uses the optical flow map to calculate the pixel movement trend between the first shooting position and the second shooting position, and then generates the key points in the image of the second shooting position from the key points in the field template according to the pixel movement trend and the homography matrix corresponding to the first shooting position. In this way, a new homography matrix is ​​further calculated from the key points in the field template and the key points in the image of the second shooting position. In this way, the amount of calculation for regenerating the homography matrix can be further reduced, and a new homography matrix can be generated in real time. The above-mentioned technology of the present disclosure is specifically described by taking the following embodiments as an example.

[0162] Refer to Fig.21 , Fig.21 A flowchart showing an image processing method for perspective conversion in some embodiments of the present disclosure is shown. The image processing method for perspective conversion is applicable to an electronic device, which may be Figure 1 The image processing device 100, Figure 1 The components in the image processing device 100 are used to perform steps S2110 to S2150 in the image processing method for viewing angle conversion. Fig.21 As shown, first, in step S2110, a previous image and a current image are captured.

[0163] In some embodiments, a previous image and a current image may be captured by the image acquisition circuit 110. In some embodiments, the previous image may be a frame before the current image among a plurality of images continuously captured by the image acquisition circuit 110. In some embodiments, the shooting position of the image acquisition circuit 110 may be continuously changed. In some embodiments, a field may be captured at a first shooting position to generate a previous image, and a field may be captured at a second shooting position to generate a current image. In some embodiments, the first shooting position and the second shooting position correspond to different shooting angles.

[0164] In step S2120 , feature extraction is performed on the previous image and the current image to calculate an optical flow information matrix from a first feature set of the previous image and a second feature set of the current image.

[0165] In some embodiments, the operation of extracting features from the previous image and the current image includes: detecting a plurality of first candidate feature points and a plurality of second candidate feature points from the previous image and the current image, respectively. Then, a feature matching operation may be performed on the plurality of first candidate feature points and the plurality of second candidate feature points to generate a first feature set of the previous image and a second feature set of the current image.

[0166] In some embodiments, a plurality of first candidate feature points and a plurality of second candidate feature points may be detected from a previous image and a current image respectively by using a corner detection operation or an edge detection operation, etc. In some embodiments, a SIFT (scale-invariant feature transform) algorithm, a SURF (speeded up robust features) algorithm, or an ORB (oriented fast and rotated brief) algorithm, etc. may be used to perform feature mapping between the plurality of first candidate feature points and the plurality of second candidate feature points and delete the unmatched first candidate feature points and the unmatched second candidate feature points.

[0167] In some embodiments, after feature matching, feature filtering may be performed on the plurality of first candidate feature points and the plurality of second candidate feature points to generate a first feature set of a previous image and a second feature set of a current image. In some embodiments, the feature filtering operation may include: calculating a plurality of feature vectors between the plurality of matched first candidate feature points and the plurality of matched second candidate feature points (for example, taking one of the matched first candidate feature points as a starting point and the corresponding matched second candidate feature point as an end point, thereby generating one of the feature vectors), and filtering the plurality of feature vectors using an outlier algorithm. In some embodiments, the step of filtering the plurality of feature vectors using an outlier algorithm may include: finding at least one outlier vector from the plurality of feature vectors using an outlier algorithm, thereby deleting at least one outlier vector from the plurality of feature vectors. In some embodiments, the outlier algorithm may be a local outlier factor (LOF) algorithm or a random cut forest (RCF) algorithm, etc.

[0168] In some embodiments, the operation of calculating the optical flow information matrix from the first feature set of the previous image and the second feature set of the current image includes: an optical flow map can be generated based on the first feature set of the previous image and the second feature set of the current image. Then, the optical flow information matrix can be calculated based on the optical flow map. In some embodiments, the Lucas-Kanade algorithm or the dense optical flow (Farneback) algorithm can be used to generate an optical flow map based on the first feature set of the previous image and the second feature set of the current image. In some embodiments, the optical flow information matrix indicates the trend of pixel movement between the previous image and the current image. In some embodiments, an affine transformation operation can be used to generate a transformation matrix (e.g., an optical flow information matrix) from the optical flow map, and this transformation matrix indicates the trend of position change between the pixels of the previous image and the pixels of the current image.

[0169] The following is an actual example to illustrate the generation of optical flow map. Fig. 22 , Fig. 22 Schematic diagram showing the generation of an optical flow map OF_MAP in some embodiments of the present disclosure. Fig. 22As shown, a plurality of first candidate feature points and a plurality of second candidate feature points can be detected from the previous image P_IMG and the current image C_IMG respectively by using the corner detection operation. Then, a first feature set of the previous image P_IMG and a second feature set of the current image C_IMG can be generated from the plurality of first candidate feature points and the plurality of second candidate feature points by using the SIFT algorithm and the regional outlier factor algorithm. Then, based on the first feature set of the previous image P_IMG and the second feature set of the current image C_IMG, an optical flow map OF_MAP can be generated by using the Lucas-Kanad algorithm.

[0170] Furthermore, if Fig.21 As shown, in step S2130, the optical flow information matrix is ​​used to infer the position of the second key point set in the previous image in the current image to store it as the third key point set. In some embodiments, the previous homography matrix can be used to perform perspective conversion processing on the first key point set in the field template to generate the second key point set in the previous image. In some embodiments, the previous homography matrix can be a matrix composed of Figure 2 The homography matrix generated from the field template and the previous image in the step may also be a homography matrix pre-stored in the memory 120, or may be a homography matrix obtained by a traditional calculation method. In some embodiments, the second key point set in the previous image may be converted into a third key point set in the current image using an optical flow information matrix.

[0171] In some embodiments, the field template may have a target image and a first set of key points. In some embodiments, the target image may be an advertising marker (e.g., Fig.19 In some embodiments, the key points in the first key point set may be marking points of multiple intersections between multiple site lines pre-set on the site template. In some embodiments, the number of these marking points may be any positive integer not less than 4, without particular limitation.

[0172] The following is an actual example to illustrate the first key point set. Fig.23 , Fig.23 Schematic diagram of the field template TEM in some other embodiments of the present disclosure is shown. Fig.23 As shown, the field template TEM has a plurality of field lines, and a plurality of key points KP are marked on a plurality of intersections between the field lines.

[0173] The following is an actual example to illustrate the second key point set and the third key point set. Fig.24 , Fig.24 Schematic diagram showing the second key point set and the third key point set in some embodiments of the present disclosure. Fig.24 As shown, the previous homography matrix can be used to Fig.23The multiple key points KP in the field template TEM are converted into the multiple key points KP in the previous image P_IMG as the second key point set, and the optical flow information matrix is ​​used to convert the multiple key points KP in the previous image P_IMG into the multiple key points KP in the current image C_IMG as the third key point set.

[0174] Furthermore, if Fig.21 As shown, in step S2140 , a current homography matrix between the first key point set and the third key point set is calculated.

[0175] In some embodiments, the third key point set in the current image may be updated based on the second key point set and the fundamental matrix in the previous image, wherein the fundamental matrix indicates the corresponding perspective conversion relationship between the previous image and the current image. In some embodiments, the fundamental matrix may be calculated based on the starting point and the end point indicated by the multiple optical flows in the optical flow map. In some embodiments, the fundamental matrix may be calculated based on the first feature set of the previous image and the second feature set of the current image. In other words, the fundamental matrix can be calculated by using the feature points of the previous image and the corresponding feature points of the current image (i.e., the starting point and the end point of the optical flow), and the calculation of the fundamental matrix is ​​a commonly used calculation by those skilled in the art, and therefore, it will not be described in detail here.

[0176] In some embodiments, the step of updating the third key point may include: using a basic matrix to convert the second key point set in the previous image into a plurality of straight lines in the current image. Then, based on the plurality of straight lines, the third key point set in the current image may be updated.

[0177] In some embodiments, it may be determined whether each of the third keypoint set in the current image is located on one of a plurality of corresponding straight lines in the current image. Then, at least one noise keypoint in the third keypoint set may be deleted to update the third keypoint set in the current image, wherein each of the at least one noise keypoint is not located on a corresponding straight line in the current image. In some embodiments, a keypoint corresponding to the at least one noise keypoint in the first keypoint set may be deleted.

[0178] The following is an example to illustrate the principle of the basic matrix. Fig.25 , Fig.25 Schematic diagram showing the perspective conversion relationship in some embodiments of the present disclosure. Fig.25As shown, the field can be photographed from the optical center O of the first photographing position and the optical center O' of the second photographing position to generate the image IMG_O and the image IMG_O'. The pixel point x is generated by photographing the field point X in the field. However, since the depth of the image IMG_O is unknown, all the field points X on the ray r extending from the optical center O and the pixel point x can form the pixel point x on the image IMG_O.

[0179] Next, since it is uncertain which field point X forms the pixel point x on the image IMG_O, the ray r can form an epipolar line l' on the image IMG_O', and all the pixel points x' on the epipolar line l' may correspond to the pixel point x, where the pixel point x' and the pixel point x are generated by shooting the same field point X. In addition, the line between the optical center O and the optical center O' can generate the epipolar point e and the epipolar point e' on the image IMG_O and the image IMG_O', respectively. The epipolar point e' is located on the epipolar line l', and the epipolar point e and the pixel point x can form another epipolar line l. All the pixel points on the epipolar line l will correspond to the epipolar line l'. In other words, all the pixel points on the epipolar line l can be converted to the pixel points on the epipolar line l' using the basic matrix, and the basic matrix can be calculated based on the corresponding perspective conversion relationship between these pixel points. Therefore, any pixel point on the image IMG_O can be converted to the epipolar line on the image IMG_O' using the basic matrix. In this way, the key points of the second key point set can be converted to a straight line on the current image using the same method, and it is determined whether the corresponding key point of this key point in the third key point set is on this straight line. If it is determined that it is not on this straight line, the corresponding key point is deleted.

[0180] The following is an actual example to illustrate the updated third key point set. Fig.26 , Fig.26 Schematic diagram showing the updated third key point set in some embodiments of the present disclosure. Fig.26 As shown, the basic matrix and Fig.24 Multiple key points KP of the previous image P_IMG in Fig.24 The multiple key points KP of the current image C_IMG in are filtered to generate multiple filtered key points FKP of the current image C_IMG as the updated third key point set. It is worth noting that due to Fig.24 Some key points KP of the current image C_IMG in the image C_IMG (i.e., the key points on the center line of the court and the key points overlapping with the characters) have been filtered out. At this time, the number of multiple filtered key points FKP of the current image C_IMG is less than Fig.24 The number of multiple key points KP in the current image C_IMG.

[0181] Furthermore, if Fig.21As shown, in step S2150, the target image is projected to the corresponding position on the current image using the current homography matrix. Fig.19 as well as Fig. 20 The method will use the current homography matrix to transform the target image (for example, Fig.19 The pixels of the advertising marker AD) are transformed in perspective to be projected onto the current image.

[0182] Through the above steps, users will not feel that the shape of the advertising mark is strange when viewing the field image. In addition, this approach only uses the four arithmetic operations and differentials in mathematical calculations, and the number of calculations of feature points and key points is not large. This approach will greatly reduce the amount of calculations and improve the immediacy of tracking. The use of optical flow calculations will also prevent the target image from being stuck when projected onto the current image.

[0183] In summary, the image processing device for perspective conversion proposed in the present disclosure can use the optical flow method to update the homography matrix to greatly reduce the amount of calculation for regenerating the homography matrix. In addition, the homography matrix can be updated instantly between different frames. Regardless of how the shooting position changes, a new homography matrix can be calculated immediately. On the other hand, the general method of calculating the homography matrix may cause a sense of stuttering when the target image is projected onto the current image, while the image processing device for perspective conversion disclosed in the present disclosure can greatly increase the sense of smoothness after projecting onto continuous images.

[0184] Although specific embodiments of the present disclosure have been disclosed with respect to the above embodiments, these embodiments are not intended to limit the present disclosure. Various substitutions and modifications may be performed in the present disclosure by a person of ordinary skill in the relevant art without departing from the principles and spirit of the present disclosure. Therefore, the scope of protection of the present disclosure is determined by the claims.

Claims

1. An image processing device for perspective conversion, comprising: An image acquisition circuit for capturing a previous image and a current image; A memory, used to store a plurality of instructions, a target image in a field template, and a first key point set in the field template; as well as A processor is used to process the plurality of instructions to perform the following operations: Performing a feature extraction operation on the previous image and the current image to calculate an optical flow information matrix from a first feature set of the previous image and a second feature set of the current image; Using the optical flow information matrix, infer the position of the second key point set in the previous image in the current image to store it as a third key point set; Calculating a current homography matrix between the first key point set and the third key point set; as well as The target image is projected to a corresponding position on the current image using the current homography matrix.

2. The image processing device for perspective conversion according to claim 1, wherein the second key point set is obtained by converting the first key point set on the field template through a previous homography matrix.

3. The image processing device for perspective conversion according to claim 1, wherein the previous image is a frame before the current image, and the operation of performing the feature extraction operation on the previous image and the current image comprises: Detecting a plurality of first candidate feature points and a plurality of second candidate feature points from the previous image and the current image respectively; as well as A feature matching operation is performed on the plurality of first candidate feature points and the plurality of second candidate feature points to generate the first feature set of the previous image and the second feature set of the current image.

4. The image processing device for perspective conversion according to claim 3, wherein the processor is further configured to perform the following operations: The third key point set in the current image is updated based on the second key point set and a basic matrix in the previous image, wherein the basic matrix indicates a corresponding perspective conversion relationship between the previous image and the current image.

5. The image processing device for perspective conversion according to claim 4, wherein the basic matrix is ​​generated by the following operation: The fundamental matrix is ​​calculated based on the first feature set of the previous image and the second feature set of the current image.

6. The image processing device for perspective conversion according to claim 4, wherein the operation of updating the third key point set comprises: converting the second key point set in the previous image into a plurality of straight lines in the current image using the basic matrix; as well as Based on the plurality of straight lines, the third key point set in the current image is updated.

7. The image processing device for perspective conversion according to claim 6, wherein the operation of updating the third key point set in the current image based on the plurality of straight lines comprises: determining whether each of the third key point set in the current image is located on one of the plurality of corresponding straight lines in the current image; as well as At least one noise keypoint in the third keypoint set is deleted to update the third keypoint set in the current image, wherein each of the at least one noise keypoint is not located on the corresponding straight line in the current image.

8. An image processing method for perspective conversion, comprising: Taking a previous image and a current image, and storing a target image in a field template and a first key point set in the field template; Performing a feature extraction operation on the previous image and the current image to calculate an optical flow information matrix from a first feature set of the previous image and a second feature set of the current image; Using the optical flow information matrix, infer the position of the second key point set in the previous image in the current image to store it as a third key point set; Calculating a current homography matrix between the first key point set and the third key point set; as well as The target image is projected to a corresponding position on the current image using the current homography matrix.

9. The image processing method for perspective conversion according to claim 8, wherein the second key point set is obtained by converting the first key point set on the field template through a previous homography matrix.

10. The image processing method for perspective conversion according to claim 8, wherein the previous image is a frame before the current image, and the step of performing the feature extraction on the previous image and the current image comprises: Detecting a plurality of first candidate feature points and a plurality of second candidate feature points from the previous image and the current image respectively; as well as A feature matching operation is performed on the plurality of first candidate feature points and the plurality of second candidate feature points to generate the first feature set of the previous image and the second feature set of the current image.

11. The image processing method for perspective conversion according to claim 10, further comprising: The third key point set in the current image is updated based on the second key point set and a basic matrix in the previous image, wherein the basic matrix indicates a corresponding perspective conversion relationship between the previous image and the current image.

12. The image processing method for perspective conversion according to claim 11, wherein the basic matrix is ​​generated by the following steps: The fundamental matrix is ​​calculated based on the first feature set of the previous image and the second feature set of the current image.

13. The image processing method for perspective conversion according to claim 11, wherein the step of updating the third key point set comprises: converting the second key point set in the previous image into a plurality of straight lines in the current image using the basic matrix; as well as Based on the plurality of straight lines, the third key point set in the current image is updated.

14. The image processing method for perspective conversion according to claim 13, wherein the step of updating the third key point set in the current image based on the plurality of straight lines comprises: determining whether each of the third key point set in the current image is located on one of the plurality of corresponding straight lines in the current image; as well as At least one noise keypoint in the third keypoint set is deleted to update the third keypoint set in the current image, wherein each of the at least one noise keypoint is not located on the corresponding straight line in the current image.

Images