A method of image processing and related apparatus

Abstract

The application is suitable for the technical field of image processing. The embodiment of the application provides a kind of image processing method and related device, and the original image is corrected and the re-projection matrix H2' is determined in the internal parameter matrix M in the distortion correction process, then the re-projection matrix H2' is projected as target image according to the image after distortion correction, the image of high-rise monitoring can be preprocessed, and the re-projected image is obtained, to solve the problem that the current high-rise detection effect is poor.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to an image processing method and related apparatus. Background Technology

[0002] Currently, with the development of society and the economy, there are more and more high-rise buildings in cities. At the same time, many incidents of objects being thrown from high-rise buildings have also occurred. To monitor these incidents, many places install specialized surveillance equipment to monitor high-rise buildings in order to determine responsibility for related accidents.

[0003] Existing high-rise building surveillance systems primarily focus on early warning and detection of objects being thrown from heights. However, due to the "near objects appear larger, far objects appear smaller" imaging principle in existing surveillance equipment, direct event detection on the images can be ineffective when monitoring tall buildings, as objects at higher elevations may have too small a pixel size for proper detection.

[0004] Therefore, there is an urgent need for an image processing method to preprocess images from high-rise building surveillance to improve detection efficiency. Summary of the Invention

[0005] This application provides an image processing method and related apparatus that can preprocess images of high-rise building monitoring to obtain reprojected images, thus solving the problem of poor detection results in high-rise buildings.

[0006] In a first aspect, embodiments of this application provide an image processing method, comprising: acquiring an original image; performing distortion correction on the original image and determining an intrinsic parameter matrix M during the distortion correction process; determining a reprojection matrix H2′ based on the intrinsic parameter matrix M; and reprojecting the distortion-corrected image into a target image based on the reprojection matrix H2′.

[0007] This application provides an image processing method that performs distortion correction on the original image and determines the reprojection matrix H2′ through the intrinsic parameter matrix M during the distortion correction process. Then, the distortion-corrected image is reprojected into the target image according to the reprojection matrix H2′. This method can preprocess images of high-rise building monitoring to obtain the reprojected image, thus solving the problem of poor detection effect of high-rise buildings.

[0008] In conjunction with the first aspect, in one implementation of the embodiments of this application, determining the reprojection matrix H2′ based on the intrinsic parameter matrix M includes: determining the reprojection matrix H2′ based on the determined projection intrinsic parameter matrix M2′, the determined projection rotation matrix R′, and the intrinsic parameter matrix M.

[0009] In conjunction with the first aspect, in one implementation of the embodiments of this application, the projection rotation matrix R′ is determined by the following steps: selecting four reference points in the building area of ​​the image after distortion correction; determining the circumscribed rectangle of the four reference points; determining the homography matrix H based on the four reference points and the four vertices of the circumscribed rectangle; and determining the projection intrinsic parameter matrix M′ and the projection rotation matrix R′ based on the homography matrix H and the intrinsic parameter matrix M.

[0010] In conjunction with the first aspect, in one implementation of the embodiments of this application, determining the projection intrinsic parameter matrix M′ and the projection rotation matrix R′ based on the homography matrix H and the intrinsic parameter matrix M includes: calculating the product of the homography matrix H and the intrinsic parameter matrix M to obtain a first matrix Q; performing matrix decomposition on the first matrix Q to obtain an upper triangular matrix and an orthogonal unit matrix; setting the upper triangular matrix as the projection intrinsic parameter matrix M′, and setting the orthogonal unit matrix as the projection rotation matrix R′.

[0011] In conjunction with the first aspect, in one implementation of the embodiments of this application, the projection rotation matrix R′ can be determined by the following steps: obtaining a preset set of xyz rotation angles; and determining the projection rotation matrix R′ based on the set of xyz rotation angles.

[0012] In conjunction with the first aspect, in one implementation of the embodiments of this application, the projection intrinsic parameter matrix M2′ is determined by the following steps: calculating a new projection intrinsic parameter matrix M2′ based on the projection intrinsic parameter matrix M′ and the homography matrix H.

[0013] In conjunction with the first aspect, in one implementation of this application, calculating a new projection intrinsic parameter matrix M2′ based on the projection intrinsic parameter matrix M′ and the homography matrix H includes: selecting a rectangle RV of the distortion-corrected image; projecting the rectangle RV onto the homography matrix H to obtain a polygon RV′; and calculating a first scaling factor (scale) based on the scaling relationship between the rectangle RV and the polygon RV′. x Second multiple scale y According to the first multiple scale x Second multiple scale y Transform the projection intrinsic parameter matrix M′ into a new projection intrinsic parameter matrix M2′.

[0014] In conjunction with the first aspect, in one implementation of this application embodiment, determining the reprojection matrix H2′ based on the projection intrinsic parameter matrix M2′, the projection rotation matrix R′, and the intrinsic parameter matrix M includes: calculating the reprojection matrix H2′ using the reprojection matrix calculation formula; the reprojection matrix calculation formula is: H'2=M'2R'M -1 .

[0015] Secondly, embodiments of this application provide an image processing apparatus, comprising: an acquisition module for acquiring an original image; a processing module for performing distortion correction on the original image and determining an intrinsic parameter matrix M during the distortion correction process, and determining a reprojection matrix H2′ based on the intrinsic parameter matrix M; and a reprojection module for reprojecting the distortion-corrected image into a target image based on the reprojection matrix H2′.

[0016] Thirdly, embodiments of this application provide a terminal device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in the first aspect.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method as described in the first aspect.

[0018] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute any of the methods described in the first aspect above.

[0019] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0020] The beneficial effects of the embodiments of this application compared with the prior art are:

[0021] This application provides an image processing method and related apparatus. The method corrects distortion of the original image and determines the reprojection matrix H2′ through the intrinsic parameter matrix M in the distortion correction process. Then, the image after distortion correction is reprojected into the target image according to the reprojection matrix H2′. This method can preprocess the images of high-rise building monitoring to obtain the reprojected image, thus solving the problem of poor detection effect of high-rise buildings. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart illustrating an image processing method provided in an embodiment of this application;

[0024] Figure 2 A flowchart detailing the steps of an image processing method provided in this application embodiment;

[0025] Figure 3 This is a schematic diagram of the image after distortion correction;

[0026] Figure 4 A simplified schematic diagram of the image after distortion correction;

[0027] Figure 5 This is a schematic diagram of the circumscribed rectangle in an embodiment of this application;

[0028] Figure 6 This is a schematic diagram of rectangular transformation in an embodiment of this application;

[0029] Figure 7 This is a schematic diagram of the target image in an embodiment of this application;

[0030] Figure 8 This is a simplified view of the target image in an embodiment of this application;

[0031] Figure 9 Example diagram of the terminal device provided in the embodiments of this application;

[0032] Figure 10 A schematic diagram of an image processing apparatus provided in an embodiment of this application. Detailed Implementation

[0033] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0034] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0035] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0036] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0037] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0038] This application provides an image processing method, primarily applied to monitoring equipment for high-rise building surveillance. Of course, in some possible embodiments, this method can also be applied to processing images captured by mobile phones or other devices. The following provides a detailed description of its application to monitoring equipment; other application scenarios can be implemented with reference to this application's embodiments, which will not elaborate further on these scenarios.

[0039] Figure 1 A flowchart illustrating an image processing method provided in an embodiment of this application. The process includes:

[0040] 101. Obtain the original image;

[0041] In this embodiment, the original image may be an image captured by a surveillance device. The original image includes the building area. Generally, the surveillance device is installed on the lower floors of a tall building and then captures images looking towards the top of the building. Therefore, the original image typically includes the building area.

[0042] 102. Perform distortion correction on the original image and determine the intrinsic parameter matrix M during the distortion correction process;

[0043] In the embodiments of this application, a camera calibration algorithm can be used to correct distortion in the original image.

[0044] For example, distortion correction can be performed using the Zhang Zhengyou calibration method. The specific model is as follows:

[0045]

[0046] Among them, (x w ,y w ,z wLet (u, v) be the real-world 3D coordinates, and (u, v) be the pixel coordinates on the image. R is a 3x3 rotation matrix, t is a 3x1 translation vector, f is the camera's focal length, dx and dy are the physical dimensions of the pixel along the x and y axes, respectively, and (u0, v0) are the coordinates of the origin of the image coordinate system. Parameters R, t, ... (u0, v0) can both be calculated using the camera calibration algorithm.

[0047] When the origin of the coordinate system of the 3D real world coincides with that of the image camera, the parameter t is 0, and the formula at this time is:

[0048]

[0049] Where M is the intrinsic parameter matrix of the camera. The intrinsic parameter matrix M can be obtained using the formula above.

[0050] 103. Determine the reprojection matrix H2′ based on the intrinsic parameter matrix M;

[0051] In this embodiment of the application, the reprojection matrix H2′ can be calculated using the reprojection matrix calculation formula;

[0052] The formula for calculating the reprojection matrix is: H2′=M2′R′M -1 .

[0053] Where M2′ is the determined projection intrinsic parameter matrix M2′, R′ is the determined projection rotation matrix R′, and M is the intrinsic parameter matrix M.

[0054] When the surveillance camera is fixed, the projection intrinsic parameter matrix M2′ and the projection rotation matrix R′ are pre-calculated. When processing multiple frames of images, it is only necessary to determine the intrinsic parameter matrix M, and the reprojection matrix H2′ can be determined according to the above reprojection matrix calculation formula.

[0055] When the surveillance camera is not fixed, or when a fixed surveillance camera is processing images for the first time, it can be done through... Figure 2 The steps shown are for calculating the projection intrinsic parameter matrix M2′ and the projection rotation matrix R′:

[0056] 1031. Select four reference points for the building area;

[0057] Figure 3 This is a schematic diagram of the image after distortion correction. Figure 4 This is a simplified diagram of the image after distortion correction. It can be seen that the image still retains the "near objects appear larger, far objects appear smaller" imaging relationship. If event detection is performed directly on the image, when the monitored building is tall, objects at higher elevations will be poorly detected due to their small pixel size.

[0058] In this embodiment, four reference points can be selected within the building area of ​​the image after distortion correction. Generally, four points on the building that form a rectangle in the orthographic projection direction of the camera are selected. To ensure that the selected points are rectangular in the orthographic projection direction, some corner points of the building itself with vertical relationships can be utilized, such as… Figure 3 , Figure 4 The four reference points obtained are: P1(u1,v1), P2(u2,v2), P3(u3,v3), and P4(u4,v4). These four points only need to satisfy the condition that they form a rectangle in the orthographic projection direction. The selection method can be manual or calculated using other computer-related algorithms; this application does not limit this approach.

[0059] In this embodiment, the greater the distance between the four selected points, the better. This is to prevent the selected points from not forming a rectangle, and greater distance reduces errors from manual selection.

[0060] 1032. Determine the bounding rectangle of the four reference points;

[0061] Figure 5 This is a schematic diagram of the circumscribed rectangle in an embodiment of this application. It can be seen that the circumscribed rectangle of trapezoid P1P2P3P4 can be P1′P2′P3′P4′. This can be determined manually or calculated using other computer-related algorithms. For example, if the four reference points are P1(u1,v1), P2(u2,v2), P3(u3,v3), and P4(u4,v4), the coordinates of the top-left corner P1′ can be (min(u2,u1),min(v1,v4)), the coordinates of the bottom-left corner P2′ can be (min(u2,u1),max(v2,v3)), the coordinates of the bottom-right corner P3′ can be (max(u3,u4),max(v2,v3)), and the coordinates of the top-right corner P4′ can be (max(u3,u4),min(v1,v4)).

[0062] 1033. Determine the homography matrix H based on the four reference points and the four vertices of the circumscribed rectangle;

[0063] In this embodiment of the application, the four reference points can first be set as P1(u1,v1), P2(u2,v2), P3(u3,v3), and P4(u4,v4);

[0064] Then, the four vertices of the circumscribed rectangle are set as P1′(u1′,v1′), P2′(u2′,v2′), P3′(u3′,v3′), and P4′(u4′,v4′);

[0065] Then, solve the first relation to determine the homography matrix H;

[0066] The first relation is:

[0067] in, Let be the matrix corresponding to the four vertices of the circumscribed rectangle, and α be the matrix that makes α = ... A normalization coefficient with the last term being 1. This is the matrix corresponding to the four reference points.

[0068] To solve this problem, the homography matrix H can be expanded to obtain:

[0069]

[0070] So:

[0071] u i ′=α(h1u i +h2v i +h3);

[0072] v i ′=α(h4u i +h5v i +h6);

[0073] 1=α(h7u i +h8v i +h9);

[0074] Eliminating α yields:

[0075] u i ′(h7u i +h8v i +h9)=(h1u i +h2v i +h3);

[0076] v i ′(h7u i +h8v i +h9)=(h4u i +h5v i +h6);

[0077] After sorting, we can obtain:

[0078] h7u i u i ′+h8v i u i ′+h9u i ′-h1u i -h2v i -h3 = 0;

[0079] h7u i vi ′+h8v i v i ′+h9v i ′-h4u i -h5v i -h6 = 0;

[0080] Rewriting the above equation as matrix and vector multiplication, we have:

[0081] A i h = 0;

[0082]

[0083] h = [h1h2h3h4h5h6h7h8h9];

[0084] As can be seen from the above formula, one pair of matching points can provide two linear equations. h has nine unknowns. Let h9 = 1, and the remaining eight unknowns of h can be solved with eight equations from four pairs of matching points.

[0085] After solving for h, the homography matrix H can be obtained.

[0086] 1034. Determine the reprojection matrix H2′ based on the homography matrix H and the intrinsic parameter matrix M;

[0087] In this embodiment, reprojection refers to reprojecting the distortion-corrected image onto the target image. To achieve reprojection, the reprojection matrix H2′ can be determined based on the homography matrix H and the intrinsic parameter matrix M. This is performed in the following steps:

[0088] First: Determine the projection intrinsic parameter matrix M′ and the projection rotation matrix R′ based on the homography matrix H and the intrinsic parameter matrix M;

[0089] The reprojection process can be understood as a rotation and central projection of the camera coordinate system on the original basis.

[0090] The original image satisfies the following formula:

[0091]

[0092] The reprojected image then satisfies:

[0093]

[0094] because:

[0095]

[0096] Therefore, we can conclude that:

[0097] H=M′R′M -1 ;

[0098] Since the camera's intrinsic parameter matrix M can be obtained through step 102, and the homography matrix H is also obtained in step 105, therefore:

[0099] Q = HM = M′R′;

[0100] In the formula, the projection intrinsic parameter matrix M′ is an upper triangular matrix, and the projection rotation matrix R′ is an orthogonal matrix. Therefore, by performing "RQ" decomposition on the matrix, the projection intrinsic parameter matrix M′ and the projection rotation matrix R′ can be obtained. It can be understood that matrix Q can theoretically be decomposed into RQ, so the upper triangular matrix and the identity orthogonal matrix after decomposition can be regarded as the intrinsic parameter matrix and the rotation matrix, respectively.

[0101] Therefore, based on the homography matrix H and the intrinsic parameter matrix M, the product of the homography matrix H and the intrinsic parameter matrix M can be calculated to obtain the first matrix Q; then, the first matrix Q can be decomposed (e.g., RQ decomposition) to obtain the upper triangular matrix and the identity orthogonal matrix; finally, the obtained upper triangular matrix can be set as the projection intrinsic parameter matrix M′, and the obtained identity orthogonal matrix can be set as the projection rotation matrix R′.

[0102] In some embodiments, the method of obtaining the projection rotation matrix R′ by selecting four reference points can be omitted. The projection rotation matrix R′ can then be determined by the following steps: obtaining a preset set of xyz rotation angles; and determining the projection rotation matrix R′ based on this set of xyz rotation angles. In this case, the intrinsic parameter matrix M and the projection intrinsic parameter matrix M′ can be set to be equal.

[0103] In some cases, the distortion-corrected image can be directly projected based on the projection intrinsic parameter matrix M′. However, the resulting image contains numerous black borders, making it unsuitable for direct display. Therefore, it is necessary to adjust the projection intrinsic parameter matrix M′ to obtain a new projection intrinsic parameter matrix M2′. The adjustment method is not unique; the following methods can be used for adjustment.

[0104] Second: Calculate the new projection intrinsic parameter matrix M2' based on the projection intrinsic parameter matrix M' and the homography matrix H;

[0105] First, select the rectangle RV in the original image after distortion correction;

[0106] Figure 6 This is a schematic diagram of the rectangular transformation in an embodiment of this application. In this embodiment, the rectangle RV of the distortion-corrected image is first selected, with the four vertices of RV1(u1,v1), RV2(u2,v2), RV3(u3,v3), and RV4(u4,v4). The selection method can be manual or calculated using other computer-related algorithms; this embodiment does not limit the selection process.

[0107] Then, the rectangle RV is projected onto the homography matrix H to obtain the polygon RV′;

[0108] like Figure 6 As shown, the rectangle RV is projected onto the homography matrix H to obtain the polygon RV′. The four vertices of the polygon RV′ are RV1′(u1′,v1′), RV2′(u2′,v2′), RV3′(u3′,v3′), and RV4′(u4′,v4′).

[0109] Then, the first scaling factor (scale) is calculated based on the scaling relationship between the rectangle RV and the polygon RV′. x Second multiple scale y ;

[0110] Specifically, the first multiple (scale) can be calculated according to the first multiple formula. x And calculate the second multiple scale according to the second multiple formula. y ;

[0111] The formula for the first multiple is: scale x =W / W′;

[0112] The formula for the second multiple is:

[0113] Where W is the width of rectangle RV, and W′ is the width of the largest rectangle embedded within polygon RV′. and These are the given parameters, i.e., the parameters of the intrinsic parameter matrix M2′ in the next step.

[0114] Figure 6 This illustrates the case where a polygon RV′ contains a maximum rectangle, constrained by vertices RV1′(u1′,v1′), RV2′(u2′,v2′), RV3′(u3′,v3′), and RV4′(u4′,v4′). The top-left vertex of this maximum rectangle is (max(u2′,u1′),max(v4′,v1′)), and the bottom-right vertex is (min(u4′,u3′),min(v2′,v3′)). The width W′ and height H′ of the maximum rectangle are then:

[0115] W′=min(u4′,u3′)-max(u2′,u1′);

[0116] H′=min(v2′,v3′)-max(v4′,v1′).

[0117] Finally, based on the first multiple scale x Second multiple scale yTransform the projection intrinsic parameter matrix M′ into a new projection intrinsic parameter matrix M2′.

[0118] Let the expression for the projection intrinsic parameter matrix M′ be:

[0119]

[0120] The parameter γ is introduced because the selected four reference points cannot guarantee a strict rectangular relationship. This is understandable because, although the requirement is to select four reference points that form a rectangle on orthographic projection, due to human or image errors, the 90-degree angle between the vertices cannot be strictly guaranteed; the selected points may actually be 89 degrees. If the selected rectangle is 90 degrees, then parameter γ is 0. Parameter γ = -cotθ / dx represents the physical dimension of a pixel along the x-axis, which is 1 / dx the actual length of a pixel in the x-direction. Here, θ is the angle between the horizontal and vertical coordinate axes in the pixel coordinate system. This embodiment introduces a parameter that does not strictly require the vertices of the four selected points to be 90 degrees, allowing for an error in visual selection and improving the robustness of the proposed solution.

[0121] Then, by combining the projection intrinsic parameter matrix M′ obtained from the RQ decomposition and its expression, the parameters can be determined. The specific values ​​of γ, u0′, and v0′.

[0122] Then, based on the width W′ of the inset maximum rectangle, the parameters... The first multiple, scalex, and the second multiple, scaley, can be obtained using the first multiple formula and the second multiple formula.

[0123] And the new projection center (u0″, v0″) can be determined as:

[0124] u”0=[u0′-max(u2′,u1′)]*scale x ;

[0125] v”0=[v0′-max(v4′,v1′)]*scale y ;

[0126] Then, the new projection intrinsic parameter matrix M2′ is determined as follows:

[0127]

[0128] According to the first multiple formula and the second multiple formula, we can obtain: Therefore, in this embodiment of the application, the reprojection scheme determined according to the above-mentioned projection intrinsic parameter matrix M2′ has consistent pixel resolution in the width and height directions after reprojection, making the picture more in line with human visual perception.

[0129] Third: Determine the reprojection matrix H2′ based on the projection intrinsic parameter matrix M2′, the projection rotation matrix R′, and the intrinsic parameter matrix M.

[0130] In this embodiment of the application, the reprojection matrix H2′ can be calculated using the reprojection matrix calculation formula;

[0131] The formula for calculating the reprojection matrix is: H2′=M2′R′M -1 .

[0132] 104. Reproject the distortion-corrected image into the target image based on the reprojection matrix H2′.

[0133] First, the coordinate correspondence between the distortion-corrected image and the target image is determined based on the reprojection matrix H2′. This coordinate correspondence is as follows:

[0134]

[0135] in. The coordinate matrix corresponding to the target image. This is the coordinate matrix corresponding to the distortion-corrected image. α″ is the normalization coefficient.

[0136] Where the integer coordinates (u″, v″) correspond to (u, v), if they have a decimal part, interpolation can be performed using an interpolation algorithm similar to bilinear interpolation, or the coordinates can be directly rounded down.

[0137] Then, the distortion-corrected image is reprojected into the target image based on the coordinate position correspondence.

[0138] Figure 7 This is a schematic diagram of the target image in an embodiment of this application. Figure 8 This is a simplified view of the target image in an embodiment of this application. As can be seen, points P1″, P2″, P3″, and P4″ in the original image are finally transformed into P1″, P2″, P3″, and P4″.

[0139] 105. Perform parabolic detection based on the target image.

[0140] In applications involving objects falling from high-rise buildings, object detection can also be performed based on the target image. Specifically, firstly, moving objects in the image can be identified by detecting the foreground. Foreground detection algorithms can use methods such as codebooks. Then, based on whether the trajectory of the moving object in the image conforms to the law of parabolic fall, it can be determined whether a parabolic event has occurred. The law of parabolic fall includes whether the object falls gradually from top to bottom, and whether the trajectory generally conforms to the law of parabola.

[0141] Figure 9This is an example diagram of a terminal device provided in an embodiment of this application. The terminal device 9 includes a processor 901 and a memory 902. The processor 901 and the memory 902 communicate with each other. The processor 901 is also connected to multiple lights via a communication module. The memory 902 stores a computer program 903. When the processor 901 executes the computer program 903, it implements the above-described functionality. Figure 1 or Figure 2 The methods of the corresponding embodiments.

[0142] Figure 10 A schematic diagram of an image processing apparatus provided in an embodiment of this application. The image processing apparatus 10 includes:

[0143] Module 1001 is used to execute or implement the above. Figure 1 Step 101 in the corresponding embodiments;

[0144] Processing module 1002 is used to execute or implement the above. Figure 1 Steps 102 and 103 in the corresponding embodiments may be used to perform or implement the above. Figure 2 The corresponding processes of each embodiment;

[0145] Reprojection module 1003 is used to perform or implement the above. Figure 1 Step 104 in the corresponding embodiments.

[0146] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0147] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0148] This application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0149] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0150] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0151] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0152] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0153] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0154] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0155] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0156] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An image processing method, characterized in that, include: Obtain the original image; the original image includes building areas; The original image is subjected to distortion correction, and the intrinsic parameter matrix M is determined during the distortion correction process; Determining the reprojection matrix H2′ based on the intrinsic parameter matrix M includes: determining the reprojection matrix H2′ based on the determined projection intrinsic parameter matrix M2′, the determined projection rotation matrix R′, and the intrinsic parameter matrix M; the projection intrinsic parameter matrix M2′ is determined by the following steps: calculating a new projection intrinsic parameter matrix M2′ based on the projection intrinsic parameter matrix M′ and the homography matrix H; The distortion-corrected image is reprojected into the target image according to the reprojection matrix H2′. Perform parabolic detection based on the target image; The step of calculating the new projection intrinsic parameter matrix M2′ based on the projection intrinsic parameter matrix M′ and the homography matrix H includes: Select the rectangle RV of the distortion-corrected image; The rectangle RV is projected onto the homography matrix H to obtain the polygon RV′; The first scaling factor (scale) is calculated based on the scaling relationship between the rectangle RV and the polygon RV′. x Second multiple scale y ; According to the first multiple scale x and the second multiple scale y The projection intrinsic parameter matrix M′ is transformed into the new projection intrinsic parameter matrix M2′; the projection rotation matrix R′ is determined through the following steps: Select four reference points for the building region in the image after distortion correction; Determine the bounding rectangle of the four reference points; The homography matrix H is determined based on the four reference points and the four vertices of the circumscribed rectangle; The projection intrinsic parameter matrix M′ and the projection rotation matrix R′ are determined based on the homography matrix H and the intrinsic parameter matrix M.

2. The method as described in claim 1, characterized in that, The step of determining the projection intrinsic parameter matrix M′ and the projection rotation matrix R′ based on the homography matrix H and the intrinsic parameter matrix M includes: The first matrix Q is obtained by calculating the product of the homography matrix H and the intrinsic parameter matrix M; Matrix decomposition of the first matrix Q yields an upper triangular matrix and an identity orthogonal matrix; The upper triangular matrix is ​​set as the projection intrinsic parameter matrix M′, and the identity orthogonal matrix is ​​set as the projection rotation matrix R′.

3. The method as described in claim 1, characterized in that, The projection rotation matrix R′ can be determined through the following steps: Obtain a preset set of xyz rotation angles; The projection rotation matrix R′ is determined based on the xyz rotation angle.

4. The method as described in claim 1, characterized in that, The step of determining the reprojection matrix H2′ based on the projection intrinsic parameter matrix M2′, the projection rotation matrix R′, and the intrinsic parameter matrix M includes: The reprojection matrix H2′ is calculated using the reprojection matrix calculation formula; The formula for calculating the reprojection matrix is: .

5. An image processing apparatus, characterized in that, include: The acquisition module is used to acquire the original image, which includes building areas. The processing module is used to perform distortion correction on the original image and determine the intrinsic parameter matrix M in the distortion correction process, and determine the reprojection matrix H2′ based on the intrinsic parameter matrix M, including: determining the reprojection matrix H2′ based on the determined projection intrinsic parameter matrix M2′, the determined projection rotation matrix R′, and the intrinsic parameter matrix M; the projection intrinsic parameter matrix M2′ is determined through the following steps: calculating a new projection intrinsic parameter matrix M2′ based on the projection intrinsic parameter matrix M′ and the homography matrix H; A reprojection module is used to reproject the distortion-corrected image into a target image according to the reprojection matrix H2′. The image processing apparatus is also used to perform parabolic detection based on the target image; The step of calculating the new projection intrinsic parameter matrix M2′ based on the projection intrinsic parameter matrix M′ and the homography matrix H includes: Select the rectangle RV of the distortion-corrected image; The rectangle RV is projected onto the homography matrix H to obtain the polygon RV′; The first scaling factor (scale) is calculated based on the scaling relationship between the rectangle RV and the polygon RV′. x Second multiple scale y ; According to the first multiple scale x and the second multiple scale y Transform the projection intrinsic parameter matrix M′ into the new projection intrinsic parameter matrix M2′; The projection rotation matrix R′ is determined through the following steps: Select four reference points for the building region in the image after distortion correction; Determine the bounding rectangle of the four reference points; The homography matrix H is determined based on the four reference points and the four vertices of the circumscribed rectangle; The projection intrinsic parameter matrix M′ and the projection rotation matrix R′ are determined based on the homography matrix H and the intrinsic parameter matrix M.

6. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 4.

Images