Acquisition device, method and electronic device for homography
By determining line segments and projecting vertices based on vanishing points, the method automates the homography calculation process, enhancing accuracy and reducing manual labor in obtaining homography.
Patent Information
- Application Number
- JP2024199349
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-11-23
- Filing Date
- 2024-11-15
- Publication Date
- 2025-06-04
AI Technical Summary
The existing methods for obtaining homography, which involve manual labeling of points in an image and searching for corresponding points in a layout diagram, are labor-intensive and prone to inaccuracies.
A method that determines line segments based on vanishing points in an image, sets a rectangle within the image, projects the vertices of this rectangle onto corresponding points in a layout diagram, and calculates the homography using these coordinates.
This approach improves the accuracy of homography calculation and reduces the workload by automating the process of setting a rectangle and matching its vertices to the layout diagram, thereby eliminating the need for manual point labeling and search.
Smart Images
Figure 2025085079000001_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to the technical field of image processing.
Background Art
[0002] Homography can represent a projective mapping from one plane to another plane. For example, a homography may be established between an image captured by a camera for a predetermined space and a layout diagram of the predetermined space, and by this homography, any point in the image can be projected onto the layout diagram.
[0003] In the prior art, usually, by manual labeling, a plurality of points are labeled on an image captured by a camera, corresponding points are searched for in the corresponding layout diagram, and then a homography matrix is calculated to obtain the homography.
[0004] It should be noted that the above description of the technical background is an explanation for clearly and completely understanding the technical solution of the present invention, and is described for those skilled in the art to understand. These technical solutions are only described as the background art part of the present invention and are not well-known to those skilled in the art.
Summary of the Invention
Problems to be Solved by the Invention
[0005] However, according to the discovery of the inventor of the present invention, in the method of obtaining the above homography, since it is necessary to manually label points in the image and search for corresponding points in the layout diagram, there are problems such as a large amount of work, inaccurate labeling, or inaccurate corresponding points found.
[0006] In view of at least one of the above technical problems, an embodiment of the present invention provides a homography acquisition device, method, and electronic device that can improve the accuracy of homography calculation and reduce the workload by determining line segments based on the vanishing points of an image, setting a rectangle in the image, and projecting each vertex of the rectangle to a corresponding point in the layout diagram, thereby calculating the homography.
Means for Solving the Problem
[0007] In one aspect of an embodiment of the present invention, there is provided a homography acquisition device including: a division unit that divides an image captured by a camera; a determination unit that determines vanishing points in the image, obtains line segments corresponding to different vanishing points, retains line segments distributed on a predetermined object based on the division result of the image, and extracts a rectangle in the image based on the line segments distributed on the predetermined object; a projection unit that projects the vertices of the rectangle to corresponding points in the layout diagram; and a calculation unit that calculates homography based on the coordinates of the vertices of the rectangle and the coordinates of the corresponding points in the layout diagram.
[0008] In another aspect of an embodiment of the present invention, there is provided a homography acquisition method including: dividing an image captured by a camera; determining vanishing points in the image, obtaining line segments corresponding to different vanishing points, retaining line segments distributed on a predetermined object based on the division result of the image, and extracting a rectangle in the image based on the line segments distributed on the predetermined object; projecting the vertices of the rectangle to corresponding points in the layout diagram; and calculating homography based on the coordinates of the vertices of the rectangle and the coordinates of the corresponding points in the layout diagram.
[0009] In another aspect of an embodiment of the present invention, there is provided an electronic device including a memory storing a computer program and a processor, wherein the processor is configured to execute the computer program to implement the above-described method for obtaining a homography.
[0010] One of the advantageous effects of the embodiments of the present invention is as follows. By determining line segments based on the vanishing point of an image, setting a rectangle in the image, and projecting each vertex of the rectangle onto corresponding points in the layout diagram, the accuracy of homography calculation can be improved and the workload can be reduced by calculating the homography.
[0011] Specific embodiments of the present invention are disclosed in detail as shown in the following description and drawings, showing ways in which the principles of the present invention can be employed. It should be noted that the embodiments of the present invention are not limited in scope. The embodiments of the present invention include various changes, modifications, and equivalents within the spirit and scope of the appended claims.
Brief Description of the Drawings
[0012] The drawings included herein are for understanding the embodiments of the present invention, form part of this specification, and are for exemplifying the embodiments of the present invention and explaining the principles of the present invention in conjunction with the written description. It should be noted that the drawings described herein are only for explaining the embodiments of the present invention, and those skilled in the art can easily obtain other drawings based on these drawings.
Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Figure 11
Figure 12
Mode for Carrying Out the Invention
[0013] The above and other features of the present invention will become clear from the drawings and the following description. In the specification and the drawings, specific embodiments of the present invention, that is, those representing some embodiments in accordance with the principles of the present invention are disclosed. Note that the present invention is not limited to the described embodiments, and the present invention includes all modifications, variations, and equivalents within the scope of the claims.
[0014] In the embodiments of the present invention, the terms "first" and "second" are for distinguishing different elements by name, and do not mean the spatial arrangement or temporal order of these elements, etc., and these elements are not limited to these terms. The term "and / or" includes any one or a combination of one or more of the recited terms. The terms "comprising", "including", and "having" mean the presence of the described features, elements, elements, or members, but do not exclude the presence or addition of one or more other features, elements, elements, or members.
[0015] In the embodiments of the present invention, singular forms such as "a", "the", etc. include plural forms, meaning "a kind" or "a category", and are not limited to "one". Also, the term "the foregoing" includes both singular and plural forms unless clearly indicated otherwise in the context. Further, unless clearly indicated otherwise in the context, the term "according to" means "at least partially according to", and the term "based on" means "at least partially based on".
[0016] Features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or used in place of features in other embodiments. The terms "comprising" or "including" mean the presence of the described features, elements, elements or members, but do not exclude the presence or addition of one or more other features, elements, elements or members.
[0017] <Example 1> Embodiments of the present invention provide a method for obtaining homography.
[0018] FIG. 1 is a schematic diagram of an example of a method for obtaining homography according to an embodiment of the present invention. As shown in FIG. 1, the method for obtaining homography includes the following steps.
[0019] Step 101: Divide the image captured by the camera.
[0020] Step 102: Determine the vanishing points in the image, obtain the line segments corresponding to different vanishing points, retain the line segments distributed on a predetermined object based on the segmentation result of the image, and extract a rectangle in the image based on the line segments distributed on the predetermined object.
[0021] Step 103: Project the vertices of the rectangle onto the corresponding points in the layout diagram.
[0022] Step 104: Calculate the homography based on the coordinates of the vertices of the rectangle and the coordinates of the corresponding points in the layout diagram.
[0023] According to an embodiment of the present invention, by determining a line segment based on the vanishing point of an image, setting a rectangle in the image, and projecting each vertex of the rectangle to a corresponding point in the layout diagram to calculate a homography, a rectangle can be automatically set and its vertices can be made to correspond to the layout diagram, avoiding manual point labeling and searching for corresponding points in the layout diagram, thus improving the accuracy of homography calculation and reducing the workload.
[0024] As shown in FIG. 1, the method may further include the following steps.
[0025] Step 105: Display the image content in the rectangle in the image at a position corresponding to the rectangle in the layout diagram.
[0026] In step 105, for example, the image content within the rectangle in the image may be pasted at a position and within a range corresponding to the rectangle in the layout diagram. According to step 105, the content of the rectangular area in the image can be further displayed at the corresponding position in the layout diagram, improving the visibility of the layout diagram.
[0027] FIG. 2 is a schematic diagram of an example corresponding to steps 102, 103, and 105. As shown in FIG. 2, by step 102, a rectangle 201 is extracted from an image 200 captured by a camera. By step 103, the four vertices of the rectangle 201 are respectively projected to corresponding points 2022 in the layout diagram 202. By step 105, the image content 2011 in the rectangle 201 is displayed at a position corresponding to the rectangle 201 in the layout diagram 202. Also, the position of the camera 203 is displayed in the layout diagram 202.
[0028] In the present invention, the layout diagram may be a layout diagram for a predetermined space. For example, the predetermined space may be a space such as an office, an office building, or a house. The layout diagram may reflect the horizontal layout of the predetermined space. Note that the present invention is not limited thereto, and the layout diagram may also reflect the layout in a direction perpendicular to the horizontal direction of the predetermined space.
[0029] In the present invention, the image captured by the camera may be an image captured by the camera for at least a part of the predetermined space.
[0030] In at least one embodiment of step 101, the image may be segmented using a semantic segmentation method, and the objects in the image may be segmented into different categories, such as the floor, walls, furniture, etc. Further, the present invention may also segment the image using other methods.
[0031] Figure 3 is a schematic diagram of an example of the result of segmenting the image 200 captured by the camera. As shown in Figure 3, in the segmented image 300, the same color or grayscale may be assigned to the objects of the same category, and different colors or grayscales may be assigned to the objects of different categories. As shown in Figure 3, the segmented image 300 may have objects such as the floor 301 and the walls 302.
[0032] In step 102, based on the vanishing point of the image and the segmentation result of the image, rectangles in the image may be extracted. Here, the straight lines parallel to each other in the space intersect at the vanishing point in the image plane.
[0033] FIG. 4 is a schematic diagram of an example of a vanishing point of an image 200 captured by a camera. As shown in FIG. 4, the vanishing point of the line segment in the first direction (for example, the left - right direction in the horizontal direction) in the image 200 is 204, and the vanishing point of the line segment in the second direction (for example, the front - back direction in the horizontal direction) in the image 200 is 205, and the first direction and the second direction may be perpendicular to each other. Regarding the method for detecting the vanishing point and the line segment corresponding to the vanishing point, related technologies may be referred to, and the description is omitted in this specification.
[0034] In step 102, after detecting the line segment based on the vanishing point, the line segment on a predetermined object in the image may be retained. Here, the predetermined object may be determined based on the image segmentation result in step 101. For example, the predetermined object may be a floor, a wall, etc. In the present invention, the case where the predetermined object is a floor is taken as an example for explanation.
[0035] In at least some embodiments of step 102, in order to extract a rectangle, the line segments of the floor may be combined. For example, in FIG. 4, the line segments 2041, 2042 along the first direction on the floor 206 and the line segments 2051, 2052 along the second direction may be extracted. Here, the vanishing point of the line segments 2041, 2042 along the first direction is 204, and the vanishing point of the line segments 2051, 2052 along the second direction is 205. Thereby, the rectangle 201 surrounded by the line segments 2041, 2042 and the line segments 2051, 2052 is extracted.
[0036] In step 103, each vertex of the rectangle 201 is projected onto the corresponding point in the layout diagram 202.
[0037] FIG. 5 is a schematic diagram of an example of step 103. As shown in FIG. 5, step 103 may include the following steps.
[0038] Step 1031: The extending direction of the side of the rectangle closest to the camera is taken as the direction of the first axis.
[0039] Step 1032: Based on the point closest to the camera on the side of the rectangle closest to the camera, determine the first shape of the rectangle when the rectangle is located on one side of the camera in the direction of the second axis perpendicular to the first axis, and determine the second shape of the rectangle when the rectangle is located on the other side of the camera.
[0040] Step 1033: Based on the first shape, determine the first rectangle in the layout diagram, and based on the second shape, determine the second rectangle in the layout diagram.
[0041] Step 1034: Based on the point closest to the camera on the side of the rectangle closest to the camera, determine the third shape of the rectangle when the rectangle is located on one side of the camera in the direction of the first axis, and determine the fourth shape of the rectangle when the rectangle is located on the other side of the camera.
[0042] Step 1035: Based on the third shape, determine the third rectangle in the layout diagram, and based on the fourth shape, determine the fourth rectangle in the layout diagram.
[0043] Step 1036: Based on the positions of the first rectangle, the second rectangle, the third rectangle, and the fourth rectangle in the layout diagram, determine the rectangle in the layout diagram corresponding to the rectangle in the image according to a predetermined rule.
[0044] As shown in FIG. 4, in step 1031, the extending direction of the side of rectangle 201 closest to the camera (for example, line segment 2041) is taken as the direction X of the first axis. For example, the direction X of the first axis is the above-mentioned first direction. The direction perpendicular to the first axis is taken as the direction Y of the second axis, and the direction Y of the second axis is, for example, the above-mentioned second direction.
[0045] In step 1032, based on the point closest to the camera on the side of the rectangle 201 closest to the camera (for example, point 2043 in FIG. 4), determine the relative positions of the four vertices of the rectangle 201 when the rectangle 201 is located on one side (for example, the left side) of the camera in the direction Y of the second axis, obtain the first shape, and determine the relative positions of the four vertices of the rectangle 201 when the rectangle 201 is located on the other side (for example, the right side) of the camera, and obtain the second shape.
[0046] FIG. 6 is a schematic diagram of an example of the relative positional relationship in the direction Y of the second axis between the line segment 2041 of the rectangle 201 and the camera. In FIG. 6, when the rectangle 201 is located on one side (for example, the left side) of the camera in the direction Y of the second axis, the relative positional relationship between the line segment 2041 and the camera 203 is indicated by 61, and when the rectangle 201 is located on the other side (for example, the right side) of the camera in the direction Y of the second axis, the relative positional relationship between the line segment 2041 and the camera 203 is indicated by 62.
[0047] In step 1033, based on the first shape, determine the first rectangle in the layout diagram, and based on the second shape, determine the second rectangle in the layout diagram.
[0048] FIG. 7 is a schematic diagram of an example of the first rectangle and the second rectangle. In FIG. 7, the position of the first rectangle 2021 in the layout diagram 202 is indicated by 202a, and the position of the second rectangle 2022 in the layout diagram 202 is indicated by 202b.
[0049] In step 1034, based on the point closest to the camera on the side of the rectangle 201 closest to the camera (for example, point 2043 in FIG. 4), determine the relative positions of the four vertices of the rectangle when the rectangle is located on one side (for example, the upper side) of the camera in the direction X of the first axis, obtain the third shape, and determine the relative positions of the four vertices of the rectangle when the rectangle is located on the other side (for example, the lower side) of the camera in the direction X of the first axis, and obtain the fourth shape.
[0050] FIG. 8 is a schematic diagram of an example of the relative positional relationship in the direction X of the first axis between the line segment 2041 of the rectangle 201 and the camera. In FIG. 8, when the rectangle 201 is located on one side (for example, the upper side) of the camera in the direction X of the first axis, the relative positional relationship between the line segment 2041 and the camera 203 is indicated by 81, and when the rectangle 201 is located on the other side (for example, the lower side) of the camera in the direction X of the first axis, the relative positional relationship between the line segment 2041 and the camera 203 is indicated by 82.
[0051] In step 1035, based on the third shape obtained in step 1034, determine the third rectangle in the layout diagram, and based on the fourth shape, determine the fourth rectangle in the layout diagram.
[0052] FIG. 9 is a schematic diagram of an example of the third rectangle and the fourth rectangle. In FIG. 9, the position of the third rectangle 2023 in the layout diagram 202 is indicated by 202c, and the position of the fourth rectangle 2024 in the layout diagram 202 is indicated by 202d.
[0053] In step 1036, based on the positions of the first rectangle 2021, the second rectangle 2022, the third rectangle 2023, and the fourth rectangle 2024 in the layout diagram 202, determine the rectangle in the layout diagram 202 corresponding to the rectangle 201 in the image 200 according to a predetermined rule.
[0054] Here, the predetermined rule may include at least one of the following conditions.
[0055] Condition 1: In the layout diagram, the area of the region corresponding to the predetermined object occupied by the rectangle is the largest. For example, the area of the region corresponding to the floor occupied by the rectangle is the largest.
[0056] Condition 2: The rectangle does not exceed the range of the layout diagram.
[0057] According to the above-mentioned predetermined rules, as shown in FIGS. 7 and 9, in the layout diagram of the second rectangle 2022, the area of the region corresponding to the floor occupied is the largest, and the second rectangle 2022 does not exceed the range of the layout diagram 202. On the other hand, each of the first rectangle 2021, the third rectangle 2023, and the fourth rectangle 2024 at least partially exceeds the range of the layout diagram 202.
[0058] According to step 103, unreasonable rectangle positions in the layout diagram are excluded, and a reasonable rectangle position is determined such that the rectangle in the layout diagram (for example, the second rectangle 2022) corresponds to the rectangle (for example, rectangle 201) in the image captured by the camera, that is, each vertex of rectangle 201 in image 200 captured by the camera is projected onto each vertex of the second rectangle 2022 in layout diagram 202.
[0059] In step 104, a homography is calculated based on the coordinates of the vertices of the rectangle and the coordinates of the corresponding points in the layout diagram. FIG. 10 is a schematic diagram of an example of step 104. As shown in FIG. 10, step 104 may include the following steps.
[0060] Step 1041: Set the pixel coordinate system of the vertices of rectangle 201 in image 200 as the first coordinate system, and set the pixel coordinates of each vertex as the coordinates [p1, p2, p3, p4] in the first coordinate system.
[0061] Step 1042: Convert the world coordinate system of the corresponding points in layout diagram 202 into the pixel coordinate system of the layout diagram as the second coordinate system, and calculate the coordinates [p1’, p2’, p3’, p4’] of each vertex in the second coordinate system in layout diagram 202.
[0062] Step 1043: Calculate a homography matrix based on the coordinates of each vertex in the first coordinate system and the coordinates of each vertex in the second coordinate system to obtain the homography.
[0063] Here, in at least one example of step 1042, based on the positions of the vertices of rectangle 201 in image 200 captured by the camera in image 202, the positional relationship between at least one vertex of rectangle 201 and the camera and the relative positional relationships of the vertices in world coordinates may be calculated. Then, based on the positional relationship between at least one vertex of rectangle 201 and the camera and the relative positional relationships of the vertices in world coordinates, the positional relationship between the vertex and the camera and the relative positional relationships of the vertices in layout diagram 202 in world coordinates may be calculated. Then, based on the relationship between the number of pixel points and the actual length in layout diagram 202, the coordinates of each vertex of the rectangle in the world coordinate system in layout diagram 202 may be converted into coordinates in the pixel coordinate system of the layout diagram, and the coordinates [p1’, p2’, p3’, p4’] of each vertex of the rectangle in the second coordinate system may be obtained.
[0064] Here, the relationship between the number of pixel points and the actual length may be represented, for example, as the number of points per unit length (e.g., DPI), that is, the number of pixel points in layout diagram 202 corresponds to the actual unit length per unit length.
[0065] Also, the position of the camera in layout diagram 202 (e.g., the coordinates of the camera in the second coordinate system) may be a known quantity.
[0066] Regarding the specific method of calculating the homography matrix based on coordinates [p1, p2, p3, p4] and coordinates [p1’, p2’, p3’, p4’] in step 1043, related technologies may be referred to.
[0067] In step 105, when displaying the image content of rectangle 201 in image 200 on layout diagram 202, there are the following scenarios.
[0068] Scenario 1: When the direction of the image 200 and the direction of the layout diagram 202 match (for example, the width direction of the image 200 and the width direction of the layout diagram 202 correspond to the same direction in the world coordinate system), the image content in the rectangle 201 may be directly displayed in the corresponding rectangle of the layout diagram 202.
[0069] Scenario 2: When the direction of the image 200 and the direction of the layout diagram 202 do not match (for example, the width direction of the image 200 and the width direction of the layout diagram 202 correspond to different directions in the world coordinate system, and for example, they are perpendicular to each other), the image content in the rectangle 201 may be rotated by a certain angle (for example, rotated by 90 degrees or 270 degrees), and the rotated image content may be displayed in the corresponding rectangle of the layout diagram 202.
[0070] In some embodiments of step 105, the above Scenario 1 or Scenario 2 may be determined by a hyperparameter. For example, the difference in coordinates in the Y direction of the second axis between the point closest to the camera (for example, the point corresponding to point 2043 in FIG. 4) and the point where the camera is located in the rectangle (for example, the second rectangle 2022) in the layout diagram 202 corresponding to the rectangle 201 in the image 200, determined in step 1036, is calculated. The difference in coordinates is compared with the hyperparameter. If the difference in coordinates is smaller than the hyperparameter, it may be determined as Scenario 1. If the difference in coordinates is larger than the hyperparameter, it may be determined as Scenario 2. Then, corresponding processing may be executed according to the determination result of Scenario 1 or Scenario 2.
[0071] The above merely describes steps or processes related to the present invention, and the present invention is not limited thereto. The motion detection method may further include other steps or processes, and for the specific content of these steps or processes, reference may be made to the prior art. Also, the above merely exemplarily illustrates embodiments of the present invention by taking some structures of article recognition as examples. The present invention is not limited to these structures, and appropriate modifications may be made to these structures, and these modifications should be included within the scope of the embodiments of the present invention.
[0072] Each of the above embodiments merely exemplarily illustrates embodiments of the present invention, and the present invention is not limited thereto, and appropriate modifications may be made based on the various above embodiments. For example, each of the above embodiments may be used alone, or one or more of the above embodiments may be used in combination.
[0073] According to an embodiment of the present invention, by determining a line segment based on the vanishing point of an image, setting a rectangle in the image, and projecting each vertex of the rectangle to a corresponding point in the layout diagram, a homography is calculated, thereby automatically setting a rectangle and corresponding its vertices to the layout diagram, and avoiding manual point labeling and searching for corresponding points in the layout diagram, so that the accuracy of homography calculation can be improved and the workload can be reduced.
[0074] <Example 2> The embodiment of the present invention provides a homography acquisition device corresponding to the homography acquisition method according to Embodiment 1. The description of the same content as that in Embodiment 1 of Example 2 is omitted.
[0075] FIG. 11 is a schematic diagram of an example of a homography acquisition device. As shown in FIG. 11, the device includes the following parts.
[0076] The splitting unit 1101 splits an image captured by a camera.
[0077] The extraction unit 1102 determines the vanishing points in the image, obtains line segments corresponding to different vanishing points, retains the line segments distributed on a predetermined object based on the segmentation result of the image, and extracts a rectangle in the image based on the line segments distributed on the predetermined object.
[0078] The projection unit 1103 projects the vertices of the rectangle onto the corresponding points in the layout diagram.
[0079] The calculation unit 1104 calculates the homography based on the coordinates of the vertices of the rectangle and the coordinates of the corresponding points in the layout diagram.
[0080] Here, when the projection unit 1103 projects the vertices of the rectangle onto the corresponding points in the layout diagram, the extending direction of the side of the rectangle closest to the camera is set as the direction of the first axis, based on the point closest to the camera on the side of the rectangle closest to the camera, when the rectangle is located on one side of the camera in the direction of the second axis perpendicular to the first axis, the first shape of the rectangle is determined, and when the rectangle is located on the other side of the camera, the second shape of the rectangle is determined, based on the first shape, the first rectangle in the layout diagram is determined, and based on the second shape, the second rectangle in the layout diagram is determined, based on the point closest to the camera on the side of the rectangle closest to the camera, when the rectangle is located on one side of the camera in the direction of the first axis, the third shape of the rectangle is determined, and when the rectangle is located on the other side of the camera, the fourth shape of the rectangle is determined, based on the third shape, the third rectangle in the layout diagram is determined, and based on the fourth shape, the fourth rectangle in the layout diagram is determined, based on the positions of the first rectangle, the second rectangle, the third rectangle, and the fourth rectangle in the layout diagram, according to a predetermined rule, the rectangle in the layout diagram corresponding to the rectangle in the image is determined.
[0081] Here, the predetermined rule is In the layout diagram, the area of the region corresponding to the predetermined object occupied by the rectangle is the largest, and / or it is included that the rectangle does not exceed the range of the layout diagram.
[0082] In at least some embodiments, when calculating the homography based on the coordinates of the vertices of the rectangle and the coordinates of the corresponding points in the layout diagram, the calculation unit 1104 uses the pixel coordinate system of the vertices of the rectangle in the image as the first coordinate system, and the pixel coordinates of each vertex as the coordinates in the first coordinate system. It converts the world coordinate system of the corresponding points in the layout diagram into the pixel coordinate system of the layout diagram to obtain the second coordinate system, and calculates the coordinates of each vertex in the second coordinate system. Based on the coordinates of each vertex in the first coordinate system and the coordinates of each vertex in the second coordinate system, the homography is obtained.
[0083] In at least some embodiments, as shown in FIG. 11, the homography acquisition device 1100 further includes the following components.
[0084] The display control unit 1105 displays the image content in the rectangle in the image at the position corresponding to the rectangle in the layout diagram.
[0085] Here, when the directions of the image and the layout diagram do not match, the display control unit 1105 rotates the direction of the image content in the rectangle in the image, and displays the rotated image content at the position corresponding to the rectangle in the layout diagram.
[0086] Note that the above merely describes each component or module related to the present invention, and the present invention is not limited thereto. The homography acquisition device 1100 may include other components or modules, and for the specific contents of these components or modules, reference may be made to related technologies.
[0087] For simplicity, FIG. 11 merely illustratively shows the connection relationships or signal directions between the respective components or modules, and it is obvious to those skilled in the art that various related technologies such as bus connections can be used. The above-mentioned various components or modules may be implemented by hardware devices such as a processor and a memory, and the embodiments of the present invention are not limited thereto.
[0088] The above embodiments merely illustratively explain the embodiments of the present invention, and the present invention is not limited thereto. Appropriate modifications may be made based on the above various embodiments. For example, each of the above embodiments may be used alone, or one or more of the above embodiments may be used in combination.
[0089] <Example 3> Embodiments of the present invention provide an electronic device including the homography acquisition device 1100 described in Embodiment 2, and the content thereof is incorporated herein by reference. The electronic device may be, for example, a computer, a server, a workstation, a laptop computer, a smartphone, etc., but the embodiments of the present invention are not limited thereto.
[0090] FIG. 12 is a schematic diagram of an electronic device according to an embodiment of the present invention. As shown in FIG. 12, the electronic device 1200 according to an embodiment of the present invention includes a processor (for example, a central processing unit (CPU)) 1210 and a memory 1220. The memory 1220 is connected to the processor 1210. The memory 1220 may store various data, and may further store a program 1221 for information processing. The program 1221 is executed under the control of the processor 1210.
[0091] In some aspects, the function of the homography acquisition device 1100 may be integrated into the processor 1210. Here, the processor 1210 may be configured to implement the homography acquisition method described in Embodiment 1.
[0092] In some aspects, the homography acquisition device 1100 may be respectively configured with a processor 1210. For example, the homography acquisition device is a chip connected to the processor 1210 and may be configured to implement the functions of the homography acquisition device under the control of the processor 1210.
[0093] For example, the processor 1210 may be configured to execute steps of dividing an image captured by a camera, determining a vanishing point in the image, obtaining line segments corresponding to different vanishing points, retaining line segments distributed on a predetermined object based on the division result of the image, extracting a rectangle in the image based on the line segments distributed on the predetermined object, projecting vertices of the rectangle to corresponding points in a layout diagram, and calculating a homography based on the coordinates of the vertices of the rectangle and the coordinates of the corresponding points in the layout diagram.
[0094] In at least some embodiments, the step of projecting the vertices of the rectangle to corresponding points in the layout diagram includes a step of setting the extending direction of the side of the rectangle closest to the camera as the direction of a first axis, determining a first shape of the rectangle when the rectangle is located on one side of the camera in a direction of a second axis perpendicular to the first axis based on the point on the side of the rectangle closest to the camera that is closest to the camera, and determining a second shape of the rectangle when the rectangle is located on the other side of the camera, determining a first rectangle in the layout diagram based on the first shape, and determining a second rectangle in the layout diagram based on the second shape, determining a third shape of the rectangle when the rectangle is located on one side of the camera in the direction of the first axis based on the point on the side of the rectangle closest to the camera that is closest to the camera, and determining a fourth shape of the rectangle when the rectangle is located on the other side of the camera, Determining a third rectangle in the layout diagram based on the third shape, and determining a fourth rectangle in the layout diagram based on the fourth shape; Determining, according to a predetermined rule, a rectangle in the layout diagram corresponding to the rectangle in the image based on the positions of the first rectangle, the second rectangle, the third rectangle, and the fourth rectangle in the layout diagram.
[0095] In at least some embodiments, the predetermined rule includes that in the layout diagram, the area of the region corresponding to the predetermined object occupied by the rectangle is the largest, and / or the rectangle does not exceed the range of the layout diagram.
[0096] In at least some embodiments, the step of calculating the homography based on the coordinates of the vertices of the rectangle and the coordinates of the corresponding points in the layout diagram includes taking the pixel coordinate system of the vertices of the rectangle in the image as the first coordinate system, and taking the pixel coordinates of each vertex as the coordinates in the first coordinate system; converting the world coordinate system of the corresponding points in the layout diagram into the pixel coordinate system of the layout diagram to obtain a second coordinate system, and calculating the coordinates of each vertex in the second coordinate system; obtaining the homography based on the coordinates of each vertex in the first coordinate system and the coordinates of each vertex in the second coordinate system.
[0097] In at least some embodiments, the processor 1210 is further configured to execute the step of displaying the image content in the rectangle in the image at the position corresponding to the rectangle in the layout diagram.
[0098] Here, when the directions of the image and the layout diagram do not match, the direction of the image content in the rectangle in the image is rotated, and the rotated image content is displayed at the position corresponding to the rectangle in the layout diagram.
[0099] Also, as shown in FIG. 12, the electronic device 1200 may further include an input / output (I / O) device 1230, a display 1240, etc. Here, the functions of these components are the same as those in the prior art, and the description thereof is omitted here. Note that the electronic device 1200 does not necessarily include all the components shown in FIG. 12. Also, the electronic device 1200 may include components not shown in FIG. 12, and the prior art may be referred to.
[0100] An embodiment of the present invention provides a computer-readable program for causing a computer to execute the homography acquisition method described in Example 1 in the electronic device when executing a program in the electronic device.
[0101] An embodiment of the present invention further provides a storage medium in which a computer-readable program for causing a computer to execute the homography acquisition method described in Example 1 in the electronic device is stored.
[0102] The above apparatus and method of the present invention may be implemented by hardware, or may be implemented by combining hardware and software. The present invention relates to a computer-readable program, and when the program is executed by a logic unit, the logic unit can implement the above-described apparatus or component requirements, or can implement the above-described various methods or steps. The present invention relates to a storage medium for storing the above program, such as a hard disk, a magnetic disk, an optical disk, a DVD, a flash memory, etc.
[0103] The method / apparatus described with reference to embodiments of the present invention may be implemented in hardware, software modules executed by a processor, or a combination of both. For example, one or more of the functional block diagrams shown in the drawings, or one or more combinations of the functional block diagrams, may correspond to each software module of the flow of the computer program, or may correspond to each hardware module. These software modules may each correspond to each step shown in the drawings. These hardware modules may be realized, for example, by hardware-implementing these software modules using a field programmable gate array (FPGA).
[0104] The software module may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium known to those skilled in the art. The storage medium may be connected to the processor such that the processor reads information from, or writes information to, the storage medium, or the storage medium may be a component of the processor. The processor and the storage medium are located in an ASIC. The software module may be stored in the memory of the mobile terminal or may be stored in a memory card inserted into the mobile terminal. For example, when the device (e.g., the mobile terminal) uses a relatively large-capacity MEGA-SIM card or a large-capacity flash memory device, the software module may be stored in the MEGA-SIM card or the large-capacity flash memory device.
[0105] One or more functional blocks and / or one or more combinations of functional blocks described in the drawings may be implemented by a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any suitable combination thereof for performing the functions described in the present invention. One or more functional blocks and / or one or more combinations of functional blocks described in the drawings may be implemented, for example, in a combination of computing devices, such as a combination of a DSP and a microprocessor, a combination of multiple microprocessors, one or more microprocessors in combination with DSP communication, or any other configuration.
[0106] The present invention has been described above with reference to specific embodiments, but the above description is merely illustrative and does not limit the scope of protection of the present invention. Without departing from the spirit and principle of the present invention, various modifications and changes may be made to the present invention, and these modifications and changes also fall within the scope of the present invention.
[0107] In addition, the following supplementary notes are disclosed regarding the embodiments including the above-described examples. (Supplementary Note 1) A method for obtaining homography, comprising: dividing an image captured by a camera; determining a vanishing point in the image, obtaining line segments corresponding to different vanishing points, retaining line segments distributed on a predetermined object based on the division result of the image, and extracting a rectangle in the image based on the line segments distributed on the predetermined object; projecting the vertices of the rectangle onto corresponding points in a layout diagram; calculating homography based on the coordinates of the vertices of the rectangle and the coordinates of the corresponding points in the layout diagram. (Supplementary Note 2) The step of projecting the vertices of the rectangle onto corresponding points in the layout diagram is The step of setting the extending direction of the side of the rectangle closest to the camera as the direction of the first axis, and Based on the point closest to the camera on the side of the rectangle closest to the camera, determining the first shape of the rectangle when the rectangle is located on one side of the camera in the direction of the second axis perpendicular to the first axis, and determining the second shape of the rectangle when the rectangle is located on the other side of the camera; Based on the first shape, determining a first rectangle in the layout diagram, and based on the second shape, determining a second rectangle in the layout diagram; Based on the point closest to the camera on the side of the rectangle closest to the camera, determining the third shape of the rectangle when the rectangle is located on one side of the camera in the direction of the first axis, and determining the fourth shape of the rectangle when the rectangle is located on the other side of the camera; Based on the third shape, determining a third rectangle in the layout diagram, and based on the fourth shape, determining a fourth rectangle in the layout diagram; Based on the positions of the first rectangle, the second rectangle, the third rectangle, and the fourth rectangle in the layout diagram, determining a rectangle in the layout diagram corresponding to the rectangle in the image according to a predetermined rule, the method according to Appendix 1. (Appendix 3) The predetermined rule is In the layout diagram, the area of the region corresponding to the predetermined object occupied by the rectangle is the largest, and / or The rectangle does not exceed the range of the layout diagram, the method according to Appendix 2. (Appendix 4) The step of calculating the homography based on the coordinates of the vertices of the rectangle and the coordinates of the corresponding points in the layout diagram is Taking the pixel coordinate system of the vertices of the rectangle in the image as the first coordinate system, and using the pixel coordinates of each vertex as the coordinates in the first coordinate system; Converting the world coordinate system of the corresponding points in the layout diagram into the pixel coordinate system of the layout diagram as the second coordinate system, and calculating the coordinates of each vertex in the second coordinate system; Obtaining the homography based on the coordinates of each vertex in the first coordinate system and the coordinates of each vertex in the second coordinate system, the method according to Appendix 1. (Appendix 5) Further including the step of displaying the image content in the rectangle in the image at the position corresponding to the rectangle in the layout diagram, the method according to Appendix 1. (Appendix 6) When the direction of the image does not match the direction of the layout diagram, rotating the direction of the image content in the rectangle in the image, and displaying the rotated image content at the position corresponding to the rectangle in the layout diagram, the method according to Appendix 5. (Appendix 7) A storage medium storing a computer-readable program, wherein the computer-readable program causes a processor connected to the storage medium to Divide the image captured by the camera; Determine the vanishing points in the image, obtain the line segments corresponding to different vanishing points, retain the line segments distributed on a predetermined object based on the segmentation result of the image, and extract a rectangle in the image based on the line segments distributed on the predetermined object; Project the vertices of the rectangle onto the corresponding points in the layout diagram; Calculate the homography based on the coordinates of the vertices of the rectangle and the coordinates of the corresponding points in the layout diagram, the storage medium for executing the method. (Appendix 8) The step of projecting the vertices of the rectangle onto the corresponding points in the layout diagram includes Taking the extending direction of the side of the rectangle closest to the camera as the direction of the first axis; Based on the point on the side of the rectangle closest to the camera that is closest to the camera, determine the first shape of the rectangle when the rectangle is located on one side of the camera in the direction of the second axis perpendicular to the first axis, and determine the second shape of the rectangle when the rectangle is located on the other side of the camera; Based on the first shape, determine the first rectangle in the layout diagram, and based on the second shape, determine the second rectangle in the layout diagram; Based on the point on the side of the rectangle closest to the camera that is closest to the camera, determine the third shape of the rectangle when the rectangle is located on one side of the camera in the direction of the first axis, and determine the fourth shape of the rectangle when the rectangle is located on the other side of the camera; Based on the third shape, determine the third rectangle in the layout diagram, and based on the fourth shape, determine the fourth rectangle in the layout diagram; Based on the positions of the first rectangle, the second rectangle, the third rectangle, and the fourth rectangle in the layout diagram, determine the rectangle in the layout diagram corresponding to the rectangle in the image according to a predetermined rule. The storage medium according to Appendix 7 includes the above steps. (Appendix 9) The predetermined rule is In the layout diagram, the area of the region corresponding to the predetermined object occupied by the rectangle is the largest, and / or The rectangle does not exceed the range of the layout diagram. The storage medium according to Appendix 8 includes the above. (Appendix 10) Based on the coordinates of the vertices of the rectangle and the coordinates of the corresponding points in the layout diagram, the step of calculating the homography is Taking the pixel coordinate system of the vertices of the rectangle in the image as the first coordinate system, and taking the pixel coordinates of each vertex as the coordinates in the first coordinate system; Converting the world coordinate system of corresponding points in the layout diagram into the pixel coordinate system of the layout diagram as a second coordinate system, and calculating the coordinates of each vertex in the second coordinate system; Obtaining the homography based on the coordinates of each vertex in the first coordinate system and the coordinates of each vertex in the second coordinate system, the storage medium according to Appendix 8. (Appendix 11) The method further includes the step of displaying the image content in the rectangle in the image at a position corresponding to the rectangle in the layout diagram, the storage medium according to Appendix 7. (Appendix 12) When the direction of the image does not match the direction of the layout diagram, rotating the direction of the image content in the rectangle in the image, and displaying the rotated image content at a position corresponding to the rectangle in the layout diagram, the storage medium according to Appendix 11.
Claims
1. 1. A homography acquisition device, comprising: A division unit that divides an image captured by the camera; an extraction unit that determines a vanishing point in the image, obtains line segments corresponding to different vanishing points, reserves line segments distributed on a predetermined object based on a result of dividing the image, and extracts a rectangle in the image based on the line segments distributed on the predetermined object; a projection unit for projecting vertices of the rectangle onto corresponding points on a layout diagram; a calculation unit that calculates a homography based on coordinates of vertices of the rectangle and coordinates of corresponding points in the layout drawing.
2. Projecting the vertices of the rectangle onto corresponding points in a layout diagram includes: A direction in which a side of the rectangle that is closest to the camera extends is set as a direction of a first axis; determining a first shape of the rectangle when the rectangle is located on one side of the camera in a direction of a second axis perpendicular to the first axis based on a point of the rectangle on a side of the rectangle that is closest to the camera, and determining a second shape of the rectangle when the rectangle is located on the other side of the camera; determining a first rectangle in the layout diagram based on the first shape, and determining a second rectangle in the layout diagram based on the second shape; determining a third shape of the rectangle when the rectangle is located on one side of the camera in a direction of the first axis based on a point of the rectangle on a side of the rectangle that is closest to the camera, and determining a fourth shape of the rectangle when the rectangle is located on the other side of the camera; determining a third rectangle in the layout diagram based on the third shape, and determining a fourth rectangle in the layout diagram based on the fourth shape; and determining a rectangle in the layout drawing that corresponds to the rectangle in the image according to a predetermined rule based on positions of the first rectangle, the second rectangle, the third rectangle, and the fourth rectangle in the layout drawing.
3. The predetermined rule is: In the layout diagram, the area of the region corresponding to the predetermined object occupied by the rectangle is the largest, and / or The apparatus of claim 2 , further comprising: the rectangle not exceeding an extent of the layout drawing.
4. Calculating a homography based on coordinates of vertices of the rectangle and coordinates of corresponding points in the layout drawing, A pixel coordinate system of vertices of the rectangle in the image is defined as a first coordinate system, and pixel coordinates of each vertex are defined as coordinates in the first coordinate system; transforming a world coordinate system of the corresponding points in the layout diagram into a pixel coordinate system of the layout diagram to obtain a second coordinate system, and calculating coordinates of each vertex in the second coordinate system; and obtaining the homography based on coordinates of each vertex in the first coordinate system and coordinates of each vertex in the second coordinate system.
5. The apparatus of claim 1 , further comprising a display control unit that displays image content in the rectangle in the image at a position corresponding to the rectangle in the layout diagram.
6. The device of claim 5, further comprising: if the orientation of the image does not match the orientation of the layout diagram, rotating an orientation of image content in the rectangle in the image, and displaying the rotated image content at a position corresponding to the rectangle in the layout diagram.
7. An electronic device comprising the homography acquisition device according to any one of claims 1 to 6.
8. A method for obtaining a homography, comprising the steps of: Segmenting an image captured by a camera; determining a vanishing point in the image, obtaining line segments corresponding to different vanishing points, reserving line segments distributed on a predetermined object according to the segmentation result of the image, and extracting a rectangle in the image according to the line segments distributed on the predetermined object; projecting the vertices of the rectangle onto corresponding points in a layout diagram; and calculating a homography based on the coordinates of the vertices of the rectangle and the coordinates of corresponding points in the layout drawing.
9. The step of projecting vertices of the rectangle onto corresponding points in a layout diagram includes: A step of setting an extension direction of a side of the rectangle closest to the camera as a direction of a first axis; determining a first shape of the rectangle when it is located on one side of the camera in a direction of a second axis perpendicular to the first axis based on a point of the rectangle on a side of the rectangle that is closest to the camera, and determining a second shape of the rectangle when it is located on the other side of the camera; determining a first rectangle in the layout diagram based on the first shape, and determining a second rectangle in the layout diagram based on the second shape; determining a third shape of the rectangle when the rectangle is located on one side of the camera in a direction of the first axis based on a point of the rectangle on a side of the rectangle that is closest to the camera, and determining a fourth shape of the rectangle when the rectangle is located on the other side of the camera; determining a third rectangle in the layout diagram based on the third shape, and determining a fourth rectangle in the layout diagram based on the fourth shape; and determining a rectangle in the layout drawing corresponding to the rectangle in the image according to a predetermined rule based on positions of the first rectangle, the second rectangle, the third rectangle and the fourth rectangle in the layout drawing.
10. The predetermined rule is: In the layout diagram, the area of the region corresponding to the predetermined object occupied by the rectangle is the largest, and / or The method of claim 9 , further comprising the rectangle not exceeding an extent of the layout view.