Camera pose determination method and apparatus, and storage medium

Abstract

The application provides a camera pose determination method and device and a storage medium, relates to the technical field of visual detection, and can solve the problem of low quality detection efficiency or high cost caused by too many camera points or too many cameras in the current quality detection method. The method comprises the following steps: determining a plurality of first projection points of each surface of a plurality of surfaces of a to-be-detected device on a corresponding projection plane; for each surface of the plurality of surfaces, determining a second projection point of a camera on the projection plane according to the plurality of first projection points of the surface on the corresponding projection plane; calculating coordinate data of the second projection point in a three-dimensional coordinate system according to coordinate data of the second projection point in a two-dimensional coordinate system; determining a camera pose corresponding to the surface according to the coordinate data of the second projection point in the three-dimensional coordinate system; and the shooting direction of the camera is towards the surface and parallel to the normal vector of the surface. The application can effectively reduce the points of the camera, thereby improving the quality detection efficiency and reducing the cost.

Description

Technical Field

[0001] This application relates to the field of visual inspection technology, and in particular to a camera pose determination method, device and storage medium. Background Technology

[0002] When performing assembly quality inspection on products with complex surface structures such as engines, large castings and forgings, refrigerators, and washing machines using visual inspection, it is necessary to install cameras to take pictures of the areas on the product that need to be inspected for quality testing.

[0003] Currently, one method for quality inspection of products such as engines is to manually select multiple camera locations, place the cameras on a movable carrier, move the cameras to the designated camera locations, and then take photos of the parts of the engine that need to be inspected; or to fix multiple cameras in place so that they cover all the parts of the product to be inspected.

[0004] However, when there are too many locations requiring quality inspection, the two quality inspection methods mentioned above will result in too many camera points to cover all the locations, leading to low quality inspection efficiency, and too many cameras, leading to increased costs. Summary of the Invention

[0005] This application provides a camera pose determination method, apparatus, and storage medium, which solves the problems of low quality inspection efficiency and high cost caused by too many camera points or too many cameras in the current quality inspection methods. It can improve the accuracy of camera pose determination, thereby effectively reducing the number of camera points, improving quality inspection efficiency, and reducing costs.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] In a first aspect, this application provides a camera pose determination method, the method comprising: determining multiple first projection points of each of multiple surfaces of a device to be inspected on a corresponding projection plane; the plane containing the surface of the device to be inspected is parallel to the projection plane corresponding to the surface; for each of the multiple surfaces, determining a second projection point of the camera on the projection plane based on the multiple first projection points of the surface on the corresponding projection plane; calculating the coordinate data of the second projection point in a three-dimensional coordinate system based on the coordinate data of the second projection point in a two-dimensional coordinate system, the coordinate data of the second projection point in the three-dimensional coordinate system including the coordinate data of the second projection point in the two-dimensional coordinate system; determining the camera pose corresponding to the surface based on the coordinate data of the second projection point in the three-dimensional coordinate system; the shooting direction of the camera is towards the surface and parallel to the normal vector of the surface.

[0008] In conjunction with the first aspect described above, in one possible implementation, the method further includes: repeatedly performing the first operation on multiple first projection points until the radius of the target range of each cluster is less than a preset threshold, and determining the center point of each subset as a second projection point; the first projection points within each subset are all located within the target range; the first operation includes: dividing the multiple first projection points into M sets of first projection points; M is a positive integer; for each of the M sets of first projection points, determining the radius of the target range of each set; determining whether there is a radius greater than or equal to a preset threshold; if so, re-dividing the multiple first projection points to obtain N sets of first projection points; N is a positive integer; N is not equal to M.

[0009] In conjunction with the first aspect mentioned above, in one possible implementation, the method further includes: constructing an objective function based on a preset plane equation, the coordinates of the first projection points, and a preset shooting distance corresponding to the set of first projection points; the preset shooting distance is determined based on the focal length of the camera; solving the objective function based on each first projection point in each set of first projection points to obtain the parameter values ​​in the preset plane equation; and determining the coordinate data of the second projection points in the three-dimensional coordinate system based on the coordinate data of each second projection point in the two-dimensional coordinate system and the parameter values ​​in the preset plane equation.

[0010] In conjunction with the first aspect mentioned above, in one possible implementation, the method further includes: determining a first function based on a preset plane equation, the coordinates of the first projection point, and a preset shooting distance; determining a second function based on the angle between the shooting direction corresponding to each first projection point and the preset shooting direction corresponding to the set of first projection points; and determining a target function based on the first function and the second function.

[0011] In conjunction with the first aspect mentioned above, in one possible implementation, the method further includes: a first function E pi This can be achieved using the following formula:

[0012]

[0013] Where, m i x represents the number of first projection points in the first set of projection points; j y j z j Let A be the coordinates of the first projection point; i B i C i D i is the parameter of the preset plane equation; D is the preset shooting distance.

[0014] In conjunction with the first aspect mentioned above, in one possible implementation, the method further includes: a second function E αiThis can be achieved using the following formula:

[0015]

[0016] Among them, u i v is the normal vector of the surface; j This is the preset shooting direction.

[0017] In conjunction with the first aspect mentioned above, in one possible implementation, the method further includes: the objective function E i This can be achieved using the following formula:

[0018] E i =γE pi +(1-γ)E αi +∈

[0019] Where γ is the weighting parameter and ∈ is a constant.

[0020] Secondly, this application provides a camera pose determination device, which includes: a communication unit and a processing unit; the communication unit is used to determine multiple first projection points of each of the multiple surfaces of the device to be tested on a corresponding projection plane; the plane on which the surface of the device to be tested is located is parallel to the projection plane corresponding to the surface; the processing unit is used to determine a second projection point of the camera on the projection plane for each of the multiple surfaces based on the multiple first projection points of the surface on the corresponding projection plane; the processing unit is further used to calculate the coordinate data of the second projection point in a three-dimensional coordinate system based on the coordinate data of the second projection point in a two-dimensional coordinate system, the coordinate data of the second projection point in the three-dimensional coordinate system including the coordinate data of the second projection point in the two-dimensional coordinate system; the processing unit is further used to determine the camera pose corresponding to the surface based on the coordinate data of the second projection point in the three-dimensional coordinate system; the shooting direction of the camera is towards the surface and parallel to the normal vector of the surface.

[0021] In conjunction with the second aspect above, in one possible implementation, the processing unit is specifically used for: repeatedly executing the first operation for multiple first projection points until the radius of the target range of each cluster is less than a preset threshold, and determining the center point of each subset as multiple second projection points; the first projection points in each subset are all located within the target range; the first operation includes: dividing the multiple first projection points into M sets of first projection points; M is a positive integer; for each set in the M sets of first projection points, determining the radius of the target range of each set; determining whether there is a radius greater than or equal to a preset threshold; if so, re-dividing the multiple first projection points to obtain N sets of first projection points; N is a positive integer; N is not equal to M.

[0022] In conjunction with the second aspect above, in one possible implementation, the processing unit is specifically used to: construct an objective function based on a preset plane equation, the coordinates of the first projection point, and a preset shooting distance corresponding to the set of first projection points; solve the objective function based on each first projection point in each set of first projection points to obtain the parameter values ​​in the preset plane equation; and determine the coordinate data of the second projection point in the three-dimensional coordinate system based on the coordinate data of each second projection point in the two-dimensional coordinate system and the parameter values ​​in the preset plane equation.

[0023] In conjunction with the second aspect above, in one possible implementation, the processing unit is specifically used to: determine a first function based on a preset plane equation, the coordinates of the first projection point, and the distance to a preset shooting position; determine a second function based on the angle between the shooting direction corresponding to each first projection point and the preset shooting direction corresponding to the set of first projection points; and determine a target function based on the first function and the second function.

[0024] In conjunction with the second aspect mentioned above, in one possible implementation, the first function E pi This can be achieved using the following formula:

[0025]

[0026] Where, m i x represents the number of first projection points in the first set of projection points; j y j z j Let A be the coordinates of the first projection point; i B i C i D i is the parameter of the preset plane equation; D is the preset shooting distance.

[0027] In conjunction with the second aspect mentioned above, in one possible implementation, the second function E αi This can be achieved using the following formula:

[0028]

[0029] Among them, u i v is the normal vector of the surface; j This is the preset shooting direction.

[0030] In conjunction with the second aspect mentioned above, in one possible implementation, the objective function E i This can be achieved using the following formula:

[0031] E i =γE pi +(1-γ)E αi +∈

[0032] Where γ is the weighting parameter and ∈ is a constant.

[0033] Thirdly, this application provides a camera pose determination apparatus, which includes: a processor and a communication interface; the communication interface and the processor are coupled, and the processor is used to run computer programs or instructions to implement the camera pose determination method as described in the first aspect and any possible implementation of the first aspect.

[0034] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed on a terminal, cause the terminal to perform the camera pose determination method as described in the first aspect and any possible implementation thereof.

[0035] Fifthly, this application provides a computer program product containing instructions that, when run on a camera pose determination device, cause the camera pose determination device to perform the camera pose determination method as described in the first aspect and any possible implementation thereof.

[0036] In a sixth aspect, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run computer programs or instructions to implement the camera pose determination method as described in the first aspect and any possible implementation thereof.

[0037] Specifically, the chip provided in this application also includes a memory for storing computer programs or instructions.

[0038] It should be noted that the aforementioned computer instructions may be stored, in whole or in part, on a computer-readable storage medium. This computer-readable storage medium may be packaged together with the processor of the device, or it may be packaged separately from the processor of the device; this application does not impose any limitation on this.

[0039] In a seventh aspect, this application provides a camera pose determination system, comprising: a device to be detected, a camera, and a data processing server, wherein the data processing server is configured to perform the camera pose determination method as described in the first aspect and any possible implementation thereof.

[0040] The descriptions of aspects two through seven in this application can be referenced to the detailed description of aspect one; and the beneficial effects of the descriptions of aspects two through seven can be referenced to the analysis of the beneficial effects of aspect one, which will not be repeated here.

[0041] In this application, the name of the aforementioned camera pose determination device does not limit the device or functional module itself. In actual implementation, these devices or functional modules may appear under other names. As long as the function of each device or functional module is similar to that of this application, it falls within the scope of the claims of this application and its equivalents.

[0042] These or other aspects of this application will become more readily apparent in the following description.

[0043] The above solution offers at least the following advantages: Based on the above technical solution, the camera pose determination method provided in this application determines multiple first projection points on the corresponding projection plane for each of the multiple surfaces of the device to be inspected. For each of the multiple surfaces, based on the multiple first projection points on the corresponding projection plane, a second projection point of the camera on the projection plane is determined, i.e., the position of the camera in the two-dimensional coordinate system. Then, based on the coordinate data of the second projection point in the two-dimensional coordinate system, the coordinate data of the second projection point in the three-dimensional coordinate system is calculated. Based on the coordinate data in the three-dimensional coordinate system, the camera position corresponding to the surface is determined, and the camera's shooting direction is determined to be facing the surface and parallel to the surface's normal vector. Compared to the current quality inspection methods, which suffer from low inspection efficiency and high costs due to too many camera points or too many cameras, the above technical solution first determines the camera's position in the two-dimensional coordinate system and then, based on that position, further determines the camera's position in the three-dimensional coordinate system, greatly improving the accuracy of camera pose determination. This effectively reduces the number of camera points, improves inspection efficiency, and reduces costs. Attached Figure Description

[0044] Figure 1 This application provides a schematic diagram of the architecture of a camera pose determination system.

[0045] Figure 2 A schematic diagram showing the positions of a camera and a device under test, provided in an embodiment of this application;

[0046] Figure 3 This is a schematic diagram of the hardware structure of a camera pose determination device provided in an embodiment of this application;

[0047] Figure 4 A flowchart illustrating a camera pose determination method provided in this application embodiment;

[0048] Figure 5 A flowchart illustrating another camera pose determination method provided in this application embodiment;

[0049] Figure 6 This is a schematic diagram of a camera pose determination device provided in an embodiment of this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0052] The terms "first" and "second," etc., used in the specification and drawings of this application are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of objects.

[0053] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0054] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0055] In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0056] When performing assembly quality inspection on products with complex surface structures such as engines, large castings and forgings, refrigerators, and washing machines using visual inspection, it is necessary to install cameras to take pictures of the areas on the product that need to be inspected for quality testing.

[0057] Current quality inspection methods for products such as engines typically involve manually selecting multiple camera positions and moving the cameras to these positions to photograph the areas on the engine requiring inspection. For example, the camera is placed on a movable platform and moved to the designated camera location before taking pictures for defect detection (e.g., an engine cylinder head bolt missing inspection device and method). Alternatively, multiple cameras can be fixed in place to cover all inspection areas of the product (e.g., an engine cylinder head CCD vision inspection machine).

[0058] However, when there are too many locations requiring quality inspection, both of the above-mentioned quality inspection methods require the installation of multiple cameras to cover all inspection areas. This can lead to problems such as too many camera points, low inspection efficiency, unsatisfactory shooting angles, and inaccurate camera placement. Furthermore, if the camera placement is not properly selected, the number of cameras will be excessive, increasing costs and consuming a significant amount of inspection time.

[0059] Therefore, the camera pose determination method provided in this application determines multiple first projection points on the corresponding projection plane for each of the multiple surfaces of the device to be inspected. For each surface, based on the multiple first projection points on the corresponding projection plane, a second projection point of the camera on the projection plane is determined, i.e., the position of the camera in the two-dimensional coordinate system. Then, based on the coordinate data of the second projection point in the two-dimensional coordinate system, the coordinate data of the second projection point in the three-dimensional coordinate system is calculated. Based on the coordinate data in the three-dimensional coordinate system, the camera position corresponding to the surface is determined, and the camera's shooting direction is determined to be facing the surface and parallel to the surface's normal vector. Compared with the current quality inspection methods, which suffer from low quality inspection efficiency and high costs due to too many camera points or too many cameras, the above technical solution first determines the camera's position in the two-dimensional coordinate system, and then further determines the camera's position in the three-dimensional coordinate system based on the position in the two-dimensional coordinate system. This greatly improves the accuracy of camera pose determination, thereby effectively reducing the number of camera points, improving quality inspection efficiency, and reducing costs.

[0060] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0061] Figure 1 This is a schematic diagram of the architecture of a camera pose determination system provided in an embodiment of this application. Figure 1 As shown, the camera pose determination system includes: a device to be detected 101, a camera 102, and a data processing server 103.

[0062] The device under test 101 is connected to the camera 102. The camera 102 is connected to the data processing server 103 via a communication link. This communication link can be a wired communication link or a wireless communication link, and this application does not limit it in this regard.

[0063] The following is a detailed introduction to the fixing, positioning, and usage of camera 102.

[0064] In one example, a robotic arm may be fixed to the surface of the device 101 to be inspected, and a camera 102 may be mounted at the end of the robotic arm. The robotic arm may carry the camera 102 and move around the device 101 to acquire multiple first projection points of each of the multiple surfaces of the device 101 on a corresponding projection plane.

[0065] Specifically, in response to the manual control of the robotic arm, the robotic arm will move the camera 102 to each first projection point of the device 101 to be tested, and make manual adjustments so that the position and orientation of the camera 102 when observing the first projection point are optimal. At this time, the camera 102 records the image of the position of the first projection point and the position information of the camera 102 itself.

[0066] The position information of camera 102 itself and the position of the first projection point are determined using the world coordinate system. The camera coordinate system is obtained by manually calibrating the intrinsic and extrinsic parameters of camera 102 using a calibration method (e.g., the Zhang Zhengyou calibration method) and calibrating the robotic arm using a checkerboard pattern. Then, by using the transformation relationship between the camera coordinate system and the coordinate systems of each axis of the robotic arm, the position of the device to be inspected relative to camera 102 can be determined using the camera.

[0067] Furthermore, the camera 102 sends the image of the first projection point location and the camera 102's own position information to the data processing server.

[0068] It should be noted that, as Figure 2 As shown, the imaging plane of camera 102 is parallel to the plane containing the device under test 101. Furthermore, the plane containing the surface of the device under test 101 is parallel to the projection plane corresponding to that surface.

[0069] The data processing server 103 will be described in detail below.

[0070] In one example, the data processing server 103 receives an image from the location of a first projection point of the camera 102, along with the camera 102's own location information at that moment. It then extracts the location information of the first projection point from the image. Based on the location information of multiple first projection points, it determines the position and orientation of the camera 102 when capturing images of multiple first projection points.

[0071] When implemented in hardware, the various modules in the data processing server 103 can be integrated into, for example... Figure 3 The camera pose determination device shown is implemented in hardware. Specifically, as... Figure 3 As shown, the basic hardware structure of the camera pose determination device is introduced.

[0072] Figure 3 This is a schematic diagram of the hardware structure of a camera pose determination device provided in an embodiment of this application. Figure 3 As shown, the camera pose determination device includes at least one processor 301, a communication line 302, and at least one communication interface 304, and may also include a memory 303. The processor 301, memory 303, and communication interface 304 are connected via the communication line 302.

[0073] The processor 301 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0074] Communication line 302 may include a path for transmitting information between the aforementioned components.

[0075] Communication interface 304 is used to communicate with other devices or communication networks. It can use any transceiver-like device, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0076] The memory 303 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of including or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0077] In one possible design, the memory 303 can exist independently of the processor 301, meaning the memory 303 can be an external memory of the processor 301. In this case, the memory 303 can be connected to the processor 301 via a communication line 302 to store execution instructions or application code, and its execution is controlled by the processor 301 to implement the camera pose determination method provided in the following embodiments of this application. In another possible design, the memory 303 can also be integrated with the processor 301, meaning the memory 303 can be an internal memory of the processor 301. For example, the memory 303 can be a cache, used to temporarily store some data and instruction information.

[0078] As one possible implementation, processor 301 may include one or more CPUs, for example Figure 3 CPU0 and CPU1 in the example. As another possible implementation, the camera pose determination device may include multiple processors, such as... Figure 3 The processors 301 and 307 are included. As another possible implementation, the camera pose determination device may also include an output device 305 and an input device 306.

[0079] It should be noted that the various embodiments of this application can be referenced or learned from each other. For example, the same or similar steps, method embodiments, system embodiments and device embodiments can be referenced from each other without limitation.

[0080] Figure 4 A flowchart illustrating a camera pose determination method provided in this application embodiment is available. This method can be applied to, for example... Figure 3 In the camera pose determination device shown. Figure 4As shown, the method includes the following S401-S404.

[0081] S401. Determine multiple first projection points on the corresponding projection plane for each of the multiple surfaces of the device to be tested.

[0082] In this context, the plane containing the surface of the device to be tested is parallel to the projection plane corresponding to the surface.

[0083] In one example, such as Figure 2 As shown, consider a surface of the device to be inspected as an example. This surface has multiple inspection points. The three-dimensional coordinates of these multiple inspection points on the surface can be represented as a set P = {p1, p2, p3, ...}. Then, the multiple inspection points in set P are projected onto a projection plane to obtain the two-dimensional coordinates of the first projection points corresponding to the multiple inspection points. The two-dimensional coordinates of the multiple first projection points on the projection plane can be represented as a set P′ = {P′1, P′2, ..., P′}. n}

[0084] Furthermore, the optimal observation direction for each of the multiple detection points can be determined. The optimal observation directions for multiple first projection points can be represented as a set.

[0085] S402. For each of the multiple surfaces, determine the second projection point of the camera on the projection plane based on the multiple first projection points of the surface on the corresponding projection plane.

[0086] In one possible implementation, the first operation is repeatedly performed on multiple first projection points until the radius of the target range of each cluster is less than a preset threshold, at which point the center point of each subset is determined as the second projection point.

[0087] In this case, the first projection point within each subset is located within the target range.

[0088] In one example, the target range is circular. When the radius of the smallest circle containing each cluster is less than a preset threshold, the center of that smallest circle is taken as the center point of the first projection point in that cluster.

[0089] Specifically, the first operation includes the following steps 1-4.

[0090] Step 1: Divide the multiple first projection points into M sets of first projection points.

[0091] Where M is a positive integer.

[0092] In one example, for set P ′ Each two-dimensional coordinate in the dataset is used to form a clustering set P using a clustering algorithm and a preset number of clusters M. ′Each two-dimensional coordinate in the graph is divided into a set of M first projection points.

[0093] The clustering algorithm can be K-MEANS clustering algorithm, mean shift clustering algorithm, density-based clustering algorithm, density-based clustering algorithm, etc., and this application does not limit it.

[0094] Step 2: For each of the M sets of first projection points, determine the radius of the target range for each set.

[0095] In one example, let's assume the target area is circular. For each of the M sets of first projection points, determine a minimum circle containing all the first projection points within that set, and use the radius of this minimum circle as the radius of the target area for that set.

[0096] Step 3: Determine if there is a radius greater than or equal to the preset threshold.

[0097] In one example, consider M sets of first projection points as 3 sets of first projection points, with corresponding radii of 10 cm, 20 cm, and 15 cm. Determine whether any of the 10 cm, 20 cm, and 15 cm sets contains a radius greater than or equal to a preset threshold.

[0098] It should be noted that the preset threshold can be determined based on the actual situation, and this application does not impose any restrictions on it.

[0099] Step 4: If they exist, then re-divide the multiple first projection points to obtain N sets of first projection points.

[0100] Where N is a positive integer; N is not equal to M.

[0101] In one example, consider M sets of first projection points as 3 sets of first projection points, with corresponding radii of 10 cm, 20 cm, and 15 cm, and a preset threshold of 18 cm. If it is determined that there exists a radius larger than the preset threshold, the preset number of clusters M is changed to N, and the clustering algorithm is re-applied using the preset number of clusters N to re-cluster set P. ′ Each two-dimensional coordinate in the graph is divided into N sets of first projection points.

[0102] Furthermore, for the radii corresponding to the N sets of first projection points, it is determined again whether there are radii greater than or equal to a preset threshold. If they still exist, the preset number of categories is changed again and the clustering is re-organized until there are no radii greater than or equal to the preset threshold.

[0103] S403. Based on the coordinate data of the second projection point in the two-dimensional coordinate system, calculate the coordinate data of the second projection point in the three-dimensional coordinate system.

[0104] The coordinate data of the second projection point in the three-dimensional coordinate system includes the coordinate data of the second projection point in the two-dimensional coordinate system.

[0105] It should be noted that the process of calculating the coordinate data of the second projection point in the three-dimensional coordinate system is as follows: Figure 5 The embodiments shown will not be described again here.

[0106] S404. Determine the camera pose corresponding to the surface based on the coordinate data of the second projection point in the three-dimensional coordinate system.

[0107] The camera's shooting direction is towards the surface and parallel to the surface's normal vector.

[0108] In one example, the number of coordinate data points of the center point in the three-dimensional coordinate system is four. Then, for each of these four coordinate data points, the position in each of the four three-dimensional coordinate systems is used as the position where the camera captures the surface.

[0109] Based on the above technical solution, the camera pose determination method provided in this application determines multiple first projection points on the corresponding projection plane for each of the multiple surfaces of the device to be inspected. For each surface, based on the multiple first projection points on the corresponding projection plane, a second projection point of the camera on the projection plane is determined, i.e., the position of the camera in the two-dimensional coordinate system. Then, based on the coordinate data of the second projection point in the two-dimensional coordinate system, the coordinate data of the second projection point in the three-dimensional coordinate system is calculated, and based on the coordinate data in the three-dimensional coordinate system, the camera position corresponding to the surface is determined, and the camera's shooting direction is determined to be facing the surface and parallel to the surface's normal vector. Compared with the current quality inspection methods, which suffer from low quality inspection efficiency and high costs due to too many camera points or too many cameras, the above technical solution first determines the camera's position in the two-dimensional coordinate system, and then further determines the camera's position in the three-dimensional coordinate system based on the position in the two-dimensional coordinate system. This greatly improves the accuracy of camera pose determination, thereby effectively reducing the number of camera points, improving quality inspection efficiency, and reducing costs.

[0110] As one possible embodiment of this application, combined with Figure 4 ,like Figure 5 As shown, the process of the camera pose determination device in S403 calculating the coordinate data of the second projection point in the three-dimensional coordinate system can be implemented through the following S501-S503.

[0111] S501. Construct a target function based on the preset plane equation, the coordinates of the first projection point, and the preset shooting distance corresponding to the set of first projection points.

[0112] In one possible implementation, the process of constructing the objective function can be achieved through the following steps 5-7.

[0113] Step 5: Determine the first function based on the preset plane equation, the coordinates of the first projection point, and the preset shooting distance.

[0114] For example, the first function E pi This can be achieved using the following formula:

[0115]

[0116] Where, m i x represents the number of first projection points in the first projection point set. j y j z j Let A be the coordinates of the first projection point. i B i C i D i is the parameter of the preset plane equation, and D is the preset shooting distance.

[0117] It should be noted that the preset plane equation is A. i x+B i y+C i z+D i =0, and by substituting the three-dimensional coordinates of the first projection point into the preset plane equation, the plane in which the first projection point is located can be obtained, and then the normal vector of the plane can be obtained.

[0118] Step 6: Determine the second function based on the angle between the shooting direction corresponding to each first projection point and the preset shooting direction corresponding to the set of first projection points.

[0119] For example, the second function E αi This can be achieved using the following formula:

[0120]

[0121] Among them, u i v is the normal vector of the surface; j This is the preset shooting direction.

[0122] It should be noted that the preset shooting direction is the optimal observation direction for the first projection point.

[0123] Step 7: Determine the objective function based on the first function and the second function.

[0124] For example, the objective function E i This can be achieved using the following formula:

[0125] E i =γE pi +(1-γ)E αi +∈

[0126] Where γ is the weighting parameter and ∈ is a constant.

[0127] In one example, γ ranges from 0 to 1, used to balance E. pi With E αi The weight between them, γ can be 0.3. ∈ can be 0.01.

[0128] S502. Solve the objective function based on each first projection point in each set of first projection points to obtain the parameter values ​​in the preset plane equation.

[0129] In one example, the Levenberg-Marquardt algorithm is used to solve the nonlinear least squares problem in the objective function, yielding A in the objective function. i B i C i D i u i The value of A. Then, A i B i C i D i Substitute it into the preset plane equation so that the preset plane equation can be used in practice.

[0130] S503. Based on the coordinate data of each second projection point in the two-dimensional coordinate system and the parameter values ​​in the preset plane equation, determine the coordinate data of the second projection point in the three-dimensional coordinate system.

[0131] In one example, the coordinate data of the second projection point in the two-dimensional coordinate system is input into a preset plane equation to obtain the coordinate data of the second projection point in the three-dimensional coordinate system.

[0132] Based on the above technical solution, an objective function is constructed according to the preset plane equation, the coordinates of the first projection points, and the preset shooting positions corresponding to the set of first projection points. The objective function is solved for each first projection point in each set of first projection points to obtain the parameter values ​​in the preset plane equation. Then, the coordinate data of each second projection point in the two-dimensional coordinate system is substituted into the preset plane equation with known parameter values ​​to determine the coordinate data of the second projection point in the three-dimensional coordinate system. The above technical solution can determine the data of the second projection point in the three-dimensional coordinate system using the coordinate data of the second projection point in the two-dimensional coordinate system.

[0133] This application embodiment can divide the camera pose determination device into functional modules or functional units according to the above method example. For example, each function can be divided into a separate functional module or functional unit, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or in software functional modules or functional units. The module or unit division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0134] like Figure 6 The diagram shown is a schematic diagram of a camera pose determination device 60 provided in an embodiment of this application. The camera pose determination device 60 includes a communication unit 601 and a processing unit 602.

[0135] The communication unit 601 is used to determine multiple first projection points of each of the multiple surfaces of the device under test on the corresponding projection plane; the plane on which the surface of the device under test is located is parallel to the projection plane corresponding to the surface.

[0136] The processing unit 602 is configured to determine, for each of a plurality of surfaces, a second projection point of the camera on the projection plane based on a plurality of first projection points of the surface on the corresponding projection plane.

[0137] The processing unit 602 is further configured to calculate the coordinate data of the second projection point in the three-dimensional coordinate system based on the coordinate data of the second projection point in the two-dimensional coordinate system, wherein the coordinate data of the second projection point in the three-dimensional coordinate system includes the coordinate data of the second projection point in the two-dimensional coordinate system.

[0138] The processing unit 602 is also used to determine the camera pose corresponding to the surface based on the coordinate data of the second projection point in the three-dimensional coordinate system; the shooting direction of the camera is towards the surface and parallel to the surface normal vector.

[0139] Optionally, the processing unit 602 is specifically used for:

[0140] For multiple first projection points, the first operation is repeated until the radius of the target range of each cluster is less than a preset threshold. Then, the center point of each subset is determined as multiple second projection points. The first projection points in each subset are all located within the target range. The first operation includes: dividing the multiple first projection points into M sets of first projection points; M is a positive integer; for each of the M sets of first projection points, the radius of the target range of each set is determined; it is determined whether there is a radius greater than or equal to a preset threshold; if so, the multiple first projection points are re-divided to obtain N sets of first projection points; N is a positive integer; N is not equal to M.

[0141] Optionally, the processing unit 602 is specifically used to: construct an objective function based on a preset plane equation, the coordinates of the first projection point, and a preset shooting distance corresponding to the set of first projection points; solve the objective function based on each first projection point in each set of first projection points to obtain the parameter values ​​in the preset plane equation; and determine the coordinate data of the second projection point in the three-dimensional coordinate system based on the coordinate data of each second projection point in the two-dimensional coordinate system and the parameter values ​​in the preset plane equation.

[0142] Optionally, the processing unit 602 is specifically used to: determine a first function based on a preset plane equation, the coordinates of the first projection point, and the distance to a preset shooting position; determine a second function based on the angle between the shooting direction corresponding to each first projection point and the preset shooting direction corresponding to the set of first projection points; and determine a target function based on the first function and the second function.

[0143] Optional, the first function E pi This can be achieved using the following formula:

[0144]

[0145] Where, m i x represents the number of first projection points in the first set of projection points; j y j z j Let A be the coordinates of the first projection point; i B i C i D i is the parameter of the preset plane equation; D is the preset shooting distance.

[0146] Optional, the second function E αi This can be achieved using the following formula:

[0147]

[0148] Among them, u i v is the normal vector of the surface; j This is the preset shooting direction.

[0149] Optional, objective function E i This can be achieved using the following formula:

[0150] E i =γE pi +(1-γ)E αi +∈

[0151] Where γ is the weighting parameter and ∈ is a constant.

[0152] In one possible implementation, the camera pose determination device 60 may further include a storage unit 603. Figure 6 (shown in dashed box) The storage unit 603 stores a program or instruction. When the processing unit 602 executes the program or instruction, the camera pose determination device 60 can perform the camera pose determination method described in the above method embodiment.

[0153] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0154] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the camera pose determination method in the above method embodiments.

[0155] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the camera pose determination method in the method flow shown in the above method embodiments.

[0156] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: electrical connections having one or more wires; portable computer disks; hard disks; random access memory (RAM); read-only memory (ROM); erasable programmable read-only memory (EPROM); registers; hard disks; optical fibers; portable compact disc read-only memory (CD-ROM); optical storage devices; magnetic storage devices; or any suitable combination thereof; or any other form of computer-readable storage medium known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium may also be a component of the processor. The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). In the embodiments of this application, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0157] Since the camera pose determination device, computer-readable storage medium, and computer program product in the embodiments of this application can be applied to the above method, the technical effects that can be obtained can also be referred to the above method embodiments. The embodiments of this application will not be repeated here.

[0158] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0159] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0160] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0161] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for determining camera pose, characterized in that, The method includes: Determine multiple first projection points on the corresponding projection plane for each of the multiple surfaces of the device under test; the plane containing the surface of the device under test is parallel to the projection plane corresponding to the surface. For each of the plurality of surfaces, a second projection point of the camera on the projection plane is determined based on a plurality of first projection points of the surface on the corresponding projection plane; Based on the coordinate data of the second projection point in the two-dimensional coordinate system, the coordinate data of the second projection point in the three-dimensional coordinate system is calculated. The coordinate data of the second projection point in the three-dimensional coordinate system includes the coordinate data of the second projection point in the two-dimensional coordinate system. Based on the coordinate data of the second projection point in the three-dimensional coordinate system, the camera pose corresponding to the surface is determined; the shooting direction of the camera is towards the surface and parallel to the normal vector of the surface; The step of calculating the coordinate data of the second projection point in the three-dimensional coordinate system based on the coordinate data of the second projection point in the two-dimensional coordinate system includes: An objective function is constructed based on a preset plane equation, the coordinates of the first projection point, and a preset shooting distance corresponding to the set of first projection points; the preset shooting distance is determined based on the focal length of the camera. Based on each first projection point in each set of first projection points, the objective function is solved to obtain the parameter values ​​in the preset plane equation; Based on the coordinate data of the second projection point in the two-dimensional coordinate system and the parameter values ​​in the preset plane equation, the coordinate data of the second projection point in the three-dimensional coordinate system are determined. The step of constructing a target function based on a preset plane equation, the coordinates of the first projection point, and a preset shooting distance corresponding to the set of first projection points includes: The first function is determined based on the preset plane equation, the coordinates of the first projection point, and the preset shooting distance; The second function is determined based on the angle between the shooting direction corresponding to each first projection point and the preset shooting direction corresponding to the set of first projection points; The target function is determined based on the first function and the second function.

2. The method according to claim 1, characterized in that, Determining the second projection point of the camera on the projection plane based on a plurality of first projection points of the surface on the corresponding projection plane includes: For the plurality of first projection points, the first operation is repeated until the radius of the target range of each cluster is less than a preset threshold, and the center point of each subset is determined as the second projection point; the first projection points in each subset are all located within the target range; The first operation includes: The plurality of first projection points are divided into M sets of first projection points; M is a positive integer; For each of the M sets of first projection points, determine the radius of the target range for each set; Determine whether there exists a radius greater than or equal to the preset threshold; If they exist, the multiple first projection points are re-divided to obtain a set of N first projection points; N is a positive integer; N is not equal to M.

3. The method according to claim 1, characterized in that, The first function This can be achieved through the following formula: in, The number of first projection points in the first set of projection points; , , The coordinates of the first projection point; , , , The parameters are those of the preset plane equation; The preset shooting distance is [the distance to be specified].

4. The method according to claim 3, characterized in that, The second function This can be achieved through the following formula: in, Let be the normal vector of the surface; The preset shooting direction is defined as follows.

5. The method according to claim 4, characterized in that, The objective function This can be achieved through the following formula: in, These are weight parameters; It is a constant.

6. A camera pose determination device, characterized in that, The device includes a communication unit and a processing unit; The communication unit is used to determine multiple first projection points of each of the multiple surfaces of the device under test on a corresponding projection plane; the plane containing the surface of the device under test is parallel to the projection plane corresponding to the surface. The processing unit is configured to determine, for each of the plurality of surfaces, a second projection point of the camera on the projection plane based on a plurality of first projection points of the surface on the corresponding projection plane; The processing unit is further configured to calculate the coordinate data of the second projection point in the three-dimensional coordinate system based on the coordinate data of the second projection point in the two-dimensional coordinate system, wherein the coordinate data of the second projection point in the three-dimensional coordinate system includes the coordinate data of the second projection point in the two-dimensional coordinate system. The processing unit is further configured to determine the camera pose corresponding to the surface based on the coordinate data of the second projection point in the three-dimensional coordinate system; the shooting direction of the camera is towards the surface and parallel to the normal vector of the surface; The processing unit is specifically used for: An objective function is constructed based on a preset plane equation, the coordinates of the first projection point, and a preset shooting distance corresponding to the set of first projection points; the preset shooting distance is determined based on the focal length of the camera. Based on each first projection point in each set of first projection points, the objective function is solved to obtain the parameter values ​​in the preset plane equation; Based on the coordinate data of each second projection point in the two-dimensional coordinate system and the parameter values ​​in the preset plane equation, the coordinate data of the second projection point in the three-dimensional coordinate system are determined. The processing unit is specifically used for: The first function is determined based on the preset plane equation, the coordinates of the first projection point, and the preset shooting distance; The second function is determined based on the angle between the shooting direction corresponding to each first projection point and the preset shooting direction corresponding to the set of first projection points; The target function is determined based on the first function and the second function.

7. The apparatus according to claim 6, characterized in that, The processing unit is specifically used for: For the plurality of first projection points, the first operation is repeated until the radius of the target range of each cluster is less than a preset threshold, and then the center point of each subset is determined as the second projection point. The first projection point within each subset is located within the target range; The first operation includes: The plurality of first projection points are divided into M sets of first projection points; M is a positive integer; For each of the M sets of first projection points, determine the radius of the target range for each set; Determine whether there exists a radius greater than or equal to the preset threshold; If they exist, the multiple first projection points are re-divided to obtain a set of N first projection points; N is a positive integer; N is not equal to M.

8. The apparatus according to claim 6, characterized in that, The first function This can be achieved through the following formula: in, The number of first projection points in the first set of projection points; , , The coordinates of the first projection point; , , , The parameters are those of the preset plane equation; The preset shooting distance is [the distance to be specified].

9. The apparatus according to claim 6, characterized in that, The second function This can be achieved through the following formula: in, Let be the normal vector of the surface; The preset shooting direction is defined as follows.

10. The apparatus according to claim 6, characterized in that, The objective function This can be achieved through the following formula: in, These are weight parameters; It is a constant.

11. A camera pose determination device, characterized in that, include: A processor and a communication interface; the communication interface is coupled to the processor, the processor being used to run computer programs or instructions to implement the camera pose determination method as described in any one of claims 1-5.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by a computer, perform the camera pose determination method as described in any one of claims 1-5.

Images