Object location methods, apparatus, systems, storage media, and program products

Abstract

This application relates to an object localization method, apparatus, system, storage medium, and program product. Multiple cameras are arranged in a preset localization area of ​​the object to be detected, all cameras including at least a reference camera. The method includes: when the object to be detected arrives at the preset localization area at a current time, acquiring point cloud information of the object surface captured by the multiple cameras respectively; generating multiple second pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system based on the multiple object surface point cloud information, standard point cloud information, and all first pose transformation relationships; generating a target object pose transformation amount based on the multiple second pose transformation relationships and the zero-bias pose transformation relationship; and localizing the object to be detected based on the target object pose transformation amount. This method can accurately localize large-sized objects.

Description

Technical Field

[0001] This application relates to the field of intelligent manufacturing technology, and in particular to an object positioning method, apparatus, system, storage medium, and program product. Background Technology

[0002] In automated production lines for workpieces, the object to be inspected, such as a workpiece, needs to be positioned before subsequent processes can proceed when it arrives at the workstation. Traditional methods often use mechanical structures to restrict the degrees of freedom of the object to be inspected, thereby achieving positioning; however, this method is not accurate enough.

[0003] Currently, with the rapid development of computer vision, the use of vision to locate objects to be inspected has been widely applied in various industries. For example, a robotic arm equipped with a 2D (Two-Dimensional) camera can be used to perform 2D visual localization of the features at both ends of the object to be inspected, and finally obtain the localization result of the object in the robotic arm coordinate system.

[0004] However, current object positioning methods are still not accurate enough for large objects with few surface process features. Summary of the Invention

[0005] Therefore, it is necessary to provide an object positioning method, apparatus, system, computer equipment, computer-readable storage medium, and computer program product for accurately positioning large objects, in order to address the aforementioned technical problems.

[0006] In a first aspect, this application provides an object localization method, wherein multiple cameras are arranged in a preset localization area of ​​the object to be detected, and all cameras include at least a reference camera; the method includes:

[0007] The standard point cloud information of the object to be detected is obtained, the first pose transformation relationship between any two in the camera coordinate systems corresponding to multiple cameras, and the zero-bias pose transformation relationship when the object to be detected is in the zero-bias position in the preset positioning area. The zero-bias pose transformation relationship is the pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system corresponding to the standard point cloud information at the zero-bias moment. The reference camera coordinate system is the camera coordinate system corresponding to the reference camera.

[0008] When the object to be detected reaches the preset positioning area at the current moment, acquire point cloud information of the object surface captured by multiple cameras;

[0009] Based on the point cloud information of multiple object surfaces, the standard point cloud information and all first pose transformation relationships, generate multiple second pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system.

[0010] Based on multiple second pose transformation relationships and zero-bias pose transformation relationships, the pose transformation amount of the target object is generated;

[0011] The target object is located based on its pose transformation.

[0012] In one embodiment, based on multiple object surface point cloud information, standard point cloud information, and all first pose transformation relationships, multiple second pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system are generated, including:

[0013] The point cloud information of the object surface captured by multiple cameras is registered with the standard point cloud information to obtain the point cloud registration results corresponding to each of the multiple cameras.

[0014] For each camera, based on the first pose transformation relationship between the camera coordinate system and the reference camera coordinate system and the point cloud registration result, the second pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system is obtained.

[0015] In one embodiment, obtaining the zero-bias pose transformation relationship when the object to be detected is in a zero-bias position within a preset positioning region includes:

[0016] When the object to be detected is in the zero-bias position in the preset positioning area at the zero-bias moment, acquire the zero-bias surface point cloud information of the object to be detected by multiple cameras respectively;

[0017] Based on the zero-bias surface point cloud information of multiple objects, the standard point cloud information, and all first pose transformation relationships, multiple zero-bias pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system are generated.

[0018] In one embodiment, the target object pose transformation amount is generated based on multiple second pose transformation relationships and zero-bias pose transformation relationships, including:

[0019] For each camera, based on the second pose transformation relationship and the zero-bias pose transformation relationship, the predicted pose parameter transformation of the object to be detected between the historical time and the current time is generated;

[0020] Based on the predicted pose parameter transformations of all cameras, generate the target object pose transformation when the pose error of the object to be detected is less than a preset pose error threshold.

[0021] In one embodiment, obtaining the first pose transformation relationship between any two cameras in their respective camera coordinate systems includes:

[0022] When the object to be detected reaches the preset positioning area at a historical moment, acquire the historical object surface point cloud information captured by multiple cameras respectively.

[0023] For each camera, based on the captured historical object surface point cloud information and standard point cloud information, the third pose transformation relationship between the camera coordinate system and the standard point cloud coordinate system corresponding to the point cloud information under the point cloud point pair distance error is obtained. Here, the point cloud point pair distance error refers to the distance error between any historical object surface point in the historical object surface point cloud information and the standard point in the standard point cloud information that matches the historical object surface point in the point cloud registration process.

[0024] For any two target cameras among multiple cameras, based on the third pose transformation relationship of the two target cameras, the first pose transformation relationship between the camera coordinate systems corresponding to the two target cameras under the historical pose error of the object to be detected is obtained. Among them, the fusion error composed of the point cloud point pair distance error and the historical pose error is less than the preset fusion error value.

[0025] In one embodiment, acquiring standard point cloud information of the object to be detected includes:

[0026] The object to be detected is imported into a preset virtual simulation environment to obtain the virtual object to be detected, wherein at least a virtual camera exists in the preset virtual simulation environment;

[0027] The standard point cloud information of the virtual object to be detected is obtained by taking pictures of the virtual object with a virtual camera.

[0028] Secondly, this application also provides an object positioning device, wherein a plurality of cameras are arranged in a preset positioning area of ​​the object to be detected, and all cameras include at least a reference camera; comprising:

[0029] The information acquisition module is used to acquire the standard point cloud information of the object to be detected, the first pose transformation relationship between any two in the camera coordinate systems corresponding to multiple cameras, and the zero-bias pose transformation relationship when the object to be detected is in the zero-bias position in the preset positioning area. The zero-bias pose transformation relationship is the pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system corresponding to the standard point cloud information at the zero-bias moment. The reference camera coordinate system is the camera coordinate system corresponding to the reference camera.

[0030] The point cloud acquisition module is used to acquire point cloud information of the object surface captured by multiple cameras when the object to be detected reaches the preset positioning area at the current moment.

[0031] The pose generation module is used to generate multiple second pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system based on multiple object surface point cloud information, standard point cloud information and all first pose transformation relationships.

[0032] The pose transformation quantity generation module is used to generate the pose transformation quantity of the target object based on multiple second pose transformation relationships and zero-bias pose transformation relationships.

[0033] The localization module is used to locate the object to be detected based on the pose transformation of the target object.

[0034] Thirdly, this application also provides an object localization system, including multiple cameras and a controller arranged in a preset localization area of ​​the object to be detected; the multiple cameras are used to capture point cloud information of the object to be detected; the controller is used to perform the following steps:

[0035] The standard point cloud information of the object to be detected is obtained, the first pose transformation relationship between any two in the camera coordinate systems corresponding to multiple cameras, and the zero-bias pose transformation relationship when the object to be detected is in the zero-bias position in the preset positioning area. The zero-bias pose transformation relationship is the pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system corresponding to the standard point cloud information at the zero-bias moment. The reference camera coordinate system is the camera coordinate system corresponding to the reference camera.

[0036] When the object to be detected reaches the preset positioning area at the current moment, acquire point cloud information of the object surface captured by multiple cameras;

[0037] Based on the point cloud information of multiple object surfaces, the standard point cloud information and all first pose transformation relationships, generate multiple second pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system.

[0038] Based on multiple second pose transformation relationships and zero-bias pose transformation relationships, the pose transformation amount of the target object is generated;

[0039] The target object is located based on its pose transformation.

[0040] Fourthly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0041] The standard point cloud information of the object to be detected is obtained, the first pose transformation relationship between any two in the camera coordinate systems corresponding to multiple cameras, and the zero-bias pose transformation relationship when the object to be detected is in the zero-bias position in the preset positioning area. The zero-bias pose transformation relationship is the pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system corresponding to the standard point cloud information at the zero-bias moment. The reference camera coordinate system is the camera coordinate system corresponding to the reference camera.

[0042] When the object to be detected reaches the preset positioning area at the current moment, acquire point cloud information of the object surface captured by multiple cameras;

[0043] Based on the point cloud information of multiple object surfaces, the standard point cloud information and all first pose transformation relationships, generate multiple second pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system.

[0044] Based on multiple second pose transformation relationships and zero-bias pose transformation relationships, the pose transformation amount of the target object is generated;

[0045] The target object is located based on its pose transformation.

[0046] Fifthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0047] The standard point cloud information of the object to be detected is obtained, the first pose transformation relationship between any two in the camera coordinate systems corresponding to multiple cameras, and the zero-bias pose transformation relationship when the object to be detected is in the zero-bias position in the preset positioning area. The zero-bias pose transformation relationship is the pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system corresponding to the standard point cloud information at the zero-bias moment. The reference camera coordinate system is the camera coordinate system corresponding to the reference camera.

[0048] When the object to be detected reaches the preset positioning area at the current moment, acquire point cloud information of the object surface captured by multiple cameras;

[0049] Based on the point cloud information of multiple object surfaces, the standard point cloud information and all first pose transformation relationships, generate multiple second pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system.

[0050] Based on multiple second pose transformation relationships and zero-bias pose transformation relationships, the pose transformation amount of the target object is generated;

[0051] The target object is located based on its pose transformation.

[0052] Sixthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0053] The standard point cloud information of the object to be detected is obtained, the first pose transformation relationship between any two in the camera coordinate systems corresponding to multiple cameras, and the zero-bias pose transformation relationship when the object to be detected is in the zero-bias position in the preset positioning area. The zero-bias pose transformation relationship is the pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system corresponding to the standard point cloud information at the zero-bias moment. The reference camera coordinate system is the camera coordinate system corresponding to the reference camera.

[0054] When the object to be detected reaches the preset positioning area at the current moment, acquire point cloud information of the object surface captured by multiple cameras;

[0055] Based on the point cloud information of multiple object surfaces, the standard point cloud information and all first pose transformation relationships, generate multiple second pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system.

[0056] Based on multiple second pose transformation relationships and zero-bias pose transformation relationships, the pose transformation amount of the target object is generated;

[0057] The target object is located based on its pose transformation.

[0058] The aforementioned object localization method, apparatus, system, computer equipment, computer-readable storage medium, and computer program product, considering the large size and limited surface process features of the object to be located, deploy multiple cameras in the preset positioning area of ​​the object to be located, with all cameras including at least a reference camera. Throughout the object localization process, it does not rely on the process features of the object to be located. Instead, when the object to be located reaches the preset positioning area at the current moment, multiple object surface point cloud information is collected by multiple cameras. Then, based on the multiple object surface point cloud information, standard point cloud information, and all first pose transformation relationships, multiple second pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system at the current moment are generated. Furthermore, based on a comparative analysis of the multiple second pose transformation relationships at the current moment and the pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system at the zero-bias moment, the pose transformation amount of the target object is accurately generated. Finally, based on the target object pose transformation amount, the large-sized object to be located is accurately positioned. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a diagram illustrating the application environment of an object location method in one embodiment.

[0061] Figure 2 This is a flowchart illustrating an object location method in one embodiment;

[0062] Figure 3 This is a flowchart illustrating the object location method in another embodiment;

[0063] Figure 4 This is a structural block diagram of an object positioning device in one embodiment;

[0064] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0066] Taking a workpiece as the object to be inspected as an example, in an automated production line, when a workpiece reaches a specific station, it needs to be positioned before subsequent processing can proceed. Traditionally, mechanical structures are mostly used to restrict the workpiece's degrees of freedom to achieve positioning. However, with the rapid development of computer vision, visual methods for workpiece positioning have been widely adopted across various industries.

[0067] For example, one related technology uses multiple 2D cameras, requiring adjustments to the camera mounting angles so that the optical axes of all cameras converge at a single point after passing through the feature holes. In practical applications, this is difficult to deploy and implement. Furthermore, this method relies on the feature holes of the workpiece; if the workpiece is large and has few features, the accuracy and compatibility of this method will deteriorate.

[0068] Another related technology uses a robotic arm equipped with a 2D camera to perform 2D visual localization of the features at both ends of a large workpiece, and finally obtains the workpiece's positioning result in the robotic arm's coordinate system. This method still relies on known features of the object's surface, and it fails when the object's size exceeds the reach of the robotic arm.

[0069] It can be seen that current object localization methods are still not accurate enough for large objects with few surface process features. Considering the large size and few surface process features of the object to be localized, multiple cameras are arranged in the preset positioning area of ​​the object to be localized, and all cameras include at least a reference camera. In the entire object localization process, it is not necessary to rely on the process features of the object to be localized. Instead, when the object to be localized reaches the preset positioning area at the current moment, multiple object surface point cloud information is collected by multiple cameras. Then, based on the multiple object surface point cloud information, standard point cloud information and all first pose transformation relationships, multiple second pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system at the current moment are generated. Based on the combination of multiple second pose transformation relationships at the current moment and the pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system at the zero bias moment, the pose transformation amount of the target object is accurately generated. Then, based on the pose transformation amount of the target object, the large object to be localized is accurately located.

[0070] The object location method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with controller 104 via a network. Controller 104 is used to locate the object 106 to be detected. Multiple cameras 108 are arranged in the preset positioning area of ​​the object 106 to be detected. Figure 1 The diagram illustrates four cameras 108, and the controller 104 can control the cameras 108 to perform imaging operations on the object 106 to be detected. A data storage system can store the data that the controller 104 needs to process. The data storage system can be integrated into the controller 104, or it can be located in the cloud or on another network server.

[0071] The user triggers an object positioning control on the object positioning interface of terminal 102. Terminal 102 responds to the trigger request of the object positioning control, generates an object positioning request, and sends the object positioning request to controller 104. Controller 104 obtains the standard point cloud information of the object to be detected 106, the first pose transformation relationship between any two of the camera coordinate systems corresponding to the multiple cameras 108, and the zero-bias pose transformation relationship when the object to be detected is in the zero-bias position of the preset positioning area. When the object to be detected 106 reaches the preset positioning area at the current moment, it controls the multiple cameras 108 to capture the object surface point cloud information of the object to be detected. Controller 104 obtains the object surface point cloud information of the object to be detected 106 captured by the multiple cameras 108. Based on the multiple object surface point cloud information, the standard point cloud information, and all first pose transformation relationships, it generates multiple second pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system. Based on the multiple second pose transformation relationships and the zero-bias pose transformation relationship, it generates the target object pose transformation amount. Based on the target object pose transformation amount, it locates the object to be detected 106. Furthermore, the location result of the object to be detected 106 can be pushed to the terminal 102, which will then display it to the user.

[0072] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses.

[0073] In one exemplary embodiment, such as Figure 2 As shown, an object location method is provided, which can be applied to... Figure 1 The method is illustrated using controller 104 as an example. Multiple cameras are arranged in the preset positioning area of ​​the object to be detected, and all cameras include at least a reference camera. The method includes:

[0074] S100: Obtain the standard point cloud information of the object to be detected, the first pose transformation relationship between any two in the camera coordinate systems of multiple cameras, and the zero-bias pose transformation relationship when the object to be detected is in the zero-bias position of the preset positioning area.

[0075] Among them, the zero-bias pose transformation relationship is the pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system corresponding to the standard point cloud information at the zero-bias moment, and the reference camera coordinate system is the camera coordinate system corresponding to the reference camera.

[0076] Specifically, multiple cameras are positioned within a pre-defined positioning area of ​​the object to be inspected. In practical applications, this pre-defined positioning area is the area surrounding the production line, meaning multiple cameras are positioned around the production line of the object to be inspected. These cameras can be 3D (3D) cameras or other cameras capable of capturing point clouds of the object. The camera layout needs to ensure that after a large object to be inspected is positioned and stops, the camera's field of view can uniformly cover the four sides of the object. Generally, cameras are placed at the four corners of the object; if the object to be inspected is long, 3D cameras can be added on both sides of the object.

[0077] Furthermore, extrinsic parameter calibration is performed on all cameras to obtain the first pose transformation relationship between any two cameras in their respective camera coordinate systems. This first pose transformation relationship can be considered as the extrinsic parameter between the two corresponding cameras. In addition, it is necessary to obtain the standard point cloud information of the object to be detected, and the zero-bias pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system when the object is at a zero-bias position in the preset positioning area at the zero-bias moment. The standard point cloud coordinate system is the coordinate system corresponding to the standard point cloud information.

[0078] Regarding the zero-bias position of the object to be inspected within the preset positioning area, it's important to explain that for large objects, the positioning strategy on the production line is generally as follows: First, the object to be inspected arrives at its designated position and stops on the production line, marking this as the zero-bias position. Based on this position, all cameras perform a positioning operation and record it as the zero-bias positioning result. The relevant robotic arms are then taught their work trajectories and recorded as the zero-bias work trajectories. Subsequently, when a new object to be inspected arrives, there will be a positioning deviation. At this point, all cameras perform a positioning operation and, combined with the zero-bias positioning result, calculate a correction positioning result. This correction positioning result can be fed back to the relevant robotic arms, and a correction work trajectory can be generated.

[0079] In an exemplary embodiment, obtaining the first pose transformation relationship between any two points in the camera coordinate systems corresponding to multiple cameras can be achieved using various calibration methods, including:

[0080] The first method involves calibrating all 3D cameras using a laser tracker to obtain the extrinsic relationships between them. The camera reference coordinate system used in this method can be either the coordinate system of a specific camera or the laser tracker's coordinate system. Specifically, with the camera reference coordinate system as a reference, the laser tracker calibration logic is as follows: A reference camera is selected, and its coordinate system is used as the world coordinate system. The laser tracker establishes a mapping between the reference coordinate system and its own global coordinate system by measuring the transformation relationship between the reference camera and its own coordinate system (e.g., through a common target). Subsequently, the extrinsic parameters of other cameras are measured using the laser tracker to determine their pose relative to the reference camera, ultimately unifying them to the reference coordinate system. Alternatively, with the camera reference coordinate system as a reference, the laser tracker calibration logic directly uses the laser tracker's global coordinate system (O2). tc X Tc Y Tc Z tc As the world coordinate system, the extrinsic parameters of all cameras are defined by measuring their pose relative to the laser tracker coordinate system, without the need for additional datum transformation.

[0081] The second method involves controlling all cameras to collect historical object surface point cloud information of the object to be detected. Based on the first pose transformation relationship between the historical object surface point cloud information collected by all cameras, the standard point cloud information, and any two of the camera coordinate systems corresponding to each of the multiple cameras, a nonlinear least squares problem is constructed. By solving this nonlinear least squares problem, the accurate first pose transformation relationship is obtained.

[0082] S200: When the object to be detected reaches the preset positioning area at the current moment, acquire point cloud information of the object surface captured by multiple cameras.

[0083] Specifically, when the object to be detected reaches the preset positioning area at the current moment, multiple cameras are controlled to capture the object surface point cloud information of the object to be detected. All cameras upload the captured object surface point cloud information of the object to be detected to the controller. The controller obtains the object surface point cloud information of the object to be detected captured by multiple cameras.

[0084] S300 generates multiple second pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system based on multiple object surface point cloud information, standard point cloud information, and all first pose transformation relationships.

[0085] Specifically, since the zero-bias pose transformation relationship is the pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system at the zero-bias moment, in order to obtain the pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system at the current moment, multiple second pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system can be generated based on the standard point cloud information under the standard point cloud coordinate system, the object surface point cloud information under multiple camera coordinate systems, and the first pose transformation relationship between each camera coordinate system and the reference camera coordinate system.

[0086] S400 generates the target object pose transformation amount based on multiple second pose transformation relationships and zero-bias pose transformation relationships.

[0087] Specifically, the second pose transformation relationship is the pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system at the current moment, and the zero-bias pose transformation relationship is the pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system at the zero-bias moment. By performing nonlinear optimization on multiple second pose transformation relationships and zero-bias pose transformation relationships, the optimal target object pose transformation amount at the current moment relative to the target object at the zero-bias moment can be accurately generated. It should be noted that since the point cloud is captured by a camera, the point cloud coordinate system and the camera coordinate system in this application are essentially the same concept. That is, the standard point cloud coordinate system corresponding to the standard point cloud information is actually equivalent to the coordinate system corresponding to the camera that captured the standard point cloud information, and the camera coordinate system is actually equivalent to the coordinate system corresponding to the object surface point cloud information captured by the camera.

[0088] Furthermore, there is more than one zero-bias pose transformation relationship. The number of zero-bias pose transformation relationships corresponds to the number of cameras actually installed. Each zero-bias pose transformation relationship is paired with a second pose transformation relationship. Nonlinear optimization is performed on multiple sets of second pose transformation relationships and the corresponding zero-bias pose transformation relationships to generate the optimal target object pose transformation amount of the object to be detected at the current time relative to the object to be detected at the zero-bias time.

[0089] The S500 locates the target object based on its pose transformation.

[0090] Specifically, the initial pose information of the object to be detected when it is in a zero-bias position is obtained, and the object to be detected is accurately located based on the pose transformation of the target object and the initial pose information of the object to be detected when it is in a zero-bias position.

[0091] Furthermore, the target object pose transformation includes the change in object position and the change in object angle. The initial pose information of the object to be detected when it is in a zero-bias position includes initial position information and initial angle information. Therefore, in the process of accurately locating the object to be detected based on the target object pose transformation and initial pose information, firstly, the object to be detected is located using the initial position information and the change in object position; secondly, the object to be detected is judged based on the initial angle information and the change in object angle. Based on the results of the location and angle judgment of the object to be detected, the target pose information of the object to be detected is generated, and the target pose information of the object to be detected is used as the localization result of the object to be detected.

[0092] In the above object localization method, considering the large size and limited surface process features of the object to be located, multiple cameras are arranged in the preset positioning area of ​​the object to be located, and all cameras include at least a reference camera. Throughout the object localization process, it is not necessary to rely on the process features of the object to be located. Instead, when the object to be located reaches the preset positioning area at the current moment, multiple object surface point cloud information is collected by multiple cameras. Then, based on the multiple object surface point cloud information, standard point cloud information, and all first pose transformation relationships, multiple second pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system at the current moment are generated. Furthermore, by comparing and analyzing the multiple second pose transformation relationships at the current moment with the pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system at the zero-bias moment, the pose transformation amount of the target object is accurately generated. Finally, based on the target object pose transformation amount, the large-sized object to be located is accurately positioned.

[0093] In an exemplary embodiment, based on multiple object surface point cloud information, standard point cloud information, and all first pose transformation relationships, multiple second pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system are generated, including:

[0094] The point cloud information of the object surface captured by multiple cameras is registered with the standard point cloud information to obtain the point cloud registration results corresponding to each camera. For each camera, according to the first pose transformation relationship between the camera coordinate system and the reference camera coordinate system, the point cloud registration result is mapped to the reference camera coordinate system to obtain the second pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system.

[0095] Specifically, the point cloud information of the object surface to be detected acquired by all cameras is registered with the standard point cloud information of the object to be detected, respectively, to obtain the point cloud registration results corresponding to each camera. Since the zero-bias pose transformation relationship is the pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system at the zero-bias moment, in order to obtain the pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system at the current moment, it is necessary to transform the standard point cloud information from the standard point cloud coordinate system to multiple camera coordinate systems, and then transform it from each camera coordinate system to the reference camera coordinate system, based on the point cloud registration results corresponding to each camera and the first pose transformation relationship between each camera coordinate system and the reference camera coordinate system, to generate multiple second pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system at the current moment.

[0096] In this process, the point cloud information of the object surface captured by multiple cameras is registered with the standard point cloud information respectively. The resulting point cloud registration results for each camera are essentially the pose transformation relationship between the camera coordinate system and the standard point cloud coordinate system.

[0097] For example, if the registration result between the point cloud information of the object surface captured by camera i and the standard point cloud information is T i The first pose transformation relationship between camera i and reference camera j is T. ij Then, based on the first pose transformation relationship Tij and the point cloud registration result T, i , will T i The result converted to the reference camera j is: T i T ij , will T i T ij This serves as the second pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system.

[0098] In the above embodiments, by registering the object surface point cloud information captured by multiple cameras with the standard point cloud information respectively, the point cloud registration results corresponding to each of the multiple cameras can be accurately obtained. Then, the point cloud registration results corresponding to each of the multiple cameras are used as the pose transformation relationship between the camera coordinate system and the standard point cloud coordinate system. Combined with the first pose transformation relationship between the camera coordinate system and the reference camera coordinate system, the second pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system can be accurately obtained.

[0099] In an exemplary embodiment, obtaining the zero-bias pose transformation relationship when the object to be detected is in a zero-bias position within a preset positioning region includes:

[0100] When the object to be detected is in the zero-bias position in the preset positioning area at the zero-bias moment, the zero-bias surface point cloud information of the object to be detected is acquired by multiple cameras respectively; based on the multiple zero-bias surface point cloud information, standard point cloud information and all first pose transformation relationships, multiple zero-bias pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system are generated.

[0101] Specifically, the zero-bias moment refers to the moment when the object to be detected is in a zero-bias position within the preset positioning area, and the zero-bias pose transformation relationship is the pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system at the zero-bias moment. The generation process of the zero-bias pose transformation relationship is similar to the generation process of the second pose transformation relationship.

[0102] When the object to be detected is in the zero-bias position in the preset positioning area at the zero-bias moment, multiple cameras are controlled to capture the zero-bias surface point cloud information of the object to be detected. The zero-bias surface point cloud information of the object to be detected captured by multiple cameras is obtained. Then, based on the multiple zero-bias surface point cloud information, the standard point cloud information and all first pose transformation relationships, multiple zero-bias pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system are generated.

[0103] Furthermore, based on the zero-bias surface point cloud information of multiple objects, the standard point cloud information, and all first pose transformation relationships, multiple zero-bias pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system are generated. This includes: registering the zero-bias surface point cloud information of the objects captured by multiple cameras with the standard point cloud information respectively, obtaining the point cloud zero-bias registration results corresponding to each of the multiple cameras, wherein the point cloud zero-bias registration results are the pose transformation relationships between the camera coordinate system and the standard point cloud coordinate system at the zero-bias moment; for each camera, based on the first pose transformation relationship between the camera coordinate system and the reference camera coordinate system and the point cloud zero-bias registration results, multiple zero-bias pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system are obtained.

[0104] In the above embodiments, when the object to be detected is in a zero-bias position in a preset positioning area at the zero-bias moment, the zero-bias surface point cloud information of the object captured by multiple cameras is registered with the standard point cloud information respectively. The zero-bias registration results of the point clouds corresponding to each of the multiple cameras can be accurately obtained. Then, the zero-bias registration results of the point clouds corresponding to each of the multiple cameras are used as the pose transformation relationship between the camera coordinate system and the standard point cloud coordinate system at the zero-bias moment. Combined with the first pose transformation relationship between the camera coordinate system and the reference camera coordinate system, the second pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system can be accurately obtained.

[0105] In an exemplary embodiment, the target object pose transformation amount is generated based on multiple second pose transformation relationships and zero-bias pose transformation relationships, including:

[0106] For each camera, based on the second pose transformation relationship and the zero-bias pose transformation relationship, the predicted pose parameter transformation of the object to be detected between the historical time and the current time is generated; based on the predicted pose parameter transformation of all cameras, the pose transformation of the target object when the pose error of the object to be detected is less than the preset pose error threshold is generated.

[0107] Specifically, there are multiple second pose transformation relationships. Each second pose transformation relationship is determined by the object surface point cloud information and standard point cloud information captured by each camera at the current moment, and all first pose transformation relationships. Therefore, each second pose transformation relationship corresponds to one camera. Similarly, there are multiple zero-bias pose transformation relationships. Each zero-bias pose transformation relationship is determined by the object zero-bias surface point cloud information and standard point cloud information captured by each camera at the zero-bias moment, and all first pose transformation relationships. Therefore, each zero-bias pose transformation relationship corresponds to one camera.

[0108] Based on the above analysis, each camera corresponds to a second pose transformation relationship and a zero-bias pose transformation relationship. The second pose transformation relationship is the pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system at the current moment, and the zero-bias pose transformation relationship is the pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system at the zero-bias moment. Therefore, for each camera, the predicted pose parameter transformation amount of the object to be detected between the historical moment and the current moment is generated according to the second pose transformation relationship and the zero-bias pose transformation relationship.

[0109] Finally, the predicted pose parameter transformations for all cameras are processed using a nonlinear least squares method to generate the target object pose transformation when the pose error of the object to be detected is less than a preset pose error threshold.

[0110] For example, the pose error expression of the object to be detected includes:

[0111]

[0112] Where i and j are camera numbers, j can be considered as the reference camera number, and N is the number of cameras. T ji T represents the extrinsic parameter relationship between camera i and reference camera j. i0 T represents the pose transformation relationship between the camera i-coordinate system and the standard point cloud coordinate system at the zero bias time. i1 T represents the pose transformation relationship between the camera's i-coordinate system and the standard point cloud coordinate system at the current moment. target T represents the pose transformation of the target object to be determined. ji T i0 T represents the zero-bias pose transformation relationship of camera i at the zero-bias moment. ji Ti0 This represents the second pose transformation relationship corresponding to camera i at the current moment.

[0113] It can be seen that, for camera i, based on the second pose transformation relationship and the zero-bias pose transformation relationship corresponding to camera i, the predicted pose parameter transformation of the object to be detected between the historical time and the current time can be generated. Then, based on the predicted pose parameter transformations for all cameras, the target object pose transformation T is calculated when the pose error is less than a preset pose error threshold. target In the above expression, is the target object pose transformation amount T when the pose error is minimized. target .

[0114] The pose error of the object to be detected refers to the change in the predicted pose parameters for each camera. With the target object pose transformation amount T target The sum of the products between them.

[0115] In an exemplary embodiment, the nonlinear least squares problem of generating target object pose transformations based on multiple second pose transformation relationships and zero-bias pose transformation relationships can be solved using the Levenberg-Marquardt method to obtain the optimal target object pose transformation. It should be noted that the Levenberg-Marquardt method is an estimation method for least squares regression parameters in nonlinear regression, and is also a type of optimization algorithm. It combines the advantages of the Gauss-Newton algorithm and gradient descent to solve nonlinear least squares problems. In the LM algorithm, the objective function is usually expressed as finding the parameter vector x such that the sum of the squared errors between the model predictions and the observed values ​​is minimized.

[0116] In the above embodiments, by generating the predicted pose parameter transformation amount of the object to be detected between the historical time and the current time based on the second pose transformation relationship and the zero-bias pose transformation relationship, the nonlinear least squares method can be used to accurately generate the target object pose transformation amount when the pose error of the object to be detected is less than the preset pose error threshold.

[0117] In one exemplary embodiment, such as Figure 3 As shown, S100 includes:

[0118] S120: Obtain the standard point cloud information of the object to be detected and the zero-bias pose transformation relationship when the object to be detected is in the zero-bias position in the preset positioning area.

[0119] S140: When the object to be detected reaches the preset positioning area at a historical moment, acquire the historical object surface point cloud information captured by multiple cameras.

[0120] S160, for each camera, based on the historical object surface point cloud information and standard point cloud information captured, obtains the third pose transformation relationship between the camera coordinate system under the point cloud point pair distance error and the standard point cloud coordinate system corresponding to the standard point cloud information.

[0121] S180: For any two target cameras among multiple cameras, based on the third pose transformation relationship of the two target cameras, obtain the first pose transformation relationship between the camera coordinate systems corresponding to the two target cameras under the historical pose error of the object to be detected.

[0122] The point cloud point pair distance error refers to the distance error between any historical object surface point in the historical object surface point cloud information and a standard point in the standard point cloud information that matches the historical object surface point during the point cloud registration process. The fusion error constituted by the point cloud point pair distance error and the historical pose error is less than the preset fusion error value.

[0123] Specifically, the standard point cloud information of the object to be detected and the zero-bias pose transformation relationship when the object is in a zero-bias position in a preset positioning area are obtained. In addition, the first pose transformation relationship between any two in the camera coordinate systems corresponding to multiple cameras is also obtained. Specifically, obtaining the first pose transformation relationship between any two in the camera coordinate systems corresponding to multiple cameras includes at least two methods. This embodiment mainly describes: by controlling all cameras to collect historical object surface point cloud information of the object to be detected, a nonlinear least squares problem is constructed based on the historical object surface point cloud information, standard point cloud information, and the first pose transformation relationship between any two in the camera coordinate systems corresponding to multiple cameras. By solving this nonlinear least squares problem, the accurate first pose transformation relationship is obtained.

[0124] More specifically, when the object to be detected reaches the preset positioning area at a historical moment, the point cloud information of the object surface captured by multiple cameras at the historical moment is acquired as a set of data, and the above steps are repeated to acquire multiple sets of data.

[0125] By traversing all data, the historical point cloud information of the object surface to be detected captured by all cameras is registered with the standard point cloud information to obtain the historical point cloud registration results corresponding to each camera. Based on the historical point cloud registration results, a nonlinear least squares problem is constructed between the camera coordinate system corresponding to the historical object surface point cloud information, the standard point cloud coordinate system corresponding to the standard point cloud information, and the first pose transformation relationship to obtain the accurate first pose transformation relationship.

[0126] Based on the historical point cloud registration results, a nonlinear least squares problem is constructed between the camera coordinate system corresponding to the historical object surface point cloud information, the standard point cloud coordinate system corresponding to the standard point cloud information, and the first pose transformation relationship to obtain the accurate first pose transformation relationship. This includes: based on the historical point cloud registration results, performing nonlinear least squares processing on the third pose transformation relationship between the camera coordinate system corresponding to each camera and the standard point cloud coordinate system. The third pose transformation relationship is the pose transformation relationship when the distance error between point cloud point pairs is less than a preset reprojection error threshold. The distance error between point cloud point pairs is the distance error between any historical object surface point in the historical object surface point cloud information and the standard point in the standard point cloud information that matches the historical object surface point during the point cloud registration process. Then, by performing nonlinear least squares processing on the accurate third pose transformation relationship of each of the two target cameras and the first pose transformation relationship between the camera coordinate systems corresponding to the two target cameras, the first pose transformation relationship between the camera coordinate systems corresponding to the two target cameras is obtained. In practical applications, the Levenberg-Marquardt method can be used to solve the nonlinear least squares problem.

[0127] For example, the expression for a nonlinear least squares problem can be as follows:

[0128]

[0129] Where i and j are camera numbers, N is the number of cameras, and T i T represents the third pose transformation relationship between the i-th camera coordinate system and the standard point cloud coordinate system. j T represents the third pose transformation relationship between the j-th camera coordinate system and the standard digital model point cloud coordinate system. ij This represents the transformation relationship between camera i and camera j, where p and q are a set of corresponding points between historical object surface point cloud information captured by the cameras and standard point cloud information, and K is the set of corresponding points.

[0130] The point-to-point distance error in the point cloud registration process refers to... Historical pose error refers to In order to satisfy the nonlinear least squares problem, the point cloud point pair distance error and the historical pose error in the point cloud registration process need to meet the condition that the error is less than a preset threshold, so that the fusion error composed of the point cloud point pair distance error and the historical pose error is less than the preset fusion error value.

[0131] For each camera's point cloud, during the point cloud registration process, there is a point-to-point distance error, where p is a point in the historical object surface point cloud information, q is a point in the standard point cloud information, and T... i This represents the third pose transformation relationship between the i-th camera coordinate system and the standard point cloud coordinate system. Therefore... The process involves transforming point q from the standard point cloud information to the coordinate system of the i-th camera. Based on point p in the historical object surface point cloud information captured by camera i and point q in the i-th camera coordinate system, the distance error between point cloud pairs at camera i is detected. If the distance error is minimized, then T is set accordingly. i This serves as the optimal third pose transformation relationship between the i-th camera coordinate system and the standard point cloud coordinate system. The same steps are then applied to all cameras to obtain the third pose transformation relationship between all camera coordinate systems and the standard point cloud coordinate system under the condition that the point cloud point pair distance error for each camera is minimized (i). In this case, the point cloud point pair distance error is less than a preset reprojection error threshold.

[0132] Regarding the historical pose error of the object to be detected, This is the third pose transformation relationship between the accurate i-th camera coordinate system and the standard point cloud coordinate system after point cloud point pair distance error correction, T. j It is also the accurate third pose transformation relationship between the j-th camera coordinate system and the standard point cloud coordinate system after point cloud point pair distance error correction, T ij This represents the first pose transformation relationship between the i-th camera coordinate system and the j-th camera coordinate system. The third pose transformation relationship is derived from the relationship between the i-th camera coordinate system and the standard point cloud coordinate system. The third pose transformation relationship T between the j-th camera coordinate system and the standard point cloud coordinate system j It can generate historical predicted pose parameter transformation quantities. Then, by combining the historical predicted pose parameter transformations, the first pose transformation relationship is corrected to determine the first pose transformation relationship T that minimizes the historical pose error. ij At this point, the historical pose error is less than the preset historical pose error threshold.

[0133] Since the distance error between point cloud points and the historical pose error are both minimized at this time, the fusion error between the distance error between point cloud points and the historical pose error is also minimized. In other words, the fusion error formed by the distance error between point cloud points and the historical pose error is less than the preset fusion error value, which can make the generation process of the first pose transformation relationship between any two in the camera coordinate system corresponding to each of the multiple cameras accurate.

[0134] In the above embodiments, when calibrating the extrinsic parameters of the 3D camera, external equipment such as laser trackers can be used, and workpiece data from the production line can be directly collected for calibration. Furthermore, by obtaining the third pose transformation relationship between the camera coordinate system with the minimum point cloud point-to-point distance error and the standard point cloud coordinate system corresponding to the standard point cloud information, and the first pose transformation relationship between the camera coordinate systems corresponding to the two target cameras when the historical pose error of the object to be detected is minimum, the first pose transformation relationship between any two of the camera coordinate systems corresponding to multiple cameras can be accurately generated.

[0135] In an exemplary embodiment, obtaining standard point cloud information of the object to be detected includes:

[0136] The object to be detected is imported into a preset virtual simulation environment to obtain the virtual object to be detected. The preset virtual simulation environment contains at least a virtual camera. The virtual object to be detected is photographed by the virtual camera to obtain the standard point cloud information of the object to be detected.

[0137] Specifically, a possible method for obtaining standard point cloud information of the object to be inspected is as follows: import the CAD (Computer-Aided Design) model of the object to be inspected into a virtual simulation environment (such as Blender, Unity, etc.), take pictures of the CAD model with a virtual camera to obtain the standard CAD model point cloud, which is the virtual object to be inspected, and then take pictures of the virtual object to be inspected with a virtual camera set up in the virtual simulation environment to obtain the standard point cloud information of the object to be inspected.

[0138] In the above embodiments, by importing the object to be detected into a preset virtual simulation environment, a virtual object of the object to be detected can be accurately constructed, and standard point cloud information of the object to be detected can be accurately generated.

[0139] In an exemplary embodiment, taking the object to be detected as a workpiece and the preset positioning area as a production line as an example, considering the large size of the workpiece and the lack of surface process features, this application provides a general method for locating large-sized objects. Multiple 3D cameras are used to acquire point cloud data of the workpiece surface, and the workpiece is located by point cloud registration.

[0140] layout:

[0141] When placing 3D cameras around the production line, it is necessary to ensure that the camera's field of view can uniformly cover the perimeter of the workpiece after it has been positioned and stopped.

[0142] Algorithm flow:

[0143] 1. External parameter calibration: Perform external parameter calibration on all 3D cameras to obtain the external parameter relationship between the 3D cameras.

[0144] There are two calibration methods:

[0145] Method 1: Use a laser tracker to calibrate all 3D cameras and obtain the external parameter relationships between them.

[0146] Method 2: After the large workpiece is in place and stops, control all 3D cameras to capture the point cloud of the workpiece surface. Collect this as a set of data. Repeat the above steps to collect multiple sets of data. Iterate through all the data, register the point clouds captured by all 3D cameras with the standard CAD model point cloud, and construct a nonlinear least squares problem between the "camera point cloud coordinate system, the standard CAD model point cloud coordinate system, and camera extrinsic parameters." An example of the nonlinear least squares problem is as follows:

[0147]

[0148] Where i and j are camera numbers, N is the number of cameras, and T i T represents the pose transformation relationship between the i-th camera coordinate system and the standard point cloud coordinate system. j T represents the pose transformation relationship between the j-th camera coordinate system and the standard point cloud coordinate system. ij This represents the pose transformation relationship between camera i and camera j, where p and q are a set of corresponding points between the historical object surface point cloud information captured by the i-th camera and the standard point cloud information, K is the set of corresponding points, and T is the distance between them. i T j It is obtained based on the registration results between the corresponding camera coordinate system and the standard point cloud coordinate system.

[0149] This nonlinear least squares problem can be solved using the Levenberg–Marquardt method, which yields the extrinsic parameter relationships between all 3D cameras.

[0150] 2. Object location:

[0151] For large workpieces, the positioning strategy on the production line is generally as follows: First, the large workpiece is positioned and stopped on the production line, and this is recorded as the zero-offset position. Based on the zero-offset position, all 3D cameras perform a positioning operation and record it as the zero-offset positioning result. The relevant robotic arms are taught their work trajectories and recorded as the zero-offset work trajectories. Subsequently, when a new workpiece arrives, there will be a positioning deviation. At this time, all 3D cameras perform a positioning operation and, combined with the zero-offset positioning result, calculate the correction positioning result. The correction positioning result can be fed back to the relevant robotic arms, and a correction work trajectory can be generated.

[0152] Once a large workpiece has reached its designated position and come to a stable stop on the production line, all cameras acquire point cloud data of the workpiece's surface. The data acquired at the current point and the data acquired at the zero-bias moment are then processed as follows:

[0153] The point cloud information of the object surface acquired by all 3D cameras is registered with the standard point cloud information. Then, based on the extrinsic parameter relationship between the cameras as defined above, the registration results of all cameras are transformed into the reference camera coordinate system, thereby obtaining the pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system.

[0154] Thus, for each camera, the pose transformation relationship between the camera reference coordinate system and the standard point cloud coordinate system at the current moment, and the pose transformation relationship between the camera reference coordinate system and the standard point cloud coordinate system at the zero bias moment can be obtained. By performing nonlinear optimization based on these relationships of all cameras, an optimal pose change of the workpiece at the current moment relative to the workpiece at the zero bias moment can be obtained, thereby realizing the positioning of the workpiece.

[0155] The nonlinear least squares problem in object localization is as follows:

[0156]

[0157] Where i and j are camera numbers, j can be considered as the reference camera number, and N is the number of cameras. T ij T represents the extrinsic parameter relationship between camera i and camera j. i0 T represents the pose transformation relationship between the i-th camera coordinate system and the standard point cloud coordinate system at time zero bias. i1 T represents the pose transformation relationship between the i-th camera coordinate system and the standard point cloud coordinate system at the current moment. target This represents the change in the workpiece's pose that needs to be determined.

[0158] For this nonlinear least squares problem, the Levenberg–Marquardt method can be used to obtain the optimal workpiece positioning result.

[0159] It can be seen that traditional 2D methods are difficult to be compatible with various different workpieces, and the positioning algorithm performs poorly when the workpiece size is large and the workpiece surface has few process features. This application uses a 3D method to capture the point cloud features of the workpiece surface, which is independent of the workpiece's process features and can accommodate larger workpiece sizes. Therefore, this technical solution has the following advantages: 1. Good compatibility with different types of workpieces; 2. Independence from the workpiece surface process features; 3. Good compatibility with large-sized workpieces.

[0160] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0161] Based on the same inventive concept, this application also provides an object positioning device for implementing the object positioning method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more object positioning device embodiments provided below can be found in the limitations of the object positioning method described above, and will not be repeated here.

[0162] In one exemplary embodiment, such as Figure 4 As shown, multiple cameras are arranged in the preset positioning area of ​​the object to be detected, and all cameras include at least a reference camera; an object positioning device is provided, including: an information acquisition module 100, a point cloud acquisition module 200, a pose generation module 300, a pose transformation quantity generation module 400, and a positioning module 500, wherein:

[0163] The information acquisition module 100 is used to acquire the standard point cloud information of the object to be detected, the first pose transformation relationship between any two in the camera coordinate systems corresponding to multiple cameras, and the zero-bias pose transformation relationship when the object to be detected is in the zero-bias position in the preset positioning area. The zero-bias pose transformation relationship is the pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system corresponding to the standard point cloud information at the zero-bias moment. The reference camera coordinate system is the camera coordinate system corresponding to the reference camera.

[0164] The point cloud acquisition module 200 is used to acquire point cloud information of the object surface of the object to be detected captured by multiple cameras when the object to be detected reaches the preset positioning area at the current time.

[0165] The pose generation module 300 is used to generate multiple second pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system based on multiple object surface point cloud information, standard point cloud information and all first pose transformation relationships.

[0166] The pose transformation quantity generation module 400 is used to generate the pose transformation quantity of the target object based on multiple second pose transformation relationships and zero-bias pose transformation relationships.

[0167] The positioning module 500 is used to locate the object to be detected based on the pose transformation of the target object.

[0168] In one embodiment, the pose generation module 300 is further configured to register the object surface point cloud information captured by multiple cameras with the standard point cloud information respectively to obtain the point cloud registration results corresponding to each of the multiple cameras; for each camera, based on the first pose transformation relationship between the camera coordinate system and the reference camera coordinate system and the point cloud registration results, the second pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system is obtained.

[0169] In one embodiment, the information acquisition module 100 is further configured to acquire object zero-bias surface point cloud information of the object to be detected captured by multiple cameras when the object to be detected is at a zero-bias position in a preset positioning area at the zero-bias moment; and generate multiple zero-bias pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system based on the multiple object zero-bias surface point cloud information, the standard point cloud information and all first pose transformation relationships.

[0170] In one embodiment, the pose transformation generation module 400 is further configured to, for each camera, generate the predicted pose parameter transformation amount of the object to be detected between the historical time and the current time based on the second pose transformation relationship and the zero-bias pose transformation relationship; and generate the target object pose transformation amount when the pose error of the object to be detected is less than a preset pose error threshold based on the predicted pose parameter transformation amounts corresponding to all cameras.

[0171] In one embodiment, the information acquisition module 100 is further configured to acquire historical object surface point cloud information captured by multiple cameras when the object to be detected reaches the preset positioning area at a historical moment; for each camera, based on the captured historical object surface point cloud information and standard point cloud information, obtain the third pose transformation relationship between the camera coordinate system under the point cloud point pair distance error and the standard point cloud coordinate system corresponding to the standard point cloud information; for any two target cameras among the multiple cameras, based on the third pose transformation relationship of the two target cameras, obtain the first pose transformation relationship between the camera coordinate systems corresponding to the two target cameras under the historical pose error of the object to be detected, wherein the fusion error constituted by the point cloud point pair distance error and the historical pose error is less than the preset fusion error value.

[0172] For each camera, the information acquisition module 100 is also used to import the object to be detected into a preset virtual simulation environment to obtain the virtual object to be detected, wherein at least a virtual camera exists in the preset virtual simulation environment; and to obtain the standard point cloud information of the virtual object to be detected by taking pictures of the virtual object to be detected through the virtual camera.

[0173] Each module in the aforementioned object positioning device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0174] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores standard point cloud information of the object to be detected, the first pose transformation relationship between any two cameras in their respective camera coordinate systems, and other data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements an object localization method.

[0175] Those skilled in the art will understand that Figure 5 The structure shown is a block diagram of a partial structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0176] In one embodiment, this application also provides an object localization system, including multiple cameras and a controller arranged in a preset localization area of ​​an object to be detected; the multiple cameras are used to capture point cloud information of the object to be detected; the controller is used to perform the following steps:

[0177] The standard point cloud information of the object to be detected is obtained, the first pose transformation relationship between any two in the camera coordinate systems corresponding to multiple cameras, and the zero-bias pose transformation relationship when the object to be detected is in the zero-bias position in the preset positioning area. The zero-bias pose transformation relationship is the pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system corresponding to the standard point cloud information at the zero-bias moment. The reference camera coordinate system is the camera coordinate system corresponding to the reference camera.

[0178] When the object to be detected reaches the preset positioning area at the current moment, acquire point cloud information of the object surface captured by multiple cameras;

[0179] Based on the point cloud information of multiple object surfaces, the standard point cloud information and all first pose transformation relationships, generate multiple second pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system.

[0180] Based on multiple second pose transformation relationships and zero-bias pose transformation relationships, the pose transformation amount of the target object is generated;

[0181] The target object is located based on its pose transformation.

[0182] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0183] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0184] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0185] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0186] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0187] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method of locating an object, characterized by, The preset positioning area of the to-be-detected object is arranged with a plurality of cameras, all of which at least include a reference camera; the method comprises: acquiring standard point cloud information of the to-be-detected object, a first pose transformation relationship between any two of the camera coordinate systems corresponding to the plurality of cameras respectively, and a zero-offset pose transformation relationship when the to-be-detected object is in a zero-offset position in the preset positioning area, wherein the zero-offset pose transformation relationship is a pose transformation relationship between a reference camera coordinate system and a standard point cloud coordinate system corresponding to the standard point cloud information at a zero-offset moment, and the reference camera coordinate system is a camera coordinate system corresponding to the reference camera; in the case that the to-be-detected object reaches the preset positioning area at the current moment, acquiring object surface point cloud information of the to-be-detected object respectively photographed by the plurality of cameras; generating a plurality of second pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system according to the plurality of object surface point cloud information, the standard point cloud information and all the first pose transformation relationships; generating a target object pose transformation quantity according to the plurality of second pose transformation relationships and the zero-offset pose transformation relationship; positioning the to-be-detected object based on the target object pose transformation quantity; wherein the acquiring of the first pose transformation relationship between any two of the camera coordinate systems corresponding to the plurality of cameras respectively comprises: in the case that the to-be-detected object reaches the preset positioning area at a historical moment, acquiring historical object surface point cloud information of the to-be-detected object respectively photographed by the plurality of cameras; for each of the cameras, obtaining a third pose transformation relationship between the camera coordinate system and the standard point cloud coordinate system corresponding to the standard point cloud information under a point cloud point pair distance error according to the historical object surface point cloud information and the standard point cloud information, wherein the point cloud point pair distance error refers to a distance error of any historical object surface point in the historical object surface point cloud information and a standard point in the standard point cloud information matched with the historical object surface point in a point cloud registration process; for any two target cameras in the plurality of cameras, obtaining a first pose transformation relationship between the camera coordinate systems corresponding to the two target cameras respectively according to the third pose transformation relationships of the two target cameras, wherein a fusion error composed of the point cloud point pair distance error and a historical pose error is less than a preset fusion error value.

2. The method of claim 1, wherein, the generating of the plurality of second pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system according to the plurality of object surface point cloud information, the standard point cloud information and all the first pose transformation relationships comprises: respectively registering the object surface point cloud information photographed by the plurality of cameras and the standard point cloud information to obtain a point cloud registration result corresponding to each of the plurality of cameras; for each of the cameras, obtaining a second pose transformation relationship between the reference camera coordinate system and the standard point cloud coordinate system according to the first pose transformation relationship between the camera coordinate system and the reference camera coordinate system and the point cloud registration result.

3. The method of claim 1, wherein, Obtaining a zero-offset pose transformation relationship when the to-be-detected object is in a zero-offset position in a preset positioning area, comprising: When the to-be-detected object is in a zero-offset position in a preset positioning area at a zero-offset time, obtaining object zero-offset surface point cloud information of the to-be-detected object photographed by the plurality of cameras respectively; According to a plurality of the object zero-offset surface point cloud information, the standard point cloud information and all the first pose transformation relationships, a plurality of zero-offset pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system are generated.

4. The method of claim 3, wherein, According to a plurality of the second pose transformation relationships and the zero-offset pose transformation relationship, a target object pose transformation quantity is generated, comprising: For each of the cameras, according to the second pose transformation relationship and the zero-offset pose transformation relationship, a predicted pose parameter transformation quantity of the to-be-detected object between a historical time and a current time is generated; According to the predicted pose parameter transformation quantities corresponding to all the cameras, a target object pose transformation quantity is generated when the pose error of the to-be-detected object is less than a preset pose error threshold.

5. The method of claim 1, wherein, Obtaining standard point cloud information of the to-be-detected object, comprising: Importing the to-be-detected object into a preset virtual simulation environment to obtain a to-be-detected virtual object, wherein the preset virtual simulation environment at least has a virtual camera; Photographing the to-be-detected virtual object by the virtual camera to obtain the standard point cloud information of the to-be-detected object.

6. An object positioning apparatus characterized by comprising: A preset positioning area of a to-be-detected object is arranged with a plurality of cameras, all of which at least include a reference camera; the device comprises: An information acquisition module is configured to acquire standard point cloud information of the to-be-detected object, first pose transformation relationships between any two of camera coordinate systems corresponding to the plurality of cameras, and a zero-offset pose transformation relationship when the to-be-detected object is in a zero-offset position in a preset positioning area, wherein the zero-offset pose transformation relationship is a pose transformation relationship between a reference camera coordinate system and a standard point cloud coordinate system corresponding to the standard point cloud information at a zero-offset time, and the reference camera coordinate system is a camera coordinate system corresponding to the reference camera; A point cloud acquisition module is configured to acquire object surface point cloud information of the to-be-detected object photographed by the plurality of cameras respectively when the to-be-detected object reaches a preset positioning area at a current time; A pose generation module is configured to generate a plurality of second pose transformation relationships between the reference camera coordinate system and the standard point cloud coordinate system according to a plurality of the object surface point cloud information, the standard point cloud information and all the first pose transformation relationships; A pose transformation quantity generation module is configured to generate a target object pose transformation quantity according to a plurality of the second pose transformation relationships and the zero-offset pose transformation relationship; A positioning module is configured to position the to-be-detected object based on the target object pose transformation quantity. The information acquisition module is further configured to acquire historical object surface point cloud information of the to-be-detected object photographed by the plurality of cameras respectively in a case where the to-be-detected object arrives at the preset positioning area at a historical time; for each of the cameras, a third pose transformation relationship between a camera coordinate system of the camera and a standard point cloud coordinate system corresponding to the standard point cloud information is obtained according to the photographed historical object surface point cloud information and the standard point cloud information, wherein the point cloud point pair distance error refers to a distance error of any historical object surface point in the historical object surface point cloud information and a standard point in the standard point cloud information matched with the historical object surface point in a point cloud registration process; for any two target cameras in the plurality of cameras, a first pose transformation relationship between camera coordinate systems corresponding to the two target cameras respectively is obtained according to third pose transformation relationships of the two target cameras, wherein a fusion error composed of the point cloud point pair distance error and a historical pose error is less than a preset fusion error value.

7. An object positioning system comprising a plurality of cameras arranged in a preset positioning area of an object to be detected and a controller, characterized in that, The plurality of cameras are configured to photograph point cloud information of the to-be-detected object; and the controller is configured to execute the object positioning method according to any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 5.

Images