Object positioning method and related device

Abstract

The invention provides an object positioning method and a related device, and relates to the field of image acquisition. In the application, the candidate depth image comprising the depth sub-image of the at least one candidate object can be matched with the reference depth image of the target object to screen out the depth sub-image matched with the reference depth image, and then the candidate object corresponding to the depth sub-image is used as the target object. According to the method, the candidate object matched with the depth image of the target object is determined through image matching, the candidate object is the target object, the spatial position of the target object is determined subsequently, object positioning is achieved, then the target object is found quickly through the spatial position, and subsequent grabbing and other operations are performed.

Description

Technical Field

[0001] The present application relates to the field of image acquisition, and more specifically, to an object positioning method and related devices. Background Art

[0002] With the continuous development of automation, more and more scenarios require the location information of objects. For example, in the field of robot grasping, the position of the object to be grasped must be determined before the object grasping operation can be performed.

[0003] Then, how to realize object positioning is a technical problem that technical personnel in this field need to solve urgently. Summary of the invention

[0004] In view of this, the present application provides an object positioning method and related devices to solve the problem of object positioning.

[0005] In order to solve the above technical problems, this application adopts the following technical solutions:

[0006] An object positioning method, comprising:

[0007] Acquire a candidate depth image; the candidate depth image includes at least one depth sub-image of a candidate object;

[0008] Acquire a reference depth image; the reference depth image is a depth image of the target object;

[0009] Performing a matching operation on the candidate depth image and the reference depth image to select a depth sub-image matching the reference depth image from the candidate depth image;

[0010] The candidate object corresponding to the depth sub-image is taken as a target object, and the spatial position of the target object is determined.

[0011] Optionally, acquiring a candidate depth image includes:

[0012] Using cameras at the same height and at different horizontal positions to collect multiple initial depth images respectively; there is a partial overlap between at least two initial depth images;

[0013] Extracting feature points of the initial depth image;

[0014] The initial depth image is stitched using the feature points to obtain a candidate depth image.

[0015] Optionally, multiple initial depth images are collected respectively by using cameras at the same height and at different horizontal positions, including:

[0016] After adjusting the camera to a specified height, acquiring an initial depth image using the camera;

[0017] The position of the camera is adjusted in the horizontal direction to acquire a new initial depth image by using the camera again, and the step of adjusting the position of the camera in the horizontal direction is returned to be executed until image acquisition of all candidate objects is completed and then stops.

[0018] Optionally, performing a matching operation on the candidate depth image and the reference depth image to select a depth sub-image matching the reference depth image from the candidate depth image includes:

[0019] Using a preset template matching method, at different image resolutions, an image matching operation is performed on the depth sub-image in the candidate depth image and the reference depth image to obtain a similarity between the depth sub-image and the reference depth image;

[0020] A depth sub-image having a similarity with the reference depth image greater than a similarity threshold is selected.

[0021] Optionally, determining the location information of the target object includes:

[0022] Determining an image position of the target object in the candidate depth image;

[0023] The spatial position corresponding to the image position of the target object is determined by using the mapping relationship between the image position and the spatial position.

[0024] Optionally, after taking the candidate object corresponding to the depth sub-image as the target object, the method further includes:

[0025] Receiving a height query instruction of a target point of the target object;

[0026] In response to the height query instruction, the height information of the target point is determined; wherein the height information is calculated by using a trigonometric function relationship to calculate the position where the laser is reflected to the photosensitive element and the installation angle of the laser and the photosensitive element.

[0027] An object positioning device, comprising:

[0028] An image acquisition module, configured to acquire a candidate depth image; the candidate depth image includes at least one depth sub-image of a candidate object;

[0029] An image acquisition module is used to acquire a reference depth image; the reference depth image is a depth image of a target object;

[0030] A matching module, configured to perform a matching operation on the candidate depth image and the reference depth image, so as to select a depth sub-image matching the reference depth image from the candidate depth image;

[0031] The position determination module is used to take the candidate object corresponding to the depth sub-image as the target object and determine the spatial position of the target object.

[0032] An electronic device comprises at least one processor and a memory connected to the processor, wherein:

[0033] The memory is used to store computer programs;

[0034] The processor is used to execute the computer program so that the electronic device can implement the above-mentioned object positioning method.

[0035] An object positioning system comprises a camera, a motion device and the above-mentioned electronic device; the camera is mounted on the motion device via a switching structure;

[0036] The motion device is used to respond to the position adjustment instruction and drive the camera to move according to the position adjustment instruction;

[0037] The camera is used to collect a candidate depth image and send the candidate depth image to the electronic device.

[0038] Optionally, the camera includes a laser galvanometer stereo camera; the motion device is a robot; and the adapter structure and the mounting surface of the laser galvanometer stereo camera are connected using a threaded connector.

[0039] The present application provides an object positioning method and related devices. In the present application, a candidate depth image including at least one depth sub-image of a candidate object can be used to perform a matching operation with a reference depth image of a target object to screen out a depth sub-image that matches the reference depth image, and then the candidate object corresponding to the depth sub-image is used as the target object. That is, the present invention first determines a candidate object that matches the depth image of the target object through image matching, and the candidate object is the target object. Subsequently, the spatial position of the target object is determined to achieve object positioning, and then the target object is quickly found through the spatial position, and then subsequent operations such as grasping are performed. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0041] Figure 1 A schematic diagram of an object positioning system provided by an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of a switching structure provided by an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of the installation of a camera and a switching structure provided by an embodiment of the present invention;

[0044] Figure 4 A schematic diagram of an interface of an image acquisition software provided by an embodiment of the present invention;

[0045] Figure 5 A schematic diagram of an operating handle interface provided by an embodiment of the present invention;

[0046] Figure 6 A schematic diagram of a flow chart of an object positioning method provided by an embodiment of the present invention;

[0047] Figure 7 A schematic diagram of a depth image provided by an embodiment of the present invention;

[0048] Figure 8 A schematic diagram of a depth image of a matched object provided by an embodiment of the present invention;

[0049] Fig. 9 A depth image processing flow chart provided by an embodiment of the present invention;

[0050] Fig.10 A schematic diagram of image acquisition provided by an embodiment of the present invention;

[0051] Fig.11 A schematic diagram showing a height display provided by an embodiment of the present invention;

[0052] Fig.12 A schematic structural diagram of an object positioning device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] With the continuous development of automation, more and more scenarios require the location information of objects, such as robot welding and grasping. Taking the grasping field as an example, the robot needs to determine the position of the object to be grasped before it can perform the object grasping operation.

[0055] Then, how to realize object positioning is a technical problem that technical personnel in this field need to solve urgently.

[0056] To this end, an embodiment of the present invention places a laser galvanometer stereo camera and a robot on the same workbench, and adopts an eye-in-hand approach (i.e., the camera is arranged at the end of the robotic arm) to realize functions such as collection, stitching, template matching, measurement, and object positioning of three-dimensional 3D visual information. The system can be used as a common technical system and carried in multiple robot operation scenes to realize object positioning in different scenes. Among them, the more commonly used fields can be application scenarios with a small field of view such as 3C or scenes with a large field of view.

[0057] Based on the above content, an embodiment of the present invention provides an object positioning method, and the execution subject can be an electronic device such as an industrial computer and a server. The electronic device can control the opening and closing of the camera. Among them, most scenes require the depth information of the object, so the camera used in the embodiment of the present invention is a laser galvanometer stereo camera that can collect depth information.

[0058] Reference Figure 1 , Figure 1 A schematic diagram of the installation scenario of the laser galvanometer stereo camera is given, which is also a schematic diagram of the object positioning system. Figure 1 In the figure, the laser galvanometer stereo camera 2 is installed on the motion device through the adapter structure 1. In one embodiment of the present invention, the motion device can be a robot 3 of the scene, and the robot 3 is installed on a robot base 4.

[0059] The robot 3 is generally a mechanical arm, and the position of the camera can be adjusted by adjusting the rotation direction and angle of the mechanical arm, thereby adjusting the shooting angle of the camera to shoot different areas.

[0060] In another implementation of the present invention, the structure of the switching structure 1 can refer to Figure 2 shown. Figure 2 In the embodiment, the upper end of the adapter structure 1 is connected to the robot 3, which can be connected by a threaded connector, such as a bolt and a nut. The lower end of the adapter structure 1 is connected to the camera. The adapter structure and the camera, such as the laser galvanometer stereo camera 2, are connected by a threaded connector. The threaded connector includes a bolt and a nut.

[0061] In an implementation of the present invention, the installation diagram of the camera and the adapter structure 1 can refer to Figure 3 shown.

[0062] Figure 3 The first and second pictures in the figure are schematic diagrams of the camera structure. The camera width is 101, the camera length is 538, and the camera height is 117.8.

[0063] The third and fourth pictures are schematic diagrams of the installation of the camera and the adapter structure 1. 51.5 is the distance between the electrical interface and the installation surface, and 31.37 is the distance between the two electrical interfaces. The fourth picture shows the dimensions of the installation surface, including 4 6mm ( ) diameter bolt holes, 2 5mm ( ) diameter through hole, 37.5 is the distance from the center of the through hole to the top of the camera, 51 is the distance between the two threaded holes in the width direction, 140 is the distance between the two threaded holes in the length direction, 100 is the distance between the two through holes, 30 is the distance from the threaded hole to the outer edge of the side, and 10 is the distance from the threaded hole to the inner edge of the side.

[0064] It should be noted that Figure 3 It is only an installation diagram of the camera and the adapter structure 1. In actual scenarios, due to different camera types or camera manufacturers, the camera sizes are different, so when the camera and the adapter structure 1 are installed, the positions of the bolts and nuts are set differently, which can be determined according to the actual configuration.

[0065] In this embodiment, the adapter structure and the laser galvanometer stereo camera mounting surface are connected using bolts and nuts, which has a simple structure and is easy to install. The installation position can be adjusted by connecting different screw holes to meet the installation requirements of different scenarios.

[0066] In another implementation of the present invention, the camera can be controlled by the electronic device to perform operations such as image acquisition, such as Figure 4 shown. Figure 4 A schematic diagram of software for controlling a camera in an electronic device to take photos.

[0067] The user clicks the Find Device button on the operation interface of the electronic device, and then all cameras connected to the electronic device can be found. The user selects the current camera for image acquisition in the drop-down list of cameras, clicks Open Device, and the electronic device can control the camera. Figure 4 The left area of ​​​​the camera will start to display the depth image taken by the camera in real time. If the displayed image meets the user's needs, the user can click the Save TIFF (Tag Image File Format) button. At this time, the current image will be captured and saved in TIFF format.

[0068] In another aspect of the present invention, if Figure 4The depth image displayed in the left area does not meet the user's needs. For example, if the shooting angle is not suitable, you can use the camera's built-in operating handle to adjust the horizontal position and vertical height of the camera. The operating handle is equipped with buttons for adjusting the vertical height, horizontal position, and operating mode. Through this button, the user can adjust the position of the robot's arm. The robot arm drives the camera to move, thereby adjusting the position of the camera to adjust the image captured by the camera. The robot's current position, mode, coordinates and other information will be displayed on the display interface of the operating handle, such as Figure 5 As shown, when the user operates the handle, the content displayed on the display interface of the operating handle will change accordingly.

[0069] The images captured by the camera can be displayed on Figure 4 The left area, so users can observe Figure 4 The image in the left area can be used to adjust the robot position. Figure 4 When the image in the left area meets the user's needs, you can stop adjusting the robot position and click the Save TIFF button to complete the image acquisition operation.

[0070] After the current image acquisition is completed, the robot position can be further adjusted to perform the next image acquisition operation until all the required images are acquired and the process stops.

[0071] In another implementation of the present invention, the image captured by the camera can be sent to the electronic device so that the electronic device can perform Figure 4 The positioning operation in image processing in , through which the spatial position of an object can be determined, such as (x, y, z), where x represents the left and right space, y represents the front and back space, and z represents the top and bottom space.

[0072] Based on the above content, an embodiment of the present invention provides an object positioning method, referring to Figure 6 , which may include:

[0073] S11. Collect candidate depth images.

[0074] The candidate depth image includes at least one depth sub-image of a candidate object.

[0075] In actual scenarios, the application scenarios of the embodiments of the present invention can be rail transportation, depalletizing, etc. Taking depalletizing as an example, depalletizing refers to the process of taking out items from a packaging box or a pallet and restacking them. In this process, it is necessary to determine which object needs to be taken out and the specific location of the object, so as to take the object according to the location and restack it.

[0076] In the depalletizing scene, a large number of objects are stacked together, and it is necessary to determine the object to be taken from them. In the embodiment of the present invention, the object to be taken is determined by image matching.

[0077] In order to achieve image matching, it is necessary to collect images of objects in the depalletizing scene. To this end, in an embodiment of the present invention, the above-mentioned camera, such as a laser galvanometer stereo camera, is used to collect depth images of objects in the depalletizing scene. The image collected by the laser galvanometer stereo camera can be called a candidate depth image.

[0078] In one implementation, the candidate depth image includes at least one depth sub-image of a candidate object. For example, if there are ten candidate objects in total and one needs to be selected, images of the ten candidate objects can be collected. For example, during the collection, the laser galvanometer stereo camera can be adjusted to a certain height so that the camera's field of view can cover all candidate objects. Then, the user clicks the Save TIFF button, and the camera can take pictures of the ten candidate objects to obtain the candidate depth image.

[0079] In this embodiment, if the area occupied by the candidate objects is small, a candidate depth image including depth sub-images of all candidate objects can be acquired through a single image acquisition, wherein the candidate depth image includes a depth sub-image of at least one candidate object, and in a specific scenario, may include a depth sub-image of each candidate object.

[0080] However, in actual scenes, there is another scene, that is, the area occupied by the candidate object is larger, and the camera is generally located within the range of 1.5m to 3m from the object to be measured in the vertical direction. When the relative distance in the vertical direction is 1.5m, the horizontal field of view of the camera is 1200mm*1400mm, and when the relative distance in the vertical direction is 3m, the horizontal field of view of the camera is 2700mm*3300mm. If the distance between the camera and the object to be measured, that is, the above-mentioned candidate object, is adjusted to 3m, it is still not guaranteed that all candidate objects will be adjusted to the horizontal field of view of the camera. At this time, in another implementation of the present invention, multiple initial depth images including partial candidate area images can be obtained by performing multiple image acquisitions on the candidate objects, and then the initial depth images are spliced ​​to obtain candidate depth images.

[0081] S12: Obtain a reference depth image.

[0082] The reference depth image is a depth image of the target object.

[0083] Taking the target object as a sensor valve of a certain type of railway freight car as an example, the above-mentioned laser galvanometer stereo camera is used to collect the depth image of the target object to obtain a reference depth image. The schematic diagram of the reference depth image can be shown as follows Figure 7As shown, the reference depth image may be subsequently matched with the above-mentioned depth sub-image to determine the position of the target object from a large number of candidate objects.

[0084] S13: performing a matching operation on the candidate depth image and the reference depth image to select a depth sub-image that matches the reference depth image from the candidate depth image.

[0085] In this embodiment, since the structures of different objects are different, their corresponding depth images are also different. Therefore, if the depth sub-image in the candidate depth image has a high degree of match with the reference depth image, it means that the object corresponding to the depth sub-image is most likely the same object as the object in the embodiment of the present invention, and the depth sub-image that matches the reference depth image can be determined in the candidate depth image.

[0086] S14: Taking the candidate object corresponding to the depth sub-image as a target object, and determining a spatial position of the target object.

[0087] In this embodiment, after the matched depth sub-image is determined, the candidate object corresponding to the depth sub-image is used as the target object. For example, taking the target object as a sensor valve of a certain model of railway freight car as an example, its reference depth image is as follows: Figure 7 As shown in the example, in the scenario, it is necessary to select a certain type of sensor valve of the railway freight car from ten sensor valves. Then, the ten sensors can be imaged, and then image matching operations can be performed to finally match four sensor valves. The schematic diagram can be as follows: Figure 8 As shown, these four sensor valves are the sensor valves of a certain model of railway freight car being searched for.

[0088] After determining the sensor valves of a certain type of railway freight car, it is necessary to determine the specific positions of the four sensor valves. The specific position refers to the spatial position, such as (x, y, z). Subsequently, the specific sensor valve can be uniquely located based on the position, and operations such as grabbing the sensor valve can be performed.

[0089] In this embodiment, a candidate depth image including at least one depth sub-image of a candidate object can be used to perform a matching operation with a reference depth image of the target object to screen out a depth sub-image that matches the reference depth image, and subsequently use the candidate object corresponding to the depth sub-image as the target object. That is, the present invention first determines a candidate object that matches the depth image of the target object through image matching, and the candidate object is the target object. Subsequently, the spatial position of the target object is determined to achieve object positioning, and then the target object is quickly found through the spatial position, and then subsequent operations such as grasping are performed.

[0090] The above embodiment mentions collecting candidate depth images. In another implementation of the present invention, a specific implementation of collecting candidate depth images is provided. Fig. 9 , which may include:

[0091] S21. Collect a plurality of initial depth images respectively using cameras at the same height and at different horizontal positions.

[0092] There is a partial overlap between at least two initial depth images.

[0093] In actual scenes, the area occupied by all candidate objects may be large or small. For candidate objects with a small area, a candidate depth image including depth sub-images of all candidate objects can be acquired through a single image acquisition. At this time, the camera can be adjusted to a suitable height by controlling the robot's operating handle, and then the candidate object can be photographed to obtain the candidate depth image.

[0094] For all candidate objects with a large area, candidate depth images can be obtained by taking multiple photos and stitching the images. In one implementation of the present invention, it can be:

[0095] After the camera is adjusted to a specified height, an initial depth image is acquired by using the camera, the position of the camera is adjusted in the horizontal direction, so as to acquire a new initial depth image again by using the camera, and the step of adjusting the position of the camera in the horizontal direction is returned to be executed until image acquisition is completed for all candidate objects and then stops.

[0096] Specifically, firstly, the height of the camera is adjusted to a specified height between 1.5 m and 3 m by using an operating handle, and then the camera is used to take pictures of some candidate objects to acquire an initial depth image.

[0097] The process of camera image acquisition is as follows: Fig.10 As shown, the specific implementation is:

[0098] For cameras using a multi-line laser binocular system, a single measurement can only reconstruct the point cloud of the multi-line laser projection area. Therefore, it is necessary to combine galvanometer scanning to deflect the motion of the multi-line laser, measure the position one by one, and finally achieve the output of the full-frame point cloud / depth map. The reconstruction process of the laser galvanometer is as follows: Fig.10 As shown in the figure, the measurement process mainly includes polar line correction, laser line extraction, binocular laser line matching, and depth map calculation. After the galvanometer movement is completed, the full-frame point cloud / depth map is output. The schematic diagram of the full-frame point cloud / depth map is shown in Figure 7 shown.

[0099] In actual scenarios, such as Figure 4As shown, on the display interface of the electronic device, click Save TIFF, and the camera can Fig.10 The steps described above perform image acquisition to obtain an initial depth image.

[0100] After acquiring an initial depth image, the position of the camera needs to be moved horizontally. For example, for some candidate objects, the image of the left area of ​​the candidate object is first acquired to obtain an initial depth image.

[0101] After the image acquisition of the left area is completed, the camera can be moved horizontally so that the camera acquires the image of the right area of ​​the candidate object. It should be noted that, in the embodiment of the present invention, since the areas occupied by all candidate objects are photographed separately and the final candidate depth image is obtained by subsequent image stitching, the images captured by the camera need to have partial overlap. For example, when photographing the left area and the right area of ​​the candidate object respectively, the initial depth images obtained by the two photographs need to have partial overlap so that the overlapping part can be used for image stitching operation.

[0102] Therefore, in the embodiment of the present invention, the position of the camera is adjusted in the horizontal direction to acquire a new initial depth image using the camera again. If the images of all candidate objects have been acquired at this time, the image acquisition operation can be stopped. If the area occupied by all candidate objects is large and needs to be divided into the left, middle and right parts for image acquisition respectively, the next image acquisition operation can be continued.

[0103] In an implementation of the present invention, in an actual scene, if the height of the measured object is too high or too low and exceeds the depth of field of the camera, the robot can drive the camera to take pictures at different heights, increasing the range of the camera height, so as to obtain images of all candidate objects through two or more pictures. For example, if the area occupied by all candidate objects is large, the camera height can be raised to 3m, at which time the horizontal field of view of the camera becomes larger, and more candidate objects can be photographed, so that images of all candidate objects can be obtained through one or two pictures.

[0104] If images of all candidate objects cannot be obtained by taking two photos, images of all candidate objects can be obtained by taking three or more photos. However, when taking two photos, it is necessary to ensure that there is partial overlap between at least two initial depth images. For example, if the initial depth images obtained by the first and second photos partially overlap, and the initial depth images obtained by the second and third photos partially overlap, this ensures that the overlapping parts can be used for subsequent image stitching operations.

[0105] S22: Extract feature points of the initial depth image.

[0106] In an embodiment of the present invention, after obtaining multiple initial depth images, it is necessary to perform a stitching operation on them to obtain a complete candidate depth image including all candidate objects. When stitching images, a feature matching method can be used. In order to achieve feature matching, it is necessary to perform a feature point extraction operation on each initial depth image. Feature point extraction can be achieved using a feature point extraction algorithm.

[0107] S23: Using the feature points, perform a stitching operation on the initial depth image to obtain a candidate depth image.

[0108] In this embodiment, feature point matching can be achieved through feature matching, so that the initial depth image can be stitched using the matched feature points, and finally a complete candidate depth image including all candidate objects can be obtained.

[0109] In this embodiment, when the area occupied by all candidate objects may be large, multiple depth maps are obtained through multi-position acquisition, and the depth maps are stitched to obtain a complete depth map with a large field of view including all candidate objects, so as to realize subsequent matching operations.

[0110] After obtaining the candidate depth image through the above steps, a matching operation can be performed on the candidate depth image and the reference depth image to select a depth sub-image matching the reference depth image from the candidate depth image. Specifically, the matching operation includes the following steps:

[0111] 1) Using a preset template matching method, at different image resolutions, an image matching operation is performed on a depth sub-image in the candidate depth image and the reference depth image to obtain a similarity between the depth sub-image and the reference depth image.

[0112] The preset template matching method in the embodiment of the present invention may be a template matching method based on a pyramid principle, such as an image pyramid.

[0113] Image pyramid is an image representation method that constructs a series of images of different scales by continuously reducing the resolution of the image. Image pyramid is a collection of sub-images of different resolutions of the same image. It is generated by continuously downsampling the original image, that is, a low-resolution approximate image (small size) is generated from a high-resolution image (large size). This multi-scale representation not only helps to reduce the amount of calculation, but also enhances the robustness of matching. In template matching, image pyramid can search for targets at different scales, thereby coping with the problem of target scale changes.

[0114] In the implementation of the present invention, for the reference depth image, an image pyramid can be used to construct reference depth images of different image resolutions. For the depth sub-image in the candidate depth image, according to its image resolution, a reference depth image that matches the image resolution of the depth sub-image is selected from the reference depth images of different image resolutions, and an image matching operation is performed on the two to determine the similarity between each depth sub-image in the candidate depth image and the reference depth image.

[0115] 2) Filtering depth sub-images whose similarity with the reference depth image is greater than a similarity threshold.

[0116] Specifically, if the similarity between the reference depth image and the depth sub-image is higher, it means that the structures of the actual objects corresponding to the two are more similar, and the probability that the two are the same object is higher. In this embodiment of the present invention, the depth sub-image whose similarity with the reference depth image is greater than the similarity threshold can be screened. Among them, the similarity threshold can be configured according to the actual situation, such as a fixed value or an adjustable value. If the similarity between the reference depth image and the depth sub-image is greater than the similarity threshold, it means that the two are most likely the same object, and the prompt information of determining the target object can be output.

[0117] In another implementation of the present invention, if a plurality of depth sub-images having a similarity with the reference depth image greater than the similarity threshold are found through the similarity threshold, it indicates that there are a plurality of target objects among all the candidate objects.

[0118] In another implementation of the present invention, after setting a similarity threshold, if there is no depth sub-image whose similarity with the reference depth image is greater than the similarity threshold, the threshold can be slightly lowered. If the threshold is lowered multiple times and a suitable depth sub-image still cannot be found, it means that the target object does not exist among all the candidate objects.

[0119] In this embodiment, a template matching method based on the pyramid principle is used to find a depth sub-image whose similarity with the reference depth image is greater than a similarity threshold, and further the target object can be found from the candidate objects. The template matching method based on the pyramid principle can be used to match images from different image resolutions. By matching on images of different scales, the feature information at different scales can be comprehensively considered, thereby enhancing the robustness of the matching. In addition, the template matching method based on the pyramid principle can achieve dimensionality reduction operations, reduce the amount of calculation, and improve the matching speed. In addition, a suitable similarity threshold is set, so that even if the target object has a certain degree of deformation or occlusion in the image, a matching area can still be found at certain scales.

[0120] After obtaining a depth sub-image matching the reference depth image through the above embodiment, the candidate object corresponding to the depth sub-image can be used as the target object, and the spatial position of the target object can be determined. In another implementation of the present invention, a specific implementation of determining the spatial position of the target object is given. Specifically, the following steps are included:

[0121] 1) Determine the image position of the target object in the candidate depth image.

[0122] Specifically, since a depth sub-image matching the reference depth image has been determined from the candidate depth image, the image position of the depth sub-image in the candidate depth image can be determined, and the image position is the image position of the target object in the candidate depth image.

[0123] 2) Using the mapping relationship between the image position and the spatial position, determine the spatial position corresponding to the image position of the target object.

[0124] In a specific implementation of the present invention, a certain position in the camera imaging corresponds one-to-one to a position in the actual three-dimensional space, and the corresponding relationship can be realized by a mapping relationship between the image position and the spatial position. Therefore, for a certain position in the image, the mapping relationship can be used to determine the spatial position corresponding to the image position of the target object, that is, to uniquely determine one (x, y, z).

[0125] In actual scenarios, such as Figure 4 As shown in the figure, the object positioning operation process is as follows:

[0126] 1) Click the Find Device button;

[0127] 2) Select the current camera in the drop-down list;

[0128] 3) Click to open the device;

[0129] 4) Click Start Acquisition, and the real-time depth image will be displayed on the left side of the interface;

[0130] 5) Move the camera to the photo taking position and click Save TIFF;

[0131] 6) If you need to expand the shooting range, move the robot to drive the camera to the next shooting point and continue to click Save TIFF;

[0132] 7) After taking pictures of the desired location, click the positioning button to stitch multiple pictures together through feature point extraction and obtain the target location through template matching.

[0133] In this embodiment, the specific position of the target object in the three-dimensional space is obtained through the mapping relationship between the image position and the spatial position, and the target object can be quickly located through the position, and operations such as grabbing the target object can be performed.

[0134] In another implementation of the present invention, after taking the candidate object corresponding to the depth sub-image as the target object, the method further includes:

[0135] 1) Receive a height query instruction of a target point of the target object.

[0136] Specifically, Fig.11 As shown, for the image of the target object finally obtained, if the user wants to check the distance from a certain point on the target object to the ground, the user can place the pointer on the point.

[0137] like Fig.11 As shown, if the user wants to measure the side ear of the valve, the user can place the pointer on the side ear, and the electronic device will receive a height query instruction of the target point of the target object.

[0138] 2) Responding to the altitude query instruction, determining the altitude information of the target point.

[0139] The height information is obtained by calculating the position where the laser is reflected to the photosensitive element and the installation angle between the laser and the photosensitive element through a trigonometric function relationship.

[0140] After receiving the height query command of the target point of the target object, the point will be automatically displayed, such as the height of the side ear. Fig.11 The height difference in the plane, that is, the height from the ground.

[0141] The height of each point in the target object is measured by the camera and transmitted to the electronic device. The camera's height measurement principle is laser triangulation, and the laser galvanometer stereo camera is built using two laser triangulation methods. The positional relationship of the laser reflected back to the CCD (Charge-coupled Device) photosensitive element and the installation angle between the laser and the CCD photosensitive element are used to obtain the height information of the measured object through trigonometric functions. This height information is the actual height of a certain point, with a unit of 0.1mm and an accuracy of within 1mm.

[0142] In this embodiment, the height of a certain point of an object can be obtained through a pointer to assist in locating the object.

[0143] In one implementation of the present invention, 3D vision is particularly widely used in application scenarios with a small field of view, such as 3C, and many scenarios require depth information of the object being measured, such as robot welding and grasping. For this scenario, the embodiment of the present invention can not only provide depth information of the object being measured, but also further provide information such as the position and height of the object. Therefore, the positioning method in the present invention can be used as a common technology in robot application scenarios in different industries, such as rail transportation, depalletizing, etc. In addition, the present invention realizes the collection and processing of three-dimensional information within a large field of view through image stitching, which can be directly applied to the development of new products with related needs.

[0144] Based on the above-mentioned embodiment of the object positioning method, another embodiment of the present invention provides an object positioning device, referring to Fig.12 , which may include:

[0145] An image acquisition module 11 is used to acquire a candidate depth image; the candidate depth image includes at least one depth sub-image of a candidate object;

[0146] The image acquisition module 12 is used to acquire a reference depth image; the reference depth image is a depth image of the target object;

[0147] A matching module 13, configured to perform a matching operation on the candidate depth image and the reference depth image, so as to select a depth sub-image matching the reference depth image from the candidate depth image;

[0148] The position determination module 14 is configured to take the candidate object corresponding to the depth sub-image as a target object and determine the spatial position of the target object.

[0149] In one implementation, the image acquisition module 11 includes:

[0150] The image acquisition submodule is used to respectively acquire a plurality of initial depth images using cameras at the same height and at different horizontal positions; there is a partial overlap between at least two initial depth images;

[0151] A feature point extraction submodule, used to extract feature points of the initial depth image;

[0152] The stitching submodule is used to perform a stitching operation on the initial depth image using the feature points to obtain a candidate depth image.

[0153] In one implementation, the image acquisition submodule includes:

[0154] A first image acquisition unit, configured to acquire an initial depth image using the camera after adjusting the camera to a specified height;

[0155] The second image acquisition unit is used to adjust the position of the camera in the horizontal direction to obtain a new initial depth image by using the camera again, and return to the step of adjusting the position of the camera in the horizontal direction until the image acquisition of all candidate objects is completed and stops.

[0156] In one implementation, the matching module 13 includes:

[0157] A matching submodule, configured to perform an image matching operation on a depth sub-image in the candidate depth image and the reference depth image at different image resolutions using a preset template matching method to obtain a similarity between the depth sub-image and the reference depth image;

[0158] The screening submodule is used to screen depth sub-images whose similarity with the reference depth image is greater than a similarity threshold.

[0159] In one implementation, the location determination module 14 is specifically configured to:

[0160] An image position of the target object in the candidate depth image is determined, and a spatial position corresponding to the image position of the target object is determined by using a mapping relationship between the image position and the spatial position.

[0161] In one implementation, the method further includes:

[0162] The height determination module is used to receive a height query instruction of a target point of the target object, respond to the height query instruction, and determine the height information of the target point; wherein the height information is calculated by using a trigonometric function relationship to calculate the position of the laser reflected to the photosensitive element and the installation angle of the laser and the photosensitive element.

[0163] In this embodiment, a candidate depth image including at least one depth sub-image of a candidate object can be used to perform a matching operation with a reference depth image of the target object to screen out a depth sub-image that matches the reference depth image, and subsequently use the candidate object corresponding to the depth sub-image as the target object. That is, the present invention first determines a candidate object that matches the depth image of the target object through image matching, and the candidate object is the target object. Subsequently, the spatial position of the target object is determined to achieve object positioning, and then the target object is quickly found through the spatial position, and then subsequent operations such as grasping are performed.

[0164] It should be noted that, for the working process of each module and sub-module in this embodiment, please refer to the corresponding description in the above embodiment, which will not be repeated here.

[0165] An embodiment of the present application also provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0166] The memory is used to store computer programs;

[0167] The processor is used to execute the computer program so that the electronic device can implement the above-mentioned object positioning method.

[0168] The embodiment of the present application also provides an object positioning system, including a camera, a motion device and the above-mentioned electronic device; the camera is installed on the motion device through a switching structure;

[0169] The motion device is used to respond to the position adjustment instruction and drive the camera to move according to the position adjustment instruction. The position adjustment instruction is an instruction output to the motion device by the user through the control of the operating handle.

[0170] The camera is used to collect a candidate depth image and send the candidate depth image to the electronic device.

[0171] Wherein, the camera comprises a laser galvanometer stereo camera; the motion device is a robot; the adapter structure and the mounting surface of the laser galvanometer stereo camera are connected by a threaded connector.

[0172] The structure of the object positioning system is specifically referenced Figure 1 For the specific implementation process, please refer to the above corresponding instructions.

[0173] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any object positioning method provided in the embodiment of the present application.

[0174] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any object positioning method provided in the embodiment of the present application.

[0175] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for locating an object, characterized in that: include: Acquire a candidate depth image; the candidate depth image includes at least one depth sub-image of a candidate object; Obtain a reference depth image; The reference depth image is a depth image of the target object; Performing a matching operation on the candidate depth image and the reference depth image to select a depth sub-image matching the reference depth image from the candidate depth image; The candidate object corresponding to the depth sub-image is taken as a target object, and the spatial position of the target object is determined.

2. The object positioning method according to claim 1, characterized in that: Collect candidate depth images, including: Using cameras at the same height and at different horizontal positions to collect multiple initial depth images respectively; there is a partial overlap between at least two initial depth images; Extracting feature points of the initial depth image; The initial depth image is stitched using the feature points to obtain a candidate depth image.

3. The object positioning method according to claim 2, characterized in that: Multiple initial depth images are collected using cameras at the same height and at different horizontal positions, including: After adjusting the camera to a specified height, acquiring an initial depth image using the camera; The position of the camera is adjusted in the horizontal direction to acquire a new initial depth image by using the camera again, and the step of adjusting the position of the camera in the horizontal direction is returned to be executed until image acquisition of all candidate objects is completed and then stops.

4. The object positioning method according to claim 1, characterized in that: Performing a matching operation on the candidate depth image and the reference depth image to select a depth sub-image matching the reference depth image from the candidate depth image, including: Using a preset template matching method, at different image resolutions, an image matching operation is performed on the depth sub-image in the candidate depth image and the reference depth image to obtain a similarity between the depth sub-image and the reference depth image; A depth sub-image having a similarity with the reference depth image greater than a similarity threshold is selected.

5. The object positioning method according to claim 1, characterized in that: Determining the location information of the target object includes: Determining an image position of the target object in the candidate depth image; The spatial position corresponding to the image position of the target object is determined by using the mapping relationship between the image position and the spatial position.

6. The object positioning method according to claim 1, characterized in that: After taking the candidate object corresponding to the depth sub-image as the target object, the method further includes: Receiving a height query instruction of a target point of the target object; In response to the height query instruction, the height information of the target point is determined; wherein the height information is calculated by using a trigonometric function relationship to calculate the position where the laser is reflected to the photosensitive element and the installation angle of the laser and the photosensitive element.

7. An object positioning device, characterized in that: include: An image acquisition module, configured to acquire a candidate depth image; the candidate depth image includes at least one depth sub-image of a candidate object; An image acquisition module is used to acquire a reference depth image; the reference depth image is a depth image of a target object; A matching module, configured to perform a matching operation on the candidate depth image and the reference depth image, so as to select a depth sub-image matching the reference depth image from the candidate depth image; The position determination module is used to take the candidate object corresponding to the depth sub-image as the target object and determine the spatial position of the target object.

8. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the object positioning method as described in any one of claims 1 to 6.

9. An object positioning system, characterized in that: The electronic device comprises a camera, a sports device and the electronic device as claimed in claim 8; the camera is mounted on the sports device via a switching structure; The motion device is used to respond to the position adjustment instruction and drive the camera to move according to the position adjustment instruction; The camera is used to collect a candidate depth image and send the candidate depth image to the electronic device.

10. The object positioning system according to claim 9, characterized in that: The camera comprises a laser galvanometer stereo camera; the motion device is a robot; the adapter structure and the mounting surface of the laser galvanometer stereo camera are connected by a threaded connector.

Images