Object positioning method, device, system, readable storage medium and program product

Abstract

The application relates to an object positioning method, device, system, readable storage medium and program product. The method comprises the following steps: acquiring an object angle range and an object edge direction vector of a target object; extracting at least one candidate corner point and at least one candidate edge line segment combination from a shooting image corresponding to the target object; extracting at least one candidate geometric feature combination from all candidate corner points and all candidate edge line segment combinations according to the object angle range and the object edge direction vector; screening a target object shape from a candidate object shape corresponding to each candidate geometric feature combination, and positioning the target object according to the target object shape. The method can accurately position the object.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent manufacturing, and in particular to an object positioning method, device, system, readable storage medium and program product. BACKGROUND

[0002] In the field of intelligent manufacturing, the object positioning method is a core function relied on by multiple production line processes such as automatic sorting of materials, size measurement, and quality detection. Therefore, positioning of objects such as materials is particularly important in industrial automation production lines.

[0003] The existing object positioning method mainly includes a deep learning method of automatically learning multi-scale features using a convolutional neural network, and a traditional algorithm of designing features to describe textures and edges. Taking the deep learning method as an example, specifically, a convolutional neural network is trained through a certain number of training images and manually labeled object positioning results in the training images, and then the object in the photographed image is positioned according to the trained convolutional neural network; taking the traditional algorithm as an example, it mainly includes Hough transform detection, that is, transforming the photographed image into a parameter space to find possible edge line segment features, and then positioning the object in the photographed image according to the found edge line segment features.

[0004] However, the positioning effect of the current object positioning method is still not accurate enough. SUMMARY

[0005] Therefore, it is necessary to provide an accurate object positioning method, device, system, computer readable storage medium and computer program product in view of the above technical problems.

[0006] In a first aspect, the present application provides an object positioning method, comprising:

[0007] obtaining an object angle range and an object edge direction vector of a target object;

[0008] extracting at least one candidate corner point and at least one candidate edge line segment combination from a photographed image corresponding to the target object;

[0009] extracting at least one candidate geometric feature combination from all candidate corner points and all candidate edge line segment combinations according to the object angle range and the object edge direction vector;

[0010] filtering a target object shape from a candidate object shape corresponding to each candidate geometric feature combination, and positioning the target object according to the target object shape.

[0011] In one of the embodiments, the candidate corner points are multiple; extracting at least one candidate corner point and at least one candidate edge line segment combination from the photographed image corresponding to the target object comprises:

[0012] extracting at least one first candidate corner point and at least one candidate edge line segment from the photographed image corresponding to the target object;

[0013] screening at least one candidate edge line segment combination from the at least one candidate edge line segment according to the object angle range and the object edge direction vector, and determining a second candidate corner point corresponding to each candidate edge line segment combination;

[0014] obtaining a plurality of candidate corner points in combination of the first candidate corner point and the second candidate corner point.

[0015] In one embodiment, the extracting of the at least one candidate corner point and the at least one candidate edge line segment combination from the photographed image corresponding to the target object comprises:

[0016] segmenting the photographed image corresponding to the target object into single-channel images in different color spaces;

[0017] extracting the at least one candidate corner point and the at least one candidate edge line segment combination from each single-channel image.

[0018] In one embodiment, the screening of the target object shape from each candidate geometric feature combination comprises:

[0019] obtaining at least one candidate object shape corresponding to each candidate geometric feature combination respectively;

[0020] detecting a similarity result between each candidate object shape and a preset standard object shape of the target object respectively;

[0021] screening the target object shape from the at least one candidate object shape according to the at least one similarity result.

[0022] In one embodiment, the candidate geometric feature combination at least comprises a target corner point, and the obtaining of the at least one candidate object shape corresponding to each candidate geometric feature combination respectively comprises:

[0023] for each candidate geometric feature combination, determining the target corner point as a reference corner of the target object, and obtaining at least one to-be-matched angle range and at least one to-be-matched side length range of the target object, wherein the to-be-matched angle range is an angle range of a to-be-matched corner of the target object except the reference corner;

[0024] selecting, according to the at least one to-be-matched angle range and the at least one to-be-matched side length range, a to-be-matched geometric feature combination corresponding to each to-be-matched corner from the remaining candidate geometric feature combinations;

[0025] constructing at least one candidate object shape of the target object according to the candidate geometric feature combination and the to-be-matched geometric feature combination.

[0026] In one embodiment, detecting the similarity result between the shape of each candidate object and the preset standard object shape of the target object includes:

[0027] For each candidate object shape, detect the size and angle differences between the candidate object shape and the preset standard object shape of the target object;

[0028] Based on the size and angle differences of each candidate object shape, the similarity between each candidate object shape and the preset standard object shape is detected.

[0029] Secondly, this application also provides an object positioning device, the device comprising:

[0030] The information acquisition module is used to acquire the target object's angle range and the object's side direction vector;

[0031] The corner and line segment detection module is used to extract at least one candidate corner and at least one candidate edge line segment combination from the captured image corresponding to the target object.

[0032] The candidate geometric feature combination detection module is used to extract at least one candidate geometric feature combination from all candidate corner points and all candidate edge line segment combinations based on the object's angle range and the object's edge direction vector.

[0033] The object localization module is used to filter the target object shape from the candidate object shapes corresponding to each candidate geometric feature combination, and to locate the target object based on the target object shape.

[0034] Thirdly, this application also provides an object positioning system, which includes a visual inspection device and a processor;

[0035] The visual inspection device is used to capture images of the target object and send the captured images to the processor;

[0036] The processor is used for:

[0037] Obtain the target object's angle range and edge direction vector;

[0038] Extract at least one candidate corner point and at least one candidate edge line segment combination from the captured image corresponding to the target object;

[0039] Based on the object's angle range and the object's edge direction vector, extract at least one candidate geometric feature combination from all candidate corner points and all candidate edge line segment combinations;

[0040] The target object shape is selected from the candidate object shapes corresponding to each candidate geometric feature combination, and the target object is located based on the target object shape.

[0041] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0042] Obtain the target object's angle range and edge direction vector;

[0043] Extract at least one candidate corner point and at least one candidate edge line segment combination from the captured image corresponding to the target object;

[0044] Based on the object's angle range and the object's edge direction vector, extract at least one candidate geometric feature combination from all candidate corner points and all candidate edge line segment combinations;

[0045] The target object shape is selected from the candidate object shapes corresponding to each candidate geometric feature combination, and the target object is located based on the target object shape.

[0046] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0047] Obtain the target object's angle range and edge direction vector;

[0048] Extract at least one candidate corner point and at least one candidate edge line segment combination from the captured image corresponding to the target object;

[0049] Based on the object's angle range and the object's edge direction vector, extract at least one candidate geometric feature combination from all candidate corner points and all candidate edge line segment combinations;

[0050] The target object shape is selected from the candidate object shapes corresponding to each candidate geometric feature combination, and the target object is located based on the target object shape.

[0051] The aforementioned object localization method, apparatus, system, computer-readable storage medium, and computer program product replace common single search features such as point sets or line sets by pre-configuring the object's angle range and object edge direction vector. That is, by introducing more information, including direction vectors and angles, it accurately searches for at least one candidate geometric feature combination from at least one candidate corner point and at least one candidate edge line segment combination in the captured image. This candidate geometric feature combination extraction method can accurately extract even complex objects. Finally, it further selects more suitable target object shapes from the candidate object shapes corresponding to each candidate geometric feature combination, and locates the target object according to the target object shape. The entire object localization process is very accurate. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a diagram illustrating the application environment of an object localization method in one embodiment.

[0054] Figure 2 This is a flowchart illustrating an object localization method in one embodiment;

[0055] Figure 3 This is a flowchart illustrating the object localization method in another embodiment;

[0056] Figure 4 This is a schematic diagram of an L-shaped detector in one embodiment;

[0057] Figure 5 This is a structural block diagram of an object positioning device in one embodiment;

[0058] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0060] In the field of intelligent manufacturing, object positioning methods are a core function upon which multiple production line processes, such as automated material sorting, dimensional measurement, and quality inspection, rely. Therefore, positioning materials and other objects is particularly important in industrial automated production lines.

[0061] Existing object localization methods mainly include deep learning methods that use convolutional neural networks to automatically learn multi-scale features, and traditional algorithms that design features to describe textures and edges. Taking deep learning methods as an example, specifically, a convolutional neural network is trained using a certain number of training images and manually labeled object localization results in the training images. Then, based on the trained convolutional neural network, objects in the captured image are located. Taking traditional algorithms as an example, they mainly include Hough transform detection, which transforms the captured image into a parameter space to find possible edge line segment features, and then locates objects in the captured image based on the found edge line segment features.

[0062] However, while deep learning methods can record more complex edge features, they rely on a certain number of training images and manually labeled object localization results within those images. If errors occur in the training input data, the accuracy of the object localization method will decrease. In contrast, traditional algorithms have limitations when dealing with complex objects, such as those that cannot be described by a single feature.

[0063] Therefore, this application provides an object localization method that replaces common single search features such as point sets or line sets by pre-configuring the object's angle range and edge direction vector. That is, by introducing more information, including direction vectors and angles, it accurately searches for at least one candidate geometric feature combination from at least one candidate corner point and at least one candidate edge line segment combination in the captured image. This candidate geometric feature combination extraction method can accurately extract even complex objects. Finally, it further selects more suitable target object shapes from the candidate object shapes corresponding to each candidate geometric feature combination, and locates the target object according to the target object shape. The entire object localization process is very accurate.

[0064] The object positioning method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with visual inspection device 104 and processor 106 via a network, and visual inspection device 104 can communicate with processor 106 via the network. A data storage system can store the data that processor 106 needs to process.

[0065] When a user triggers an object image detection control for a target object on the object localization interface of terminal 102, terminal 102 responds to the trigger request of the object image detection control, generates an object image detection request for the target object, and sends the object image detection request to vision detection device 104. Vision detection device 104 detects the captured image of the target object and sends it to processor 106 to be stored in the database of processor 106, so that processor 106 can subsequently retrieve the captured image of the target object from the database and execute the following object localization steps.

[0066] Alternatively, the user triggers an object positioning control for the target object on the object positioning interface of terminal 102. Terminal 102 responds to the trigger request of the object positioning control, generates an object positioning request for the target object, and sends the object positioning request to processor 106. Processor 106 controls vision inspection device 104 to detect the captured image of the target object and feeds it back to processor 106, so that processor 106 can perform the following object positioning steps based on the captured image of the target object.

[0067] Object localization steps: The processor 106 obtains the object angle range and the object edge direction vector of the target object; extracts at least one candidate corner point and at least one candidate edge line segment combination from the captured image corresponding to the target object; extracts at least one candidate geometric feature combination from all candidate corner points and all candidate edge line segment combinations based on the object angle range and the object edge direction vector; filters the target object shape from the candidate object shapes corresponding to each candidate geometric feature combination, and locates the target object based on the target object shape.

[0068] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. The processor 106 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0069] In one exemplary embodiment, such as Figure 2 As shown, an object localization method is provided, which can be applied to... Figure 1 The processor 106 in the example will be used for illustration. Wherein:

[0070] S200: Obtain the target object's angle range and the object's side direction vector.

[0071] The target object can be an object composed of one or more simple structures, such as a rectangular object, a triangular object, or a combination of rectangles and triangles, without limitation. The object angle range and the object side direction vectors are both geometric feature description information of the target object. The object angle range is determined by combining the object angle with the allowable tolerance of the object angle. Each object angle range corresponds to at least one object side direction vector. Each type of geometric feature description information of the target object can include at least one object angle range and at least one object side direction vector corresponding to each object angle range. There is more than one type of geometric feature description information of the target object. For example, the geometric feature description information of the target object can be: obtuse angle and the two object side direction vectors corresponding to the obtuse angle, right angle and the two object side direction vectors corresponding to the right angle, triangle and the two object side direction vectors corresponding to each angle of the triangle, etc.

[0072] Specifically, after the processor acquires the captured image of the target object, it can also obtain a variety of pre-configured geometric feature description information from the database. Each geometric feature description information includes at least one object angle range of the target object, and at least one object side direction vector corresponding to each object angle range.

[0073] Among them, the object angle range of the target object refers to a range that is close to any angle of the target object, that is, the allowable tolerance range of that angle of the target object, and this tolerance range can be configured by the user; the object side direction vector refers to the object side direction vector associated with any object angle range, and there is at least one object side direction vector associated with each object angle range.

[0074] For example, if the target object is a rectangle with a 90° angle, then the corresponding angle range is 85°-95°. In this case, the direction vectors of the object's sides corresponding to this angle range can form two direction vectors of the object's sides with an angle of 85°-95°. Similarly, if the target object is a triangle with a 45° angle, then the corresponding angle range is 40°-50°. In this case, the direction vectors of the object's sides corresponding to this angle can form two direction vectors of the object's sides with an angle of 40°-50°.

[0075] In one embodiment, the object's side direction vector can be further replaced with the length of a directed line segment that matches the object's side direction vector, but the length of this line segment does not need to be the same as the side length of the target object. Subsequent detection of candidate geometric feature combinations is based on the target object's angular range and the direction vector corresponding to the length of the directed line segment that matches the object's side direction vector.

[0076] In one embodiment, specifically, when the target object is a rounded rectangle, the object angle range needs to be set to the tolerance range corresponding to 90°; when the target object is a circular or elliptical object, candidate geometric feature combinations can be extracted directly based on the object's side direction vector.

[0077] S400: Extract at least one candidate corner point and at least one candidate edge line segment combination from the captured image corresponding to the target object.

[0078] Corner points are points in a local area of ​​an image where the gray level or structure changes drastically. They are usually the intersection of object edges, corners of shapes, or abrupt changes in texture.

[0079] Specifically, multiple candidate corner points that may exist in the captured image are extracted, and multiple candidate edge segments that may exist in the captured image are extracted. Then, based on the multiple candidate edge segments, at least one associated combination of candidate edge segments is found. Each associated combination of candidate edge segments includes multiple candidate edge segments. The multiple candidate edge segments can form a certain angle between them, and the angle satisfies the object angle range in a certain geometric feature description information. Moreover, these candidate edge segments satisfy the object edge direction vector corresponding to the geometric feature description information.

[0080] In one embodiment, the method for extracting multiple candidate corner points in a captured image can be a corner detection method, including but not limited to: Shi-Tomasi corner detection method, Harris corner detection method, etc. The method for extracting multiple candidate edge line segments that may exist in the captured image includes but is not limited to: Hough transform algorithm, FastLineDetector algorithm, etc.

[0081] S600: Based on the object's angle range and the object's edge direction vector, extract at least one candidate geometric feature combination from all candidate corner points and all candidate edge line segment combinations.

[0082] Specifically, all candidate corner points and all candidate edge segment combinations in the captured images are summarized. Based on the object angle range and the object edge direction vector, candidate geometric feature combinations are detected from all candidate corner points and all candidate edge segment combinations. Each candidate geometric feature combination includes at least one candidate corner point and a candidate edge segment combination associated with each candidate corner point. The angle of each candidate corner point in the candidate geometric feature combination satisfies the corresponding object angle range, and the direction vector of the candidate edge segment in the candidate edge segment combination satisfies the corresponding object edge direction vector.

[0083] S800 filters the target object shape from the candidate object shapes corresponding to each candidate geometric feature combination, and locates the target object based on the target object shape.

[0084] Specifically, the detected candidate geometric feature combinations are all candidate geometric feature combinations that match both the object angle range and the object edge direction vector of the target object. Therefore, for each candidate geometric feature combination, other suitable candidate geometric feature combinations can be selected from all candidate geometric feature combinations, and the candidate geometric feature combination can be combined with other suitable candidate geometric feature combinations to jointly construct a candidate object shape that is close to the object shape of the target object.

[0085] After constructing the candidate object shape corresponding to each candidate geometric feature combination, the constructed candidate object shapes can be further filtered to obtain the target object shape. Then, based on the target object shape, the target object in the captured image can be detected, thereby accurately locating the target object in the actual scene.

[0086] In the above object localization method, the object angle range and object edge direction vector of the target object are pre-configured to replace common single search features such as point sets or line sets. That is, by introducing more information, including direction vectors and angles, at least one candidate geometric feature combination is accurately searched from at least one candidate corner point and at least one candidate edge line segment combination in the captured image. This candidate geometric feature combination extraction method can accurately extract even complex objects. Finally, more suitable target object shapes are further selected from the candidate object shapes corresponding to each candidate geometric feature combination, and the target object is located according to the target object shape. The entire object localization process is very accurate.

[0087] In one exemplary embodiment, such as Figure 3 As shown, there are multiple candidate corner points, S400, including:

[0088] S420, extract at least one first candidate corner point and at least one candidate edge segment from the captured image corresponding to the target object.

[0089] S440, based on the object's angle range and the object's edge direction vector, select at least one candidate edge segment combination from at least one candidate edge segment, and determine the second candidate corner point corresponding to each candidate edge segment combination.

[0090] S460 combines the first candidate corner point and the second candidate corner point to obtain multiple candidate corner points.

[0091] Among them, the first candidate corner point refers to a corner point with obvious corner point characteristics; the second candidate corner point refers to a corner point with indistinct corner point characteristics, such as the corner point of a rounded rectangle.

[0092] Specifically, the aforementioned corner detection method cannot accurately detect corners when dealing with objects such as rounded rectangles that do not have obvious corner features. In this case, candidate edge segments can be used as the main feature information for detection.

[0093] Therefore, after detecting at least one first candidate corner point with obvious corner features in the captured image using the corner detection method, at least one candidate edge segment is detected, and at least one candidate edge segment combination is selected from the at least one candidate edge segment based on the object angle range and the object edge direction vector. At this time, the second candidate corner point formed by each candidate edge segment combination is also used as supplementary information of the first candidate corner point. The candidate corner point obtained by combining the first candidate corner point and the second candidate corner point is more complete.

[0094] Taking a rectangular object as an example, the selection of at least one candidate edge segment combination that matches the object's angle range and the object's side direction vector can be as follows: any two candidate edge segments can match the corresponding object's side direction vector, and the included angle between these two candidate edge segments satisfies the object's angle range associated with the matched object's side direction vector. Then, these two candidate edge segments constitute a candidate edge segment combination.

[0095] In one embodiment, when it is also necessary to obtain the length of the directed line segment corresponding to the object's side direction vector, taking the target object as a rectangular object as an example, if the projection of the minimum distance between any two candidate edge segments onto the object's side direction vector corresponding to a certain line segment length is less than the length of that line segment, and the included angle between these two candidate edge segments satisfies the corresponding object angle range, then there is an intersection point between the two candidate edge segments or their extensions, such that the point-line segment feature between the two candidate edge segments and the intersection point satisfies the geometric feature description information. The intersection point at this time is recorded as the corresponding second candidate corner point, and the two candidate edge segments at this time are recorded as the candidate edge segment combination.

[0096] In one embodiment, before combining the first candidate corner point and the second candidate corner point to obtain multiple candidate corner points, it is necessary to remove duplicates from the first candidate corner point and the second candidate corner point so that the combined multiple candidate corner points are not repeated.

[0097] In the above embodiments, multiple first candidate corner points with obvious corner features are detected in the captured image by corner detection method, and multiple second candidate corner points without obvious corner features are obtained by line segment detection method, so as to achieve accurate detection of multiple candidate corner points in the captured image. In addition, multiple candidate edge line segment combinations in the captured image can also be detected, providing a basis for subsequent detection of candidate geometric feature combinations based on multiple candidate corner points and multiple candidate edge line segment combinations.

[0098] In an exemplary embodiment, extracting at least one candidate corner point and at least one candidate edge segment combination from the captured image corresponding to the target object includes:

[0099] The captured image of the target object is segmented into single-channel images in different color spaces; at least one candidate corner point and at least one candidate edge line segment combination are extracted from each single-channel image.

[0100] The captured image exists in a multi-channel color space. These spaces describe colors through combinations of values ​​from multiple channels, with different channels carrying different color semantics. Therefore, the essence of a multi-channel color space is to decompose color information using multiple independent "dimensions," with each channel responsible for describing a specific characteristic of the color.

[0101] Specifically, in the multi-channel color space of the captured image, since a certain image feature may not be obvious in one channel but is more obvious in other channels, the color space of the captured image can be determined first, and then the captured image can be divided into single-channel images in different color spaces. This allows each single-channel image to retain feature information containing a single color semantic. Then, feature extraction can be performed on each single-channel image to obtain at least one candidate corner point and at least one candidate edge line segment combination in each single-channel image. At this time, the candidate corner points and candidate edge line segment combinations obtained will be richer.

[0102] In one embodiment, if the captured image is an image in the RGB (Red-Green-Blue) multi-channel color space, the captured image is divided into single-channel images in different color spaces, which can be dividing the captured image into single-channel images in the R channel, G channel, and B channel respectively; if the captured image is an image in the HSV (Hue-Saturation-Value) multi-channel color space, the captured image is divided into single-channel images in different color spaces, which can be dividing the captured image into single-channel images in the H channel, S channel, and V channel respectively. This application does not limit the multi-channel type of color space, nor does it limit the type of multi-channel color space.

[0103] In the above embodiments, by segmenting the captured image into single-channel images in different color spaces, richer combinations of candidate corner points and candidate edge segments can be extracted, making subsequent object localization more accurate.

[0104] In an exemplary embodiment, filtering the target object shape from the candidate object shapes corresponding to each candidate geometric feature combination includes:

[0105] Obtain at least one candidate object shape corresponding to each candidate geometric feature combination; detect the similarity result between each candidate object shape and the preset standard object shape of the target object; and select the target object shape from at least one candidate object shape based on at least one similarity result.

[0106] Specifically, the angles of each corner of the target object and the side length associated with each corner are obtained, and the preset standard object shape of the target object is obtained based on the angles of each corner of the target object and the side length associated with each corner.

[0107] Then, at least one candidate object shape corresponding to each candidate geometric feature combination is obtained, the similarity result between each candidate object shape and the preset standard object shape of the target object is detected, and the target object shape with a similarity result exceeding the preset similarity threshold is determined from the similarity results corresponding to each candidate object shape.

[0108] Furthermore, the shape of the target object can be further filtered using other auxiliary features to obtain a more accurate shape. These other auxiliary features include, but are not limited to, object color and height.

[0109] In the above embodiments, by detecting the similarity between the shape of each candidate object and the preset standard object shape of the target object, the target object shape can be selected more accurately from multiple candidate object shapes, thereby improving the accuracy of locating the target object.

[0110] In an exemplary embodiment, the candidate geometric feature combination includes at least a target corner point; obtaining at least one candidate object shape corresponding to each candidate geometric feature combination includes:

[0111] For each candidate geometric feature combination, the target corner point is determined as the reference angle of the target object, and at least one range of angles to be matched and at least one range of side lengths to be matched are obtained for the target object, wherein the range of angles to be matched is the range of angles to be matched in the target object excluding the reference angle; based on the at least one range of angles to be matched and the at least one range of side lengths to be matched, the geometric feature combination to be matched corresponding to each angle to be matched is selected from the remaining candidate geometric feature combinations; based on the candidate geometric feature combinations and the geometric feature combinations to be matched, at least one candidate object shape of the target object is constructed.

[0112] Specifically, the candidate geometric feature combination includes at least the target corner point and the candidate edge line segment combination corresponding to each target corner point. For each candidate geometric feature combination, the target corner point can be determined as the reference angle of the target object. Since the target object has more than one corner, the other corners of the target object except the reference angle can be used as the corners to be matched. From the remaining candidate geometric feature combinations, multiple matching geometric feature combinations that can form an object shape close to the target object with the candidate geometric feature combination corresponding to the reference angle are found.

[0113] At this point, we can first obtain the range of matching angles corresponding to at least one matching angle of the target object, and the range of matching edge lengths for at least one matching edge associated with each matching angle. The at least one matching edge includes the matching edge between the matching angle itself and the reference angle, and may also include the matching edge between the matching angle and other matching angles. Then, based on the candidate geometric feature combination information corresponding to the reference angle, the range of at least one matching angle, and the range of at least one matching edge length, we select the matching geometric feature combination corresponding to each matching angle from the remaining candidate geometric feature combinations.

[0114] When there are multiple angles to be matched, first find the target corner point that satisfies the range of angles to be matched for any one of the remaining candidate geometric feature combinations. Then determine whether the side direction vector corresponding to the target corner point is set relative to the side direction vector corresponding to the reference angle, and whether the distance between the target corner point and the reference angle satisfies the range of the side lengths to be matched between the target corner point and the reference angle. If the conditions are met, it means that a certain geometric feature combination to be matched has been detected. The detected geometric feature combination to be matched is then used as a candidate geometric feature combination for a new reference angle. The above steps are repeated to find candidate geometric feature combinations that satisfy the range of angles to be matched for other angles to be matched, as well as the range of side lengths to be matched associated with each angle to be matched. These are then used as the geometric feature combinations to be matched for the new angle to be matched, until a candidate geometric feature combination that satisfies the conditions for the last angle to be matched is found. At this point, the target corner point in the candidate geometric feature combination that satisfies the conditions for the last angle to be matched is determined as the new reference angle. Then, the correctness of the geometric feature combination to be matched for each angle to be matched is verified by checking whether the other side direction vector corresponding to the new reference angle is set relative to the other side direction vector corresponding to the original reference angle, and whether the distance between the new reference angle and the original reference angle satisfies the range of side lengths to be matched between the new reference angle and the original reference angle.

[0115] For example, consider a rectangular object with a long side of 4 and a short side of 3, possessing four corners: angle a, angle b, angle c, and angle d. The sides associated with angle a are ab and ad, angle b is ab and ad, and angle a is ab and ad. If angle a is taken as the reference angle, then angles b, c, and d are the angles to be matched. For angle b, we obtain the range of angles to be matched corresponding to angle b, and the range of the lengths of the two sides ab and ad associated with angle b. We then match target corner point 1 with angle a, and select any target corner point 2 from the remaining candidate geometric feature combinations that matches the range of angles to be matched for angle b. We then determine whether the direction vector of one side corresponding to target corner point 1 is relative to the direction vector of one side corresponding to target corner point 2, and whether the distance between target corner point 1 and target corner point 2 satisfies the range of the lengths of the sides ab to be matched. If so, target corner point 2 is the target corner point to be matched for angle b. Next, from the remaining candidate geometric feature combinations, select any target corner point 3 that matches the range of angles to be matched for angle c. Determine whether the direction vector of the other side corresponding to target corner point 2 is relatively set to the direction vector of the other side corresponding to target corner point 3, and whether the distance between target corner point 2 and target corner point 3 satisfies the range of the side length to be matched for bc. Then, from the remaining candidate geometric feature combinations, select any target corner point 4 that matches the range of angles to be matched for angle d. Determine whether the direction vector of the other side corresponding to target corner point 3 is relatively set to the direction vector of the other side corresponding to target corner point 4, and whether the distance between target corner point 3 and target corner point 4 satisfies the range of the side length to be matched for cd. Finally, determine whether the direction vector of the other side corresponding to target corner point 4 is relatively set to the direction vector of the other side corresponding to target corner point 1, and whether the distance between target corner point 4 and target corner point 1 satisfies the range of the side length to be matched for da. In practical applications, the relative settings in this embodiment are allowed to have a certain range error, and the iteration order can also be changed, which is not limited here.

[0116] As can be seen, assuming that target corner point 1 is the reference corner of the rectangular object, there are three other target corner points that can form candidate object shapes that closely approximate the true shape of the target object together with target corner point 1.

[0117] Finally, based on the selected candidate geometric feature combination and at least one matching geometric feature combination, at least one candidate object shape of the target object is constructed respectively. Since each candidate object shape is allowed a certain error and does not completely correspond to the shape information of the target object, it is necessary to determine whether at least one corresponding candidate object shape can be constructed for each candidate geometric feature combination. Then, it is necessary to filter the target object shape from all candidate object shapes.

[0118] In the above embodiments, by determining the range of angles to be matched corresponding to the target object and the range of side lengths to be matched corresponding to the angles to be matched, the geometric feature combination to be matched can be accurately selected from the remaining candidate geometric feature combinations, thereby constructing at least one candidate object shape that is close to the shape information of the target object.

[0119] In an exemplary embodiment, detecting the similarity result between the shape of each candidate object and the preset standard object shape of the target object includes:

[0120] For each candidate object shape, detect the size difference and angle difference between the candidate object shape and the preset standard object shape of the target object; based on the size difference and angle difference information corresponding to each candidate object shape, detect the similarity result between each candidate object shape and the preset standard object shape.

[0121] Specifically, for each candidate object shape, the size difference information and angle difference information between the candidate object shape and the preset standard object shape of the target object are detected. The size difference information can be the difference information of each side length, such as the difference between the length of the long side of the rectangular object and the length of the long side of the candidate object shape, and the difference between the length of the short side of the rectangular object and the length of the short side of the candidate object shape. The angle difference information is the difference information between each corner of the target object and the corresponding corner of the candidate object shape.

[0122] A first weight is set for size difference information, and a second weight is set for angle difference information. The size difference information is updated according to the first weight, and the angle difference information is updated according to the second weight. The initial similarity score information is obtained based on the updated size difference information and the updated angle difference information. The initial similarity score information is normalized to obtain the final similarity score information. The final similarity score information is used as the similarity result between the corresponding candidate object shape and the preset standard object shape.

[0123] In the above embodiments, by detecting the size difference information and angle difference information between each candidate object shape and the preset standard object shape of the target object, the similarity result between each candidate object shape and the preset standard object shape can be accurately determined, thereby improving the accuracy of selecting the target object shape from multiple candidate object shapes.

[0124] The following will use a rectangular object as an example to describe in detail the process of object positioning:

[0125] 1) Determine the detector parameters of the L-shaped detector. The detector parameters include geometric feature description information such as the direction vector of the main side, the length range of the main side, the direction vector of the secondary side, the length range of the secondary side, and the angle range between the two sides of the main side and the secondary side.

[0126] 2) Confirm the multi-channel color space of the captured image of the target object, and decompose the captured image into multiple single-channel images in the RGB color space and HSV color space in order to extract as many features as possible in the subsequent process.

[0127] 3) Use the Shi-Tomasi corner detection algorithm or the Harris corner detection algorithm to extract the candidate corners that may exist in the captured image. In particular, this step can be skipped when processing detection items such as rounded rectangles that do not have obvious corner features, and the set of lines can be used as the main feature.

[0128] 4) Use the Hough transform algorithm or the FastLineDetector algorithm to extract the candidate line segments that may exist in the captured image.

[0129] 5) From the candidate line segments that may exist in the captured image, search for combinations of candidate line segments. If the projection of the minimum distance between two candidate line segments onto the direction vector corresponding to the L-shaped detector is less than the side length of the corresponding side in the detector, and the included angle is close to the angle range in the detector, then the corresponding candidate line segments or their extensions must have an intersection point, such that the point-line feature satisfies the geometric feature description information of the detector. Record these two candidate line segments as a combination of candidate line segments, and the intersection point is the corner point corresponding to the combination of candidate line segments.

[0130] 6) Remove duplicates from the searched candidate corner points. All candidate corner points after deduplication are candidate corner points that satisfy the detector.

[0131] 7) Combine all the detected corner points with all candidate line segments to determine whether there is a candidate corner point-candidate line segment combination that matches the geometric feature description information of the detector. If it matches, it is considered that the location of the candidate corner point-candidate line segment combination is a detector.

[0132] 8) Assuming that in 7), there exists a candidate corner point in the combination of the corner point and candidate line segment of the detector, and the candidate point O is a corner of a rectangular object, then there are three other candidate corner points O1, O2, and O3. The shape of the candidate object formed by the three other candidate corner points O1, O2, and O3 and the candidate point O is close to the length and width of the rectangular object.

[0133] 9) Calculate the size difference between the candidate object shape and the given rectangular size of the rectangular object, and calculate the angle difference between the four included angles of the candidate object shape and the corresponding given rectangular angles of the rectangular object. After assigning weights to the size difference and angle difference, normalize them to obtain the score index of the candidate object shape, and determine the target object shape whose final score exceeds the set threshold.

[0134] 10) Use other features, such as height and color, to perform secondary filtering on the shape of the target object, update the shape of the target object, and locate the rectangular object based on the updated shape of the target object.

[0135] This application proposes an object localization method based on detector-based feature search. By constructing a searcher that matches the edge direction vector and angle features of the target object, it searches for and combines possible regions of the target object in the captured image to construct candidate geometric feature combinations that match the target object's geometric characteristics. Then, it selects the target object shape from multiple candidate object shapes constructed based on these combinations and locates the target object according to its shape. Its advantage lies in creating descriptors with simple geometric features instead of point and line sets, allowing for the incorporation of more information, including direction vectors and angles, to record complex features during the search process. Figure 4 The image shows an L-shaped detector used for rectangular planes or objects with near-right angles. It consists of the direction vectors of the corner points and the corresponding right-angled sides.

[0136] Compared to traditional algorithms, this calculation method adds the calculation of the object's angular range and the corresponding edge direction vectors. It performs a second-order combination based on corner detection or line segment detection, while traditional algorithms use single corner or line features for object localization. The advantage of this method is that it can still achieve object localization even in complex scenes or when the target edge is arc-shaped and lacks obvious corner features, without requiring pre-training like deep learning methods.

[0137] Specifically, compared to convex hull detection and connected component analysis, this method can assist in localization based on the object's edge direction vector while considering the object's angular range. Compared to Hough transform detection, this method filters the angular range based on the detector's pose information while calculating line segment features, and can provide results that better fit the detection target when dealing with workpieces with a large number of welds, rusted workpieces, and surface grinding patterns.

[0138] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0139] Based on the same inventive concept, this application also provides an object positioning device for implementing the object positioning method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more object positioning device embodiments provided below can be found in the limitations of the object positioning method described above, and will not be repeated here.

[0140] In one exemplary embodiment, such as Figure 5 As shown, an object localization device is provided, including: an information acquisition module 200, a corner and line segment detection module 400, a candidate geometric feature combination detection module 600, and an object localization module 800, wherein:

[0141] The information acquisition module 200 is used to acquire the object angle range and the object side direction vector of the target object;

[0142] The corner and line segment detection module 400 is used to extract at least one candidate corner and at least one candidate edge line segment combination from the captured image corresponding to the target object.

[0143] The candidate geometric feature combination detection module 600 is used to extract at least one candidate geometric feature combination from all candidate corner points and all candidate edge line segment combinations based on the object's angle range and the object's edge direction vector.

[0144] The object localization module 800 is used to filter the target object shape from the candidate object shapes corresponding to each candidate geometric feature combination, and to locate the target object according to the target object shape.

[0145] In one embodiment, there are multiple candidate corner points; the corner point and line segment detection module 400 is further configured to extract at least one first candidate corner point and at least one candidate edge line segment from the captured image corresponding to the target object; based on the object angle range and the object edge direction vector, at least one candidate edge line segment combination is selected from the at least one candidate edge line segment, and a second candidate corner point is determined corresponding to each candidate edge line segment combination; by combining the first candidate corner point and the second candidate corner point, multiple candidate corner points are obtained.

[0146] In one embodiment, the corner and line segment detection module 400 is further configured to segment the captured image corresponding to the target object into single-channel images in different color spaces; and extract at least one candidate corner and at least one candidate edge line segment combination from each single-channel image.

[0147] In one embodiment, the object localization module 800 is further configured to acquire at least one candidate object shape corresponding to each candidate geometric feature combination; detect the similarity result between each candidate object shape and the preset standard object shape of the target object; and filter the target object shape from at least one candidate object shape based on at least one similarity result.

[0148] In one embodiment, the candidate geometric feature combination includes at least a target corner point; the object localization module 800 is further configured to, for each candidate geometric feature combination, determine the target corner point as the reference angle of the target object, and obtain at least one range of angles to be matched and at least one range of side lengths to be matched for the target object, wherein the range of angles to be matched is the range of angles to be matched in the target object excluding the reference angle; select the matching geometric feature combination corresponding to each matching angle from the remaining candidate geometric feature combinations based on the at least one range of angles to be matched and the at least one range of side lengths to be matched; and construct at least one candidate object shape of the target object based on the candidate geometric feature combination and the matching geometric feature combination.

[0149] In one embodiment, the object localization module 800 is further configured to detect, for each candidate object shape, the size difference information and angle difference information between the candidate object shape and the preset standard object shape of the target object; and based on the size difference information and angle difference information corresponding to each candidate object shape, detect the similarity result between each candidate object shape and the preset standard object shape.

[0150] Each module in the aforementioned object positioning device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0151] In one exemplary embodiment, an object localization system is also provided, the system including a visual inspection device and a processor;

[0152] The visual inspection device is used to capture images of the target object and send the captured images to the processor;

[0153] The processor is used for:

[0154] Obtain the target object's angle range and edge direction vector;

[0155] Extract at least one candidate corner point and at least one candidate edge line segment combination from the captured image corresponding to the target object;

[0156] Based on the object's angle range and the object's edge direction vector, extract at least one candidate geometric feature combination from all candidate corner points and all candidate edge line segment combinations;

[0157] The target object shape is selected from the candidate object shapes corresponding to each candidate geometric feature combination, and the target object is located based on the target object shape.

[0158] Specifically, the visual detector can be a detector used to capture images of the target object, such as a camera or video camera. The visual detector can be controlled by a terminal or processor to capture images of the target object and then send the captured images to the processor. Based on the captured images of the target object, the processor executes an object localization method. The specific process of the object localization method has been explained above and will not be repeated here.

[0159] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data such as captured images of the target object. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements an object localization method.

[0160] Those skilled in the art will understand that Figure 6 The structure shown is a block diagram of a partial structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0161] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0162] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0163] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0164] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0165] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0166] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0167] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for locating an object, characterized in that, The method includes: Obtain the object angle range and object side direction vector of the target object. The object angle range refers to the tolerance range that is close to any angle of the target object. The tolerance range is a pre-configured range. The object side direction vector refers to the object side direction vector associated with any of the object angle ranges. There is at least one object side direction vector associated with each object angle range. Extract at least one candidate corner point and at least one candidate edge line segment combination from the captured image corresponding to the target object; Based on the object angle range and the object edge direction vector, at least one candidate geometric feature combination is extracted from all candidate corner points and all candidate edge line segment combinations, wherein the angle of each candidate corner point in the candidate geometric feature combination satisfies the object angle range, and the direction vector of the candidate edge line segment in the candidate edge line segment combination satisfies the object edge direction vector; The target object shape is selected from the candidate object shapes corresponding to each candidate geometric feature combination, and the target object is located according to the target object shape.

2. The method according to claim 1, characterized in that, The candidate corner points are multiple; the step of extracting at least one candidate corner point and at least one candidate edge segment combination from the captured image corresponding to the target object includes: Extract at least one first candidate corner point and at least one candidate edge line segment from the captured image corresponding to the target object; Based on the object's angle range and the object's edge direction vector, at least one candidate edge segment combination is selected from the at least one candidate edge segment, and a second candidate corner point is determined for each candidate edge segment combination. By combining the first candidate corner point and the second candidate corner point, multiple candidate corner points are obtained.

3. The method according to claim 1, characterized in that, Extracting at least one candidate corner point and at least one candidate edge segment combination from the captured image corresponding to the target object includes: The captured image corresponding to the target object is segmented into single-channel images in different color spaces; Extract at least one candidate corner point and at least one candidate edge segment combination from each of the single-channel images.

4. The method according to claim 1, characterized in that, The step of filtering the target object shape from the candidate object shapes corresponding to each of the candidate geometric feature combinations includes: Obtain at least one candidate object shape corresponding to each of the candidate geometric feature combinations; The similarity results between the shape of each candidate object and the preset standard object shape of the target object are detected. Based on at least one of the similarity results, a target object shape is selected from the at least one candidate object shape.

5. The method according to claim 4, characterized in that, The candidate geometric feature combination includes at least a target corner point; obtaining at least one candidate object shape corresponding to each candidate geometric feature combination includes: For each candidate geometric feature combination, the target corner point is determined as the reference angle of the target object, and at least one range of angles to be matched and at least one range of side lengths to be matched of the target object are obtained, wherein the range of angles to be matched is the range of angles to be matched in the target object excluding the reference angle. Based on the at least one range of angles to be matched and the at least one range of side lengths to be matched, select the geometric feature combination corresponding to each angle to be matched from the remaining candidate geometric feature combinations; Based on the candidate geometric feature combination and the geometric feature combination to be matched, at least one candidate object shape of the target object is constructed.

6. The method according to claim 4, characterized in that, The detection of similarity results between the shape of each candidate object and the preset standard object shape of the target object includes: For each candidate object shape, detect the size difference information and angle difference information between the candidate object shape and the preset standard object shape of the target object; Based on the size difference information and angle difference information corresponding to each candidate object shape, the similarity result between each candidate object shape and the preset standard object shape is detected.

7. An object positioning device, characterized in that, The device includes: The information acquisition module is used to acquire the object angle range and the object side direction vector of the target object. The object angle range refers to the tolerance range that is close to any angle of the target object, and the tolerance range is a pre-configured range. The object side direction vector refers to the object side direction vector associated with any of the object angle ranges, and there is at least one object side direction vector associated with each object angle range. The corner and line segment detection module is used to extract at least one candidate corner and at least one candidate edge line segment combination from the captured image corresponding to the target object. The candidate geometric feature combination detection module is used to extract at least one candidate geometric feature combination from all the candidate corner points and all the candidate edge line segments based on the object angle range and the object edge direction vector. The angle of each candidate corner point in the candidate geometric feature combination satisfies the object angle range, and the direction vector of the candidate edge line segment in the candidate edge line segment combination satisfies the object edge direction vector. The object localization module is used to filter the target object shape from the candidate object shapes corresponding to each candidate geometric feature combination, and to locate the target object according to the target object shape.

8. An object positioning system, characterized in that, The system includes a visual inspection device and a processor; The visual inspection device is used to capture images of the target object and send the captured images to the processor; The processor is used for: The object angle range and object side direction vector of the target object are obtained. The object angle range refers to the tolerance range that is close to any angle of the target object. The tolerance range is a pre-configured range. The object side direction vector refers to the object side direction vector associated with any of the object angle ranges. There is at least one object side direction vector associated with each object angle range. Extract at least one candidate corner point and at least one candidate edge line segment combination from the captured image corresponding to the target object; Based on the object angle range and the object edge direction vector, at least one candidate geometric feature combination is extracted from all candidate corner points and all candidate edge line segment combinations, wherein the angle of each candidate corner point in the candidate geometric feature combination satisfies the object angle range, and the direction vector of the candidate edge line segment in the candidate edge line segment combination satisfies the object edge direction vector; The target object shape is selected from the candidate object shapes corresponding to each candidate geometric feature combination, and the target object is located according to the target object shape.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images