Bobbin positioning method and device based on normal vector analysis and storage medium

Abstract

The invention relates to the technical field of computer vision positioning based on image data processing, and discloses a bobbin positioning method and device based on normal vector analysis and a storage medium, and the method comprises the steps: obtaining a left visual angle image, shot by a left camera, of a bobbin and a right visual angle image, shot by a right camera, of the bobbin in a binocular camera; aiming at the left visual angle image and the right visual angle image, respectively executing the following operations to obtain a vector set of a corresponding camera: calculating an included angle between a surface normal vector of each pixel point of each image and an optical center ray of the corresponding camera; selecting a pixel with the maximum included angle in each row as a middle axis point to form a middle axis point set; calculating a direction vector from the optical center to each central axis point to form a vector set of the left and right cameras; determining a shortest distance point between each direction vector of the left camera and a corresponding vector of the right camera to form a shortest distance point set; and based on fusion calculation of the point set, determining a positioning point of the bobbin. According to the scheme, the positioning point of the bobbin can be accurately and stably obtained.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer vision positioning based on image data processing, for example to a yarn tube positioning method based on normal vector analysis, a device and a storage medium. BACKGROUND

[0002] In the automatic production process, the accurate positioning of the vision system is a key prerequisite for realizing automatic operation. Taking the positioning of the yarn tube in the spinning scene as an example, the yarn tube is the core carrier for winding and transferring the yarn, and the positioning accuracy of the yarn tube directly determines the success rate of automatic splicing and automatic tube changing. However, the existing technology has significant defects in this scene: firstly, the surface of the yarn tube is mostly made of plastic or paper material, which is easy to produce strong light reflection or mirror highlights, causing the positioning method based on traditional gray feature, edge contour or depth value extraction to be seriously disturbed by light, and the feature point recognition deviation is large, thereby causing the central axis positioning error; secondly, the cylindrical structure of the yarn tube is easy to appear geometric degeneration at a certain shooting angle (such as top view, side view close to parallel), that is, the circular cross-sectional contour is deformed and overlapped, so that the positioning method relying on edge fitting or point cloud modeling has a sharp drop in stability, and even positioning failure occurs; thirdly, a large amount of cotton fibers floating in the spinning workshop are easy to adhere to the surface of the yarn tube or block the camera imaging field of view, causing the image features required for positioning to be incomplete, and the existing positioning scheme based on complete point cloud or image segmentation has insufficient anti-shielding capability, frequently causing false detection and missed detection problems, which seriously affects the efficiency of the automatic process. The above problems are not only present in the yarn tube positioning scene, but also are common in other scenes involving the positioning of cylindrical objects.

[0003] The positioning scheme for the yarn tube provided by the related technology is mostly limited to local optimization in a single field. For example, only the yarn tube surface is treated to prevent light reflection, or only a special positioning algorithm is designed for a specific model of cylindrical parts, which lacks universality and systematicness. This leads to a narrow application range, making it difficult to adapt to the positioning needs of yarn tubes of different fields and different models, and the positioning accuracy and stability cannot meet the higher standard requirements. SUMMARY

[0004] The following presents a simplified summary of some aspects of the disclosed embodiments in order to provide a basic understanding of such embodiments. The summary is not an extensive overview of the disclosure, and is not intended to identify key / critical elements of the embodiments or to delineate the scope of the embodiments. Its sole purpose is to present some aspects of the disclosed embodiments in a simplified form as a prelude to the more detailed description that is presented later.

[0005] The present disclosure provides a yarn tube positioning method based on normal vector analysis, a device and a storage medium, which can accurately and stably obtain the positioning point of the yarn tube.

[0006] According to a first aspect of the present disclosure, a yarn tube positioning method based on normal vector analysis is provided, comprising: obtaining a left-view image of the bobbin captured by a left camera and a right-view image of the bobbin captured by a right camera in a binocular camera; For the left-view image and the right-view image, the following operations are performed respectively to obtain a vector set of the corresponding camera: determining a surface normal vector of each pixel point, calculating an included angle formed between the surface normal vector of each pixel point and a ray from the optical center of the corresponding camera to the pixel point; In each row of pixel points, the pixel point with the largest included angle is selected as a central axis pixel point to form a central axis pixel point set; calculating a directional vector of the optical center of the corresponding camera to each central axis pixel point set in the central axis pixel point set to obtain a vector set of the corresponding camera; After obtaining the vector set of the left camera and the vector set of the right camera, a shortest distance point set is formed by determining the shortest distance point between each directional vector in the vector set of the left camera and the corresponding directional vector in the vector set of the right camera; determining a positioning point of the bobbin based on the shortest distance point set.

[0007] In some embodiments, obtaining a left-view image of the bobbin captured by a left camera and a right-view image of the bobbin captured by a right camera in a binocular camera comprises: obtaining a left-view original image of the bobbin captured by the left camera and a right-view original image of the bobbin captured by the right camera in the binocular camera; segmenting a region occupied by the bobbin in the left-view original image to obtain a final left-view image, and segmenting a region occupied by the bobbin in the right-view original image to obtain a final right-view image.

[0008] In some embodiments, segmenting a region occupied by the bobbin in the left-view original image to obtain a final left-view image, and segmenting a region occupied by the bobbin in the right-view original image to obtain a final right-view image comprises: segmenting a region occupied by a specified part of the bobbin in the left-view original image to obtain a final left-view image, and segmenting a region occupied by the specified part of the bobbin in the right-view original image to obtain a final right-view image.

[0009] In some embodiments, determining a surface normal vector of each pixel point comprises: inputting the left-view image or the right-view image into a normal vector estimation model to determine the surface normal vector of each pixel point by the normal vector estimation model; The normal vector estimation model is trained based on sample images of different views of a sample object captured by the left camera and / or the right camera in the binocular camera, and the surface normal vector of each pixel point in the sample images.

[0010] In some embodiments, the surface normal vector of each pixel in the sample image is obtained by the following way: The sample images of the sample object at different viewing angles are used to establish a three-dimensional model of the sample object. Based on the three-dimensional model of the sample object, the surface normal vector of each pixel in the sample image is calculated.

[0011] In some embodiments, the shortest distance point is obtained by the following way: The directional vectors in the vector set of the left camera and the vector set of the right camera are converted to the same three-dimensional coordinate system. In the three-dimensional coordinate system, the shortest distance point of each directional vector in the vector set of the left camera and the corresponding directional vector in the vector set of the right camera is determined.

[0012] In some embodiments, the shortest distance point is any one of the following types: The point in each directional vector in the vector set of the left camera that is closest to the corresponding directional vector in the vector set of the right camera; The point in each directional vector in the vector set of the right camera that is closest to the corresponding directional vector in the vector set of the left camera; The midpoint of the shortest line segment between each directional vector in the vector set of the left camera and the corresponding directional vector in the vector set of the right camera.

[0013] In some embodiments, determining the shortest distance point of each directional vector in the vector set of the left camera and the corresponding directional vector in the vector set of the right camera to form a shortest distance point set comprises: Determining the shortest distance point of each directional vector in the vector set of the left camera and the corresponding directional vector in the vector set of the right camera; Performing median noise reduction on all the shortest distance points, and forming a shortest distance point set based on the retained shortest distance points.

[0014] In some embodiments, determining the positioning point of the yarn tube based on the shortest distance point set comprises: calculating the geometric center of the shortest distance points in the shortest distance point set, and determining the geometric center as the positioning point of the yarn tube.

[0015] According to a second aspect of the present disclosure, a yarn tube positioning device based on normal vector analysis is provided, comprising a processor and a memory storing program instructions, and the processor executes the yarn tube positioning method based on normal vector analysis provided by the first aspect of the present disclosure.

[0016] According to a third aspect of the present disclosure, a storage medium is provided, and the storage medium stores computer program instructions, and when the computer program instructions are run by a processor, the yarn tube positioning method based on normal vector analysis provided by the first aspect of the present disclosure is executed.

[0017] The yarn tube positioning method, device and storage medium based on normal vector analysis provided by the embodiments of the present disclosure can achieve the following technical effects: The yarn tube positioning method based on normal vector analysis provided by the embodiments of the present disclosure, after obtaining the left-view image and the right-view image of the yarn tube, takes the surface normal vector of each pixel point as the core judgment basis for each image. Since the surface normal vector reflects the local geometric orientation of the object rather than the optical reflection intensity, even in the case of strong light reflection or mirror highlights on the surface of the yarn tube, the axial position can still be estimated based on the surface normal vector of each pixel point, which significantly improves the positioning stability in a complex lighting environment and effectively overcomes the feature recognition deviation problem caused by uneven lighting. By calculating the included angle between the surface normal vector and the optical center ray, and selecting the pixel point with the largest included angle as the central axis pixel point, this method uses the geometric characteristic that the normal of the central axis of the cylinder is most aligned with the camera sight line direction when the central axis of the cylinder is directly opposite the camera, and independently searches for the central axis pixel point in each row of pixels. The above strategy for determining the central axis pixel point does not depend on the complete contour or point cloud structure, and even in the case of top view or side view, which can easily lead to the degradation of the contour geometry, the pixel point located at the central axis can still be extracted from the local effective area, overcoming the failure problem of traditional edge fitting or point cloud modeling methods in a specific view angle. In addition, this method determines the positioning point of the yarn tube based on the general geometric relationship between the camera optical center and the central axis pixel point, and does not depend on the prior model of specific size or material, and has good universality, which can adapt to the positioning needs of yarn tubes of different diameters and heights.

[0018] The foregoing general description and the following description are only exemplary and explanatory, and are not intended to limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0019] One or more embodiments are exemplarily illustrated by corresponding drawings, which do not constitute limitations on the embodiments, elements with the same reference numerals in the drawings are shown as similar elements, the drawings do not constitute proportional limitations, and wherein: Figure 1 is a hardware architecture schematic diagram of an embodiment of a yarn tube positioning method based on normal vector analysis provided by the embodiments of the present disclosure; Figure 2 is a schematic diagram of a yarn tube positioning method based on normal vector analysis provided by the embodiments of the present disclosure; Figure 3 is a schematic diagram of another yarn tube positioning method based on normal vector analysis provided by the embodiments of the present disclosure; Figure 4 is a schematic diagram of another yarn tube positioning method based on normal vector analysis provided by the embodiments of the present disclosure; Figure 5is a schematic diagram of another yarn tube positioning method based on normal vector analysis provided by the embodiments of the present disclosure. Figure 6 is a schematic diagram of a yarn tube positioning device based on normal vector analysis provided by the embodiments of the present disclosure. DETAILED DESCRIPTION

[0020] In order to enable a more detailed understanding of the features and technical content of the embodiments of the present disclosure, the implementation of the embodiments of the present disclosure will be described in detail below in conjunction with the drawings, which are only used for reference and do not limit the embodiments of the present disclosure. In the following technical description, in order to facilitate explanation, a plurality of details are provided to provide a full understanding of the disclosed embodiments. However, one or more embodiments can still be implemented without these details. In other cases, well-known structures and devices can be simplified to facilitate the drawings.

[0021] The terms "first", "second", and the like in the specification and claims of the embodiments of the present disclosure and the above drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion.

[0022] Unless otherwise specified, the term "a plurality of" means two or more.

[0023] In the embodiments of the present disclosure, the character " / " represents an "or" relationship between the objects before and after it. For example, A / B means A or B.

[0024] The term "and / or" is a description of the association between objects, which means that there can be three relationships. For example, A and / or B means that there are three relationships of A or B, or A and B.

[0025] The term "corresponding" can refer to an association or binding relationship. A and B correspond to each other means that there is an association or binding relationship between A and B.

[0026] The embodiments of the present disclosure provide a yarn tube positioning method and positioning device based on normal vector analysis, Figure 1A hardware architecture schematic diagram of implementing the yarn tube positioning method based on normal vector analysis is shown. In this application scenario, the yarn tube positioning device based on normal vector analysis (hereinafter referred to as the positioning device) can be in communication connection with a binocular camera. The binocular camera is an imaging system that simulates human binocular vision. The binocular camera includes two cameras with fixed spatial positions, parallel or approximately parallel optical axes, which are referred to as a left camera and a right camera. The left camera and the right camera can synchronously collect images of the same yarn tube at different angles of view. For the convenience of understanding and description, the image obtained by the left camera is defined as a left-view image, and the image obtained by the right camera is defined as a right-view image.

[0027] The embodiments of the present disclosure are applied to a spinning scene. The left camera in the binocular camera captures a left-view image of the yarn tube, and the right camera captures a right-view image of the yarn tube. The positioning device can obtain the left-view image and the right-view image of the yarn tube, and determine a positioning point of the yarn tube based on the two images.

[0028] In combination with the positioning device provided by the embodiments of the present disclosure, the embodiments of the present disclosure provide a yarn tube positioning method based on normal vector analysis. As shown in Figure 2 The yarn tube positioning method based on normal vector analysis includes the following steps: S201, the positioning device obtains a left-view image of the yarn tube captured by the left camera in the binocular camera and a right-view image of the yarn tube captured by the right camera.

[0029] In the embodiments of the present disclosure, after S201, S202 to S204 are respectively performed for the left-view image and the right-view image, to obtain a vector set of the left camera and a vector set of the right camera. Here, the specific contents of S202 to S204 are as follows: S202, the positioning device determines a surface normal vector of each pixel point, and calculates an included angle formed by the surface normal vector of each pixel point and a ray from the optical center of the corresponding camera to the pixel point.

[0030] In the embodiments of the present disclosure, the left-view image corresponds to the left camera in the binocular camera, and the right-view image corresponds to the right camera in the binocular camera. For the left-view image, the included angle formed by the surface normal vector of each pixel point in the left-view image and the ray from the optical center of the left camera to the pixel point is calculated; for the right-view image, the included angle formed by the surface normal vector of each pixel point in the right-view image and the ray from the optical center of the right camera to the pixel point is calculated. Here, the included angle formed by the ray from the optical center to the pixel point ranges from 0 to 180 degrees.

[0031] S203, the positioning device selects, in each row of pixel points, a pixel point with the largest included angle as a central axis pixel point, to form a central axis pixel point set.

[0032] In the embodiments of the present disclosure, calculating the included angle formed by the surface normal vector of each pixel point and the ray from the optical center of the corresponding camera to the pixel point and selecting the maximum included angle can be achieved by the following formula: Argp max ||n p v c ||(p∈ROI row ), wherein ROI row represents a set of pixel points in a corresponding image (such as a left-view image or a right-view image), n p represents the surface normal vector of the pixel point p in the corresponding image (such as a left-view image or a right-view image), v c represents the unit vector corresponding to the ray from the optical center of the corresponding camera (such as a left camera or a right camera) to the pixel point p.

[0033] In S204, the positioning device calculates the direction vector from the optical center of the corresponding camera to each of the sets of central-axis pixel points, to obtain a set of vectors of the corresponding camera.

[0034] As described above, the left-view image corresponds to the left camera in the binocular camera, and the right-view image corresponds to the right camera in the binocular camera. For the set of central-axis pixel points of the left-view image, the direction vector from the optical center of the left camera to each of the sets of central-axis pixel points is calculated, to obtain a set of vectors of the left camera; for the set of central-axis pixel points of the right-view image, the direction vector from the optical center of the right camera to each of the sets of central-axis pixel points is calculated, to obtain a set of vectors of the right camera.

[0035] In the embodiments of the present disclosure, after obtaining the set of vectors of the left camera and the set of vectors of the right camera, the positioning device can continue to perform S205 and S206.

[0036] In S205, the positioning device determines the shortest distance point between each direction vector in the set of vectors of the left camera and the corresponding direction vector in the set of vectors of the right camera, to form a set of shortest distance points.

[0037] Here, the shortest distance point is located at or adjacent to the central axis of the bobbin, and can effectively reflect the central geometric position of the bobbin.

[0038] In S206, the positioning device determines the positioning point of the bobbin based on the set of shortest distance points.

[0039] The yarn tube positioning method based on normal vector analysis provided by the embodiments of the present disclosure, after obtaining the left-view image and the right-view image of the yarn tube, takes the surface normal vector of each pixel point as the core judgment basis for each image. Since the surface normal vector reflects the local geometric orientation of the object rather than the optical reflection intensity, even in the case of strong light reflection or mirror highlight on the surface of the yarn tube, the axial position can still be estimated based on the surface normal vector of each pixel point, which significantly improves the positioning stability in complex lighting environments and effectively overcomes the feature recognition deviation problem caused by uneven lighting. By calculating the included angle between the surface normal vector and the optical center ray, and selecting the pixel point with the largest included angle as the central axis pixel point, this method uses the geometric characteristic that the central axis of the cylinder is directly opposite to the camera, and the normal direction of the central axis is closest to the alignment direction of the camera view line, to independently search for the central axis pixel point in each row of pixels. The above strategy for determining the central axis pixel point does not depend on the complete contour or point cloud structure, and even in the case of top view or side view, which can easily lead to the degradation of the contour geometry, the pixel point located on the central axis can still be extracted from the local effective area, overcoming the failure problem of traditional edge fitting or point cloud modeling methods in certain viewing angles. In addition, the method determines the positioning point of the yarn tube based on the general geometric relationship between the camera optical center and the central axis pixel point, and does not depend on the prior model of specific size or material, which has good universality and can adapt to the positioning needs of yarn tubes of different diameters and heights.

[0040] In some embodiments, obtaining the left-view image of the yarn tube taken by the left camera and the right-view image of the yarn tube taken by the right camera in the binocular camera includes: obtaining the original left-view image of the yarn tube taken by the left camera and the original right-view image of the yarn tube taken by the right camera in the binocular camera; segmenting the area occupied by the yarn tube in the original left-view image to obtain the final left-view image, and segmenting the area occupied by the yarn tube in the original right-view image to obtain the final right-view image. The embodiments of the present disclosure can reduce the interference of irrelevant background information by segmenting the area occupied by the yarn tube in the original image, which helps to improve the efficiency and accuracy of the subsequent processing steps.

[0041] As shown in Figure 3 The embodiments of the present disclosure provide another yarn tube positioning method based on normal vector analysis, including the following steps: S301, the positioning device obtains the original left-view image of the yarn tube taken by the left camera and the original right-view image of the yarn tube taken by the right camera in the binocular camera.

[0042] S302, the positioning device segments the area occupied by the yarn tube in the original left-view image to obtain the final left-view image, and segments the area occupied by the yarn tube in the original right-view image to obtain the final right-view image.

[0043] In the embodiments of the present disclosure, after S302, S303-S305 are performed for the left-view image and the right-view image respectively to obtain the vector set of the left camera and the vector set of the right camera. Here, the specific contents of S303-S305 are as follows: S303, the positioning device determines the surface normal vector of each pixel point, and calculates the included angle formed by the surface normal vector of each pixel point and the ray from the optical center of the corresponding camera to the pixel point.

[0044] In the embodiments of the present disclosure, the left-view image corresponds to the left camera in the binocular camera, and the right-view image corresponds to the right camera in the binocular camera. For the left-view image, the included angle formed by the surface normal vector of each pixel point in the left-view image and the ray from the optical center of the left camera to the pixel point is calculated; for the right-view image, the included angle formed by the surface normal vector of each pixel point in the right-view image and the ray from the optical center of the right camera to the pixel point is calculated.

[0045] S304, the positioning device selects the pixel point with the largest included angle in each row of pixel points as the central axis pixel point to form a central axis pixel point set.

[0046] S305, the positioning device calculates the directional vector from the optical center of the corresponding camera to each central axis pixel point set in the central axis pixel point set to obtain the vector set of the corresponding camera.

[0047] As described above, the left-view image corresponds to the left camera in the binocular camera, and the right-view image corresponds to the right camera in the binocular camera. For the central axis pixel point set of the left-view image, the directional vector from the optical center of the left camera to each central axis pixel point set in the central axis pixel point set is calculated to obtain the vector set of the left camera; for the central axis pixel point set of the right-view image, the directional vector from the optical center of the right camera to each central axis pixel point set in the central axis pixel point set is calculated to obtain the vector set of the right camera.

[0048] In the embodiments of the present disclosure, after obtaining the vector set of the left camera and the vector set of the right camera, the positioning device can continue to perform S306 and S307.

[0049] S306, the positioning device determines the shortest distance point of each directional vector in the vector set of the left camera and the directional vector corresponding to the vector set of the right camera to form a shortest distance point set.

[0050] S307, the positioning device determines the positioning point of the bobbin based on the shortest distance point set.

[0051] In some embodiments, segmenting the region occupied by the yarn tube in the original left-view image to obtain the final left-view image, and segmenting the region occupied by the yarn tube in the original right-view image to obtain the final right-view image, includes: segmenting the region occupied by a specified portion of the yarn tube in the original left-view image to obtain the final left-view image, and segmenting the region occupied by a specified portion of the yarn tube in the original right-view image to obtain the final right-view image. Here, the specified portion of the yarn tube can be determined according to actual design needs; for example, the specified portion of the yarn tube may be the top region. This embodiment of the disclosure, by segmenting the region occupied by a portion of the yarn tube in the original image, can reduce interference from irrelevant background information, thus helping to improve the efficiency and accuracy of subsequent processing steps.

[0052] In some embodiments, determining the surface normal vector of each pixel includes: inputting a left-view image or a right-view image into a normal vector estimation model, and having the normal vector estimation model determine the surface normal vector of each pixel.

[0053] like Figure 4 As shown, this disclosure provides another yarn tube positioning method based on normal vector analysis, including the following steps: S401, the positioning device acquires the left-view image of the yarn tube taken by the left camera and the right-view image of the yarn tube taken by the right camera in the binocular camera system.

[0054] In this embodiment of the disclosure, after S401, S402 to S405 are executed for the left-view image and the right-view image respectively, to obtain the vector set of the left camera and the vector set of the right camera. The specific contents of S402 to S405 are as follows: S402, the positioning device inputs the left-view image or the right-view image into the normal vector estimation model, and the normal vector estimation model determines the surface normal vector of each pixel.

[0055] S403, the positioning device calculates the angle between the surface normal vector of each pixel and the ray from the optical center of the corresponding camera to that pixel.

[0056] In this embodiment of the disclosure, the left-view image corresponds to the left camera in the binocular camera, and the right-view image corresponds to the right camera in the binocular camera. For the left-view image, the angle formed by the surface normal vector of each pixel in the left-view image and the ray from the optical center of the left camera to that pixel is calculated; for the right-view image, the angle formed by the surface normal vector of each pixel in the right-view image and the ray from the optical center of the right camera to that pixel is calculated.

[0057] S404, the positioning device selects the pixel with the largest included angle in each row of pixels as the central axis pixel, forming a central axis pixel set.

[0058] S405, the positioning device calculates a direction vector of the optical center of the corresponding camera to each of the set of central axis pixels, to obtain a vector set of the corresponding camera.

[0059] As described above, the left-view image corresponds to the left camera in the binocular camera, and the right-view image corresponds to the right camera in the binocular camera. For the set of central axis pixels of the left-view image, the direction vector of the optical center of the left camera to each of the set of central axis pixels is calculated, to obtain a vector set of the left camera; for the set of central axis pixels of the right-view image, the direction vector of the optical center of the right camera to each of the set of central axis pixels is calculated, to obtain a vector set of the right camera.

[0060] In the embodiments of the present disclosure, after obtaining the vector set of the left camera and the vector set of the right camera, the positioning device can continue to perform S406 and S407.

[0061] S406, the positioning device determines the shortest distance point between each direction vector in the vector set of the left camera and the corresponding direction vector in the vector set of the right camera, to form a set of shortest distance points.

[0062] S407, the positioning device determines the positioning point of the bobbin based on the set of shortest distance points.

[0063] In some embodiments, the normal vector estimation model is trained based on sample images of different views of a sample object taken by the left camera and / or the right camera in the binocular camera, and the surface normal vector of each pixel point in the sample images.

[0064] Optionally, the normal vector estimation model is trained based on sample images of different views of a sample object taken by the left camera in the binocular camera, and the surface normal vector of each pixel point in the sample images.

[0065] Optionally, the normal vector estimation model is trained based on sample images of different views of a sample object taken by the right camera in the binocular camera, and the surface normal vector of each pixel point in the sample images.

[0066] Optionally, the normal vector estimation model is trained based on sample images of different views of a sample object taken by the left camera and the right camera in the binocular camera, and the surface normal vector of each pixel point in the sample images.

[0067] In the embodiments of the present disclosure, the normal vector estimation model can be a deep learning model or a machine learning model, which learns the complex mapping relationship between the surface geometric features and the image appearance from a large amount of real sample data, and can more accurately predict the surface normal direction of each pixel point under complex conditions such as illumination change, texture loss or reflection interference. After training, the normal vector estimation model can quickly perform pixel-level normal vector inference on new input images without complex three-dimensional reconstruction or point cloud processing procedures, significantly improving the real-time performance of the system.

[0068] In some embodiments, the surface normal vector of each pixel point in the sample image is obtained by: establishing a three-dimensional model of the sample object based on sample images of the sample object from different viewing angles, and calculating the surface normal vector of each pixel point in the sample image based on the three-dimensional model of the sample object. Here, the three-dimensional model of the sample object obtained by three-dimensional reconstruction based on sample images from different viewing angles can accurately reflect the real geometric shape of the sample object. On this basis, the surface normal vector of the pixel point has high spatial accuracy and physical consistency.

[0069] In some embodiments, determining the shortest distance point between each direction vector in the vector set of the left camera and the corresponding direction vector in the vector set of the right camera to form a shortest distance point set comprises: determining the shortest distance point between each direction vector in the vector set of the left camera and the corresponding direction vector in the vector set of the right camera; performing median denoising on all the shortest distance points, and forming the shortest distance point set based on the retained shortest distance points.

[0070] The embodiments of the present disclosure use the median denoising strategy to effectively remove abnormal shortest distance points, and removing these interference factors helps to improve the stability and reliability of the subsequent analysis steps. The shortest distance points retained by the median denoising are the most representative points, which can better reflect the geometric structure of the real scene and help to obtain accurate positioning points.

[0071] As shown in Figure 5 The embodiments of the present disclosure provide another yarn tube positioning method based on normal vector analysis, comprising the following steps: S501, the positioning device acquires a left-view image of the yarn tube photographed by the left camera and a right-view image of the yarn tube photographed by the right camera.

[0072] In the embodiments of the present disclosure, after S501, S502 to S504 are performed on the left-view image and the right-view image respectively to obtain the vector set of the left camera and the vector set of the right camera. Here, the specific contents of S502 to S504 are as follows: S502, the positioning device determines the surface normal vector of each pixel point, and calculates the included angle formed by the surface normal vector of each pixel point and the ray from the optical center of the corresponding camera to the pixel point.

[0073] In the embodiments of the present disclosure, the left-view image corresponds to a left camera in the binocular camera, and the right-view image corresponds to a right camera in the binocular camera. For the left-view image, an included angle formed by a surface normal vector of each pixel point in the left-view image and a ray from the optical center of the left camera to the pixel point is calculated. For the right-view image, an included angle formed by a surface normal vector of each pixel point in the right-view image and a ray from the optical center of the right camera to the pixel point is calculated.

[0074] In S503, the positioning device selects, in each row of pixel points, a pixel point with the largest included angle as a central-axis pixel point to form a central-axis pixel point set.

[0075] In S504, the positioning device calculates a direction vector of each central-axis pixel point set in the central-axis pixel point set corresponding to the optical center of the camera to obtain a vector set of the camera.

[0076] As described above, the left-view image corresponds to a left camera in the binocular camera, and the right-view image corresponds to a right camera in the binocular camera. For the central-axis pixel point set of the left-view image, a direction vector of each central-axis pixel point set in the central-axis pixel point set to the optical center of the left camera is calculated to obtain a vector set of the left camera. For the central-axis pixel point set of the right-view image, a direction vector of each central-axis pixel point set in the central-axis pixel point set to the optical center of the right camera is calculated to obtain a vector set of the right camera.

[0077] In the embodiments of the present disclosure, after obtaining the vector set of the left camera and the vector set of the right camera, the positioning device can continue to perform S505 and S507.

[0078] In S505, the positioning device determines a shortest distance point of each direction vector in the vector set of the left camera and a direction vector corresponding to the vector set of the right camera.

[0079] In S506, the positioning device performs median denoising on all the shortest distance points to form a shortest distance point set based on the reserved shortest distance points.

[0080] In S507, the positioning device determines a positioning point of the bobbin based on the shortest distance point set.

[0081] In some embodiments, the shortest distance point is obtained by: converting each direction vector in the vector set of the left camera and the vector set of the right camera to the same three-dimensional coordinate system; and determining, in the three-dimensional coordinate system, a shortest distance point of each direction vector in the vector set of the left camera and a direction vector corresponding to the vector set of the right camera from an intersection region of the two direction vectors.

[0082] In the embodiments of the present disclosure, the vector sets of the left and right cameras are converted to the same three-dimensional coordinate system, and the shortest distance points between the two direction vectors are directly calculated in the three-dimensional space, which can more truly reflect the position of the center axis of the yarn pipe. This way of determining the position of the center axis of the yarn pipe is suitable for the case where the surface of the yarn pipe is blocked or not ideally aligned, avoids the projection error caused by two-dimensional image plane matching, and does not depend on pixel-level feature correspondence, and has stronger adaptability to image quality problems such as texture loss, light change and reflection. Even if only part of the center axis pixel points are effective, a reasonable shortest distance point set can be generated through three-dimensional ray intersection, so as to obtain accurate positioning points.

[0083] In the three-dimensional space, the ray (that is, a direction vector in the vector set of the left camera) extending from the optical center of the left camera along a certain center axis pixel direction and the other ray (that is, a direction vector in the vector set of the right camera) extending from the optical center of the right camera along the corresponding center axis pixel direction of the right camera are closest to each other in the space, that is, the intersection region of the two. In other words, the local space region where the shortest line segment between each direction vector in the vector set of the left camera and the corresponding direction vector in the vector set of the right camera is located is the intersection region of the two.

[0084] In some embodiments, the shortest distance point can be the point in each direction vector in the vector set of the left camera that is closest to the corresponding direction vector in the vector set of the right camera, and the point is the end point of the shortest line segment in one direction vector of the vector set of the left camera.

[0085] In some embodiments, the shortest distance point can be the point in each direction vector in the vector set of the right camera that is closest to the corresponding direction vector in the vector set of the left camera, and the point is the end point of the shortest line segment in one direction vector of the vector set of the right camera.

[0086] In some embodiments, the midpoint of the shortest line segment between each direction vector in the vector set of the left camera and the corresponding direction vector in the vector set of the right camera.

[0087] In the embodiments of the present disclosure, the shortest distance line segment between the two direction vectors is determined from the intersection region of each direction vector in the vector set of the left camera and the corresponding direction vector in the vector set of the right camera, which can be realized by the following formula: O =Intersect(Ray(O L ,V iL ),Ray(O R ,V jR ))。

[0088] P OThe shortest line segment between the direction vector i in the vector set of the left camera and the direction vector j corresponding to the vector set of the right camera can be understood as the common perpendicular of the direction vector i and the direction vector j. O L O represents the optical center of the left camera, V iL Ray(O L , V iL ) represents the i-th direction vector in the vector set of the left camera, O R O represents the optical center of the right camera, V jR Ray(O L , V jR ) represents the j-th direction vector in the vector set of the right camera. Intersect(Ray(O L , V iL ), Ray(O R , V jR )) represents the calculation process of determining the shortest distance between each direction vector in the vector set of the left camera and the direction vector corresponding to the vector set of the right camera.

[0089] In some embodiments, the positioning point of the bobbin is determined based on the shortest distance point set, including: calculating the geometric center of the shortest distance point in the shortest distance point set, and determining the geometric center as the positioning point of the bobbin. Here, the geometric center is the result of averaging the spatial positions of multiple independent shortest distance points, and taking this geometric center as the final positioning point of the bobbin can accurately reflect the spatial position of the bobbin.

[0090] As shown in Figure 6 , the bobbin positioning device based on normal vector analysis 600 provided by the embodiments of the present disclosure includes a processor 601 and a memory 602. Optionally, the bobbin positioning device based on normal vector analysis 600 can also include a communication interface 603 and a bus 604. The processor 601, the communication interface 603, and the memory 602 can communicate with each other through the bus 604. The communication interface 603 can be used for information transmission. The processor 601 can call the logical instructions in the memory 602 to execute the bobbin positioning method based on normal vector analysis of the above-mentioned embodiments.

[0091] In addition, the logical instructions in the memory 602 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium.

[0092] The memory 602, as a computer readable storage medium, can be used to store software programs, computer executable programs, such as program instructions / modules corresponding to the method in the embodiments of the present disclosure. The processor 601 executes the function application and data processing by running the program instructions / modules stored in the memory 602, that is, implements the yarn tube positioning method based on normal vector analysis in the above embodiments.

[0093] The memory 602 can include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function; the data storage area can store data created according to the use of the terminal device, etc. In addition, the memory 602 can include a high-speed random access memory, and can also include a non-volatile memory.

[0094] The embodiments of the present disclosure provide a computer readable storage medium, which stores computer executable instructions, and the computer executable instructions are configured to execute the yarn tube positioning method based on normal vector analysis.

[0095] The technical solutions of the embodiments of the present disclosure can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes one or more instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method disclosed in the embodiments of the present disclosure. The aforementioned storage medium can be a non-transitory storage medium, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes. The above description and drawings sufficiently illustrate the embodiments of the present disclosure, so that those skilled in the art can practice them. Other embodiments can include structural, logical, electrical, process, and other changes. The embodiments only represent possible changes. Unless explicitly required, individual components and functions are optional, and the order of operations can be changed. Some parts and features of some embodiments can be included in or replaced by parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and not to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms as well. Similarly, as used in this application, the term "and / or" refers to any and all possible combinations of one or more associated listed items. In addition, when used in this application, the term "comprise" and its variants "comprises" and / or "comprising" and the like refer to the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Without more limitations, the element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, or device that includes the stated element. In this document, each embodiment focuses on the differences from other embodiments, and the same or similar parts between various embodiments can be referred to each other. For the method, product, etc. disclosed in the embodiments, if it corresponds to the method part disclosed in the embodiments, the relevant part can be referred to the description of the method part.

[0096] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods for each specific application to realize the described functions, but such implementation should not be considered beyond the scope of the embodiments of the present disclosure. The skilled person can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0097] In the embodiments disclosed herein, the disclosed methods, products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the above-described device embodiments are only schematic, for example, the division of the units can only be a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms. The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to implement the embodiments. In addition, each functional unit in the embodiments of the present disclosure can be integrated in one processing unit, or each unit can be a physically independent unit, or two or more units can be integrated in one unit.

[0098] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

Claims

1. A yarn tube positioning method based on normal vector analysis, characterized in that, include: Acquire the left-view image of the yarn tube captured by the left camera and the right-view image of the yarn tube captured by the right camera in the binocular camera system. For the left-view and right-view images, perform the following operations to obtain the corresponding camera vector sets: Determine the surface normal vector of each pixel, and calculate the angle between the surface normal vector of each pixel and the ray from the optical center of the corresponding camera to that pixel; In each row of pixels, the pixel with the largest included angle is selected as the central axis pixel, forming the central axis pixel set; Calculate the direction vector from the optical center of the corresponding camera to each set of central axis pixels in the central axis pixel set, and obtain the vector set of the corresponding camera; After obtaining the vector sets of the left camera and the right camera, determine the shortest distance point between each direction vector in the vector set of the left camera and the corresponding direction vector in the vector set of the right camera, and form a set of shortest distance points. The positioning point of the yarn tube is determined based on the set of shortest distance points.

2. The yarn tube positioning method based on normal vector analysis according to claim 1, characterized in that, Acquire the left-view image of the yarn tube captured by the left camera and the right-view image of the yarn tube captured by the right camera in the binocular camera system, including: Acquire the original left-view image of the yarn tube captured by the left camera and the original right-view image of the yarn tube captured by the right camera in the binocular camera system. The area occupied by the yarn tube is segmented from the original left-view image to obtain the final left-view image, and the area occupied by the yarn tube is segmented from the original right-view image to obtain the final right-view image.

3. The yarn tube positioning method based on normal vector analysis according to claim 2, characterized in that, The process involves segmenting the region occupied by the yarn tube from the original left-view image to obtain the final left-view image, and segmenting the region occupied by the yarn tube from the original right-view image to obtain the final right-view image, including: The region occupied by a specified portion of the yarn tube is segmented from the original left-view image to obtain the final left-view image, and the region occupied by a specified portion of the yarn tube is segmented from the original right-view image to obtain the final right-view image.

4. The yarn tube positioning method based on normal vector analysis according to claim 1, characterized in that, Determine the surface normal vector for each pixel, including: The left-view image or the right-view image is input into the normal vector estimation model, and the normal vector estimation model determines the surface normal vector of each pixel. The normal vector estimation model is trained based on sample images of the sample object taken by the left and / or right cameras in the binocular camera from different perspectives, as well as the surface normal vector of each pixel in the sample image.

5. The yarn tube positioning method based on normal vector analysis according to claim 1, characterized in that, The surface normal vector of each pixel in the sample image is obtained in the following way: A 3D model of the sample object is built from sample images from different perspectives; The surface normal vector of each pixel in the sample image is calculated based on the 3D model of the sample object.

6. The yarn tube positioning method based on normal vector analysis according to claim 1, characterized in that, The shortest distance point is obtained in the following way: Transform the directional vectors in the vector sets of the left and right cameras to the same three-dimensional coordinate system; In a three-dimensional coordinate system, the shortest distance between two points is determined by the intersection region of each direction vector in the vector set of the left camera and the corresponding direction vector in the vector set of the right camera.

7. The yarn tube positioning method based on normal vector analysis according to claim 6, characterized in that, The shortest distance point is any of the following types: The point in the left camera's vector set that is closest to the corresponding direction vector in the right camera's vector set; The point in the vector set of the right camera that is closest to the corresponding direction vector in the vector set of the left camera; The midpoint of the shortest line segment between each direction vector in the left camera's vector set and the corresponding direction vector in the right camera's vector set.

8. The yarn tube positioning method based on normal vector analysis according to claim 1, characterized in that, Determine the shortest distance points between each direction vector in the left camera's vector set and the corresponding direction vector in the right camera's vector set, forming a set of shortest distance points, including: Determine the shortest distance between each direction vector in the vector set of the left camera and the corresponding direction vector in the vector set of the right camera; Median denoising is performed on all shortest distance points, and a set of shortest distance points is constructed based on the retained shortest distance points.

9. The yarn tube positioning method based on normal vector analysis according to claim 1, characterized in that, Determining the positioning point of the yarn tube based on the set of shortest distance points includes: calculating the geometric center of the shortest distance point in the set of shortest distance points, and determining the geometric center as the positioning point of the yarn tube.

10. A yarn tube positioning device based on normal vector analysis, comprising a processor and a memory storing program instructions, characterized in that, The processor executes the yarn tube positioning method based on normal vector analysis as described in any one of claims 1 to 9.

11. A storage medium, characterized in that, The storage medium stores computer program instructions, which, when executed by a processor, perform the yarn tube positioning method based on normal vector analysis as described in any one of claims 1 to 9.

Images