A method for obtaining three-dimensional shape of auxiliary materials for warehousing of cigarette auxiliary materials

Abstract

The present invention belongs to the technical field of auxiliary material storage, and discloses a method for acquiring the three-dimensional shape of auxiliary materials for storage of cigarette auxiliary materials. The three-dimensional shape acquisition method includes: using a projector to project a preset grating coding pattern onto the surface of the measurement object; two cameras are respectively set on both sides of the projector, and the two cameras synchronously collect the grating coding pattern on the surface of the measurement object; the two cameras respectively perform phase unpacking algorithms on the grating coding pattern, use the phase information to match the binocular vision of the two cameras, and reconstruct the three-dimensional point cloud of the measurement object through the reconstruction method of binocular vision. The present invention optimizes the auxiliary material storage process, improves inventory efficiency, reduces unnecessary working time, reduces labor costs, improves production and supply efficiency, and provides strong support for the intelligent management of the packaging workshop site.

Description

Technical Field

[0001] The present invention relates to the technical field of auxiliary material storage, and in particular to a method for obtaining the three-dimensional shape of auxiliary materials used for the storage of auxiliary materials for cigarettes. Background Art

[0002] The distribution of auxiliary materials in the cigarette industry is divided into two forms: manual and automatic. However, the automatic auxiliary material return process has not been fully intelligent and automated, and still uses manual counting, which has the characteristics of low counting efficiency, high labor intensity, low accuracy, energy waste, and easy errors.

[0003] According to the previous survey, there are very few companies in the industry that carry out intelligent inventory of auxiliary materials. The existing system can only identify the two-dimensional shape of auxiliary materials, and cannot intelligently identify and determine the three-dimensional shape of auxiliary materials. The existing system requires manual operation of the controller and lacks automation. There is a certain degree of human interference and error in the inventory process. Although it can improve the efficiency of warehousing, it still has problems such as long inventory time and low efficiency in large-scale production environments. Summary of the invention

[0004] The purpose of the present invention is to provide a method for obtaining the three-dimensional shape of auxiliary materials for warehousing of cigarette auxiliary materials, so as to solve the problem.

[0005] To achieve this object, the present invention adopts the following technical solutions:

[0006] A method for obtaining the three-dimensional shape of auxiliary materials for warehousing of cigarette auxiliary materials, the three-dimensional shape obtaining method comprising:

[0007] A projector is used to project a preset grating coding pattern onto the surface of the measurement object;

[0008] Two cameras are arranged axially symmetrically on both sides of the projector, and the two cameras synchronously collect the grating coding pattern on the surface of the measurement object;

[0009] The two cameras respectively perform phase unpacking algorithms on the grating coding pattern, use the phase information to match the binocular vision of the two cameras, and reconstruct a three-dimensional point cloud of the measured object through binocular vision reconstruction.

[0010] Optionally, the step of reconstructing a three-dimensional point cloud of the measured object includes:

[0011] Acquiring a depth map of the measurement object;

[0012] Calculating the three-dimensional coordinates of the measured object;

[0013] Aligning the depth map with the RGB image of the measured object and mapping them into three-dimensional coordinates;

[0014] Through the relationship between the depth map and the horizontal and vertical coordinates and the depth value under the three-dimensional coordinates, the point cloud in the target area is screened and the non-target area is eliminated to obtain the three-dimensional point cloud of the measured object.

[0015] Optionally, the step of acquiring the depth information value of the depth map of the measurement object includes:

[0016]

[0017] Where Z is the depth value of the measured object, T is the distance between the two camera image collectors; xl is the field of view width of the left camera at the lens, and xr is the field of view width of the right camera at the lens;

[0018] Let d represent the disparity value of the two cameras, that is:

[0019] d = xl-xr

[0020] Then the depth value of the spatial point is:

[0021]

[0022] Optionally, the relationship between the coordinates and the depth value of the three-dimensional coordinates of the measured object is calculated as follows:

[0023]

[0024] Wherein, X, Y, and Z represent the coordinates of the measured object in three dimensions, depth represents the depth value in the depth map, and x and y represent the coordinates of the depth map.

[0025] Optionally, aligning the depth map with the RGB image of the measured object and mapping them to three-dimensional coordinates comprises the following steps:

[0026] [X,Y,Z] is the coordinate of the 3D point cloud, [x z ,y z ,z z ] is the point cloud coordinate in the left camera coordinate system, R z is the rotation matrix from the 3D coordinate system to the left camera, T z is the translation matrix from the three-dimensional coordinate system to the left camera:

[0027]

[0028] Convert the point cloud in the left camera coordinate system to the point cloud in the RGB camera coordinate system, and then map it to the RGB image coordinates to normalize the three-dimensional space:

[0029]

[0030] Dedistortion, where r 2 =x2 +y 2 , k1, k2, k3 represent the radial distortion of the RGB camera, and p1, p2 represent the tangential distortion of the RGB camera:

[0031] x distorted =x(1+k1r 2 +k2r 4 +k3r 6 )+2p1xy+p2(r 2 +2x 2 );

[0032] y distorted =y(1+k1r 2 +k2r 4 +k3r 6 )+2p1xy+p2(r 2 +2y 2 );

[0033] The position of the point on the image is obtained by projecting the intrinsic parameter matrix onto the pixel plane, so that the depth map and the RGB map are aligned. The optical center and focal length of the left camera are (c x ,c y ) and (f x ,f y ):

[0034] u=f x x distorted +c x ;

[0035] v=f y y distorted +c y ;

[0036] The steps to obtain the three-dimensional shape of the material using the RGB image recognition results and the depth map are as follows:

[0037] po int.x=(x roi -c x )*depth roi ÷f x ;

[0038] po int.y=(y roi -c y )*depth roi ÷f y ;

[0039] po int.z=depth roi ;

[0040] Among them, point is the three-dimensional point cloud of the target material, x roi androi They are the areas of the target material on the RGB image obtained after recognition by the neural network.

[0041] Beneficial effects of the invention: The invention realizes intelligent identification and determination of auxiliary material types, intelligent calculation and recording of auxiliary material quantities, and real-time data transmission and display functions. Compared with the traditional system, the invention optimizes the auxiliary material warehousing process, improves inventory efficiency, reduces unnecessary working time, reduces labor costs, improves production and supply efficiency, and provides strong support for the intelligent management of the packaging workshop site. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the contents of the embodiments of the present invention and these drawings without paying any creative work.

[0043] Figure 1 It is a schematic diagram of the relationship between binocular stereo vision and depth of a method for acquiring the three-dimensional shape of auxiliary materials for warehousing of cigarette auxiliary materials according to the present invention;

[0044] Figure 2 It is a device diagram of a method for acquiring the three-dimensional shape of auxiliary materials for warehousing of cigarette auxiliary materials according to the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0046] In the description of the present invention, unless otherwise clearly specified and limited, the terms "connected", "connected", and "fixed" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0047] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0048] In the description of this embodiment, the terms "upper", "lower", "left", "right" and other directions or positional relationships are based on the directions or positional relationships shown in the drawings, and are only for the convenience of description and simplification of operation, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operate in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are only used to distinguish in the description and have no special meaning.

[0049] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0050] like Figure 1 and Figure 2 As shown, the present invention uses binocular stereo vision + structured light to achieve the acquisition of the three-dimensional shape of the object to be measured, and the collected RGB image and category segmentation results can be mapped to the three-dimensional model through the depth map. The present invention provides a method for acquiring the three-dimensional shape of auxiliary materials for warehousing of cigarette auxiliary materials, and the three-dimensional shape acquisition method includes:

[0051] A projector is used to project a preset grating coding pattern onto the surface of the measurement object;

[0052] Two cameras are respectively arranged on both sides of the projector, and the two cameras synchronously collect the grating coding pattern on the surface of the measurement object;

[0053] The two cameras respectively perform phase unpacking algorithms on the grating coding pattern, use phase information to match the binocular vision of the two cameras, and reconstruct a three-dimensional point cloud of the measured object through binocular vision reconstruction. The two cameras constitute a binocular stereo vision camera.

[0054] The step of reconstructing a three-dimensional point cloud of the measured object comprises:

[0055] Acquiring a depth map of the measurement object;

[0056] Calculate the horizontal and vertical coordinates and depth value of the measured object under three-dimensional coordinates;

[0057] Aligning the depth map with the RGB image of the measured object and mapping them into three-dimensional coordinates;

[0058] Through the relationship between the depth map and the horizontal and vertical coordinates and the depth value under the three-dimensional coordinates, the point cloud in the target area is screened and the non-target area is eliminated to obtain the three-dimensional point cloud of the measured object.

[0059] like Figure 1 As shown in the figure, after the general binocular stereo vision camera is calibrated, the left camera lens Cl and the right camera lens Cr are at the midpoint of the image plane, so the distance between Cl and Cr and the camera optical axis is equal. According to the principle of similar triangles, we can get:

[0060]

[0061] Where Z is the depth value of the measured object, T is the distance between the binocular stereo vision camera image collectors, xl is the field of view width of the left camera at the lens, and xr is the field of view width of the right camera at the lens;

[0062] Use d to represent the disparity value of the binocular stereo vision camera, that is:

[0063] d = xl - xr;

[0064] Then the depth value of the spatial point P is:

[0065]

[0066] After obtaining the depth information of each point, the depth map can be calculated. The specific calculation is as follows:

[0067] x=(uc x ) / f x

[0068] y=(vc y ) / f y ;

[0069] Among them, c x , c y Respectively represent the horizontal and vertical coordinates of the optical center position of the left camera; f x , f y They respectively represent the focal length of the left camera in the x-axis direction and the focal length in the y-axis direction, both in pixels; u and v represent the pixel coordinates of the depth map.

[0070] Calculating the horizontal and vertical coordinates and the depth value of the measured object under the three-dimensional coordinates includes the following steps:

[0071]

[0072] X, Y, and Z represent the coordinates of the measured object in three dimensions, depth represents the depth value in the depth map, and x and y represent the coordinates of the depth map.

[0073] In the above process, the RGB image and depth map of the object to be measured have been obtained. In this step, the 3D point cloud information of the material needs to be calculated based on the RGB image and the depth map. In order to make the coordinate information in the depth map completely correspond to the coordinate information in the RGB image, the depth map and the RGB image need to be aligned. The alignment steps are as follows:

[0074] [X,Y,Z] is the coordinate of the 3D point cloud, [x z ,y z ,z z ] is the point cloud coordinate in the left camera coordinate system, R z is the rotation matrix from the 3D coordinate system to the left camera, T z is the translation matrix from the three-dimensional coordinate system to the left camera:

[0075]

[0076] Convert the point cloud in the left camera coordinate system to the point cloud in the RGB camera coordinate system, and then map it to the RGB image coordinates to normalize the three-dimensional space:

[0077]

[0078] Dedistortion, where r 2 =x 2 +y 2 , k1, k2, k3 represent the radial distortion of the RGB camera, and p1, p2 represent the tangential distortion of the RGB camera:

[0079]

[0080] The position of the point on the image is obtained by projecting the intrinsic parameter matrix onto the pixel plane, so that the depth map and the RGB map are aligned. The optical center and focal length of the left camera are (c x ,c y ) and (f x ,f y ):

[0081] u=f x x distorted +c x ;

[0082] v=f y y distorted +c y ;

[0083] The steps to obtain the three-dimensional shape of the material using the RGB image recognition results and the depth map are as follows:

[0084] po int.x=(x roi -c x )*depth roi ÷f x ;

[0085] po int.y=(y roi -c y )*depth roi ÷f y ;

[0086] po int.z=depth roi ;

[0087] Among them, point is the three-dimensional point cloud of the target material, x roi and roi They are the target material areas on the RGB image obtained by neural network recognition. By calculating with the above formula, the 3D point cloud of the target material can be obtained, eliminating the redundant information of the non-target area. In addition, the material category corresponding to each material area recognized by the RGB image also corresponds to the category of the 3D point cloud of the area.

[0088] By aligning the depth map with the RGB map, the object area and the corresponding category can be mapped from the RGB map to the depth map, and through the relationship between the depth map and the 3D point cloud, the point cloud in the target area can be screened and the non-target area can be eliminated. In this way, we can obtain the 3D point cloud of the target material and use it in the subsequent quantity calculation.

[0089] The method for identifying the types of cigarette auxiliary materials entering the warehouse specifically includes:

[0090] Step 1: Input and convolution operation of auxiliary RGB 3D image data. The convolution operation uses the maximum pooling layer as the standard for pixel value extraction, which can maximize the retention of important features in the convolution area and ensure the accuracy of subsequent calculations.

[0091] Step 2: Cascade operation and matrix calculation. The cascade operation is to comprehensively consider the features of each convolution layer, rather than only calculating based on the final convolution layer results. This effectively avoids the problem of information loss during the convolution process. Matrix calculation is to calculate the predicted box in the image data, generally represented by (x, y, w, h) and the category of the actual box.

[0092] In addition, in order to make auxiliary material identification more robust, during the training process, we used the Task-Aligned Assigner to perform a dynamic allocation strategy, dynamically adjusting the weights according to the progress of training and the characteristics of the samples. Because it may be difficult to distinguish between positive and negative samples in the early stages of training, more attention should be paid to samples that are easily misclassified. As training progresses, the samples are better distinguished, so the weights of difficult samples should be gradually reduced, while the weights of easy-to-classify samples should be increased. The dynamic allocation strategy can be adjusted according to training loss or other indicators to better adapt to different data sets and models. The formula is as follows:

[0093] t=s α +u β ;

[0094] Among them, α and β are weight hyperparameters. s is the predicted score corresponding to the labeled category, and u is the IoU between the predicted box and the actual box. The sum of the two can measure the degree of alignment. t can simultaneously control the optimization of the classification score and IoU to achieve the degree of alignment, thereby guiding the network to dynamically focus on high-quality anchors. The higher the category score and the higher the IoU, the closer the value of t is to 1.

[0095] The specific steps are as follows:

[0096] 1. Based on the classification score and the IoU between the predicted box and the actual box, a loss function of the IoU of the associated classification and regression is obtained by weighting. The IoU loss function here is as follows:

[0097]

[0098] Among them, IoU represents the intersection-over-union ratio of the actual box and the predicted box, which is calculated as follows, where A and B represent the actual box and the predicted box respectively:

[0099]

[0100] b, b gt Represent the center points of the predicted box and the actual box respectively, and ρ represents the Euclidean distance between the two center points. c represents the diagonal distance of the minimum closure area that can contain both the predicted box and the actual box. α is the weight function, and v is used to measure the similarity of the aspect ratio, which is defined as:

[0101]

[0102] w, w gt Represent the predicted width value and the actual width value, h, h gt They represent the predicted height and the true height respectively.

[0103] 2. Calculate whether the center point of the anchor point is within the current actual box. Only the anchor point value within the current actual box can be used as a positive sample.

[0104] 3. Under the premise of meeting 2, the alignment score IoU within the actual frame is used as a positive sample, and the rest are used as negative samples for training.

[0105] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, various obvious changes, readjustments and substitutions can be made without departing from the scope of protection of the present invention. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for obtaining the three-dimensional shape of auxiliary materials for warehousing of cigarette auxiliary materials, characterized in that: The three-dimensional shape acquisition method comprises: A projector is used to project a preset grating coding pattern onto the surface of the measurement object; Two cameras are arranged axially symmetrically on both sides of the projector, and the two cameras synchronously collect the grating coding pattern on the surface of the measurement object; The two cameras respectively perform phase unpacking algorithms on the grating coding pattern, use the phase information to match the binocular vision of the two cameras, and reconstruct a three-dimensional point cloud of the measured object through binocular vision reconstruction.

2. A method for obtaining the three-dimensional shape of auxiliary materials for warehousing of cigarette auxiliary materials according to claim 1, characterized in that: The step of reconstructing a three-dimensional point cloud of the measured object comprises: Acquiring a depth map of the measurement object; Calculating the three-dimensional coordinates of the measured object; Aligning the depth map with the RGB image of the measured object and mapping them into three-dimensional coordinates; Through the relationship between the depth map and the horizontal and vertical coordinates and the depth value under the three-dimensional coordinates, the point cloud in the target area is screened and the non-target area is eliminated to obtain the three-dimensional point cloud of the measured object.

3. The method for obtaining the three-dimensional shape of auxiliary materials for warehousing of cigarette auxiliary materials according to claim 1, characterized in that: The step of obtaining the depth information value of the depth map of the measured object comprises: Where Z is the depth value of the measured object, T is the distance between the two camera image collectors; xl is the field of view width of the left camera at the lens, and xr is the field of view width of the right camera at the lens; Let d represent the disparity value of the two cameras, that is: d = xl-xr Then the depth value of the spatial point is:

4. The method for obtaining the three-dimensional shape of auxiliary materials for warehousing of cigarette auxiliary materials according to claim 1, characterized in that: The relationship between the coordinates and depth values ​​of the three-dimensional coordinates of the measured object is calculated as follows: Wherein, X, Y, and Z represent the coordinates of the measured object in three dimensions, depth represents the depth value in the depth map, and x and y represent the coordinates of the depth map.

5. The method for obtaining the three-dimensional shape of auxiliary materials for warehousing of cigarette auxiliary materials according to claim 4, characterized in that: The depth map is aligned with the RGB image of the measured object and mapped to three-dimensional coordinates, comprising the following steps: [X,Y,Z] is the coordinate of the 3D point cloud, [x z ,y z ,z z ] is the point cloud coordinate in the left camera coordinate system, R z is the rotation matrix from the 3D coordinate system to the left camera, T z is the translation matrix from the three-dimensional coordinate system to the left camera: Convert the point cloud in the left camera coordinate system to the point cloud in the RGB camera coordinate system, and then map it to the RGB image coordinates to normalize the three-dimensional space: Dedistortion, where r 2 =x 2 +y 2 , k1, k2, k3 represent the radial distortion of the RGB camera, and p1, p2 represent the tangential distortion of the RGB camera: x distorted =x(1+k1r 2 +k2r 4 +k3r 6 )+2p1xy+p2(r 2 +2x 2 ) <h2 style=";text-align:left;direction:ltr">y<h2 style=";text-align:left;direction:ltr"> distorted <h2 style=";text-align:left;direction:ltr"> =y(1+k1r<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> +k2r<h2 style=";text-align:left;direction:ltr"> 4 <h2 style=";text-align:left;direction:ltr"> +k3r<h2 style=";text-align:left;direction:ltr"> 6 <h2 style=";text-align:left;direction:ltr"> )+2p1xy+p2(r<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> +2y<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> ) The position of the point on the image is obtained by projecting the intrinsic parameter matrix onto the pixel plane, so that the depth map and the RGB map are aligned. The optical center and focal length of the left camera are (c x ,c y ) and (f x ,f y ): u=f x x distorted +c x v=f y y distorted +c y The steps to obtain the three-dimensional shape of the material using the RGB image recognition results and the depth map are as follows: po int.x=(x roi -c x )*depth roi ÷f x po int.y=(y roi -c y )*depth roi ÷f y int.z=depth roi Among them, point is the three-dimensional point cloud of the target material, x roi and roi They are the areas of the target material on the RGB image obtained after recognition by the neural network.

Images