A method for locating the storage of file boxes based on a depth camera and vision algorithms

Abstract

The present invention provides a positioning method for storing file boxes based on a depth camera and a vision algorithm, which includes collecting a scene image of a file rack through the depth camera; performing semantic segmentation processing on the scene image to obtain pixel point sets of the background, file boxes, bin boundaries, and bin codes in the scene image; obtaining the connected components of the pixel point sets of the file boxes, bin boundaries, and bin codes; pairing between the respective connected component sets, and determining the occupancy and idle status of the bins and the scene information on whether the bin positions are available according to the pairing results; obtaining the spatial positions of the target file box and the bin code to be retrieved or placed, and using a grasping mechanism to grasp or store the target file box according to the scene information. The present invention uses a depth camera to collect the image of the file rack, determines the positional relationship between the file boxes, bin boundaries, and bin codes in the image, so as to obtain the idle and occupied states of the file rack.

Description

Technical Field

[0001] The present invention relates to the technical field of file box access, and specifically to a method for locating the storage of file boxes based on a depth camera and a vision algorithm. Background Art

[0002] Paper files need to go through processes such as circulation, association, management, and storage before being closed and archived; with the development of automation technology, the archiving process of files can be managed by AGV equipment; this process involves grasping and storing file boxes from the file rack, and how to determine the location, identity of the files, and the positions of the file rack compartments is the core of automated management.

[0003] Currently, most of the technical solutions for automatically managing files by AGV are to paste two-dimensional codes on the file boxes and the positions of the file rack compartments, and cooperate with a code-reading camera to identify the two-dimensional codes to determine the positions of the file boxes and the compartments; the AGV can use this position information to perform the grasping and storing work of the file boxes; however, the pure two-dimensional code positioning scheme cannot judge the occupancy status of the compartments; in addition, the two-dimensional code reading camera cannot obtain the depth information of the scene, and the acquisition of the spatial positions of the file boxes and the compartments is not accurate enough, resulting in the need for manual determination of the idle status of the compartments in the current automated file management technology, and full automation cannot be achieved. At the same time, it is time-consuming and laborious to manually search and determine the idle status of the compartments. Summary of the Invention

[0004] Aiming at the deficiencies of the existing technology, the purpose of the present invention is to provide a method for locating the storage of file boxes based on a depth camera and a vision algorithm, which can solve the problems that the existing AGV can use this position information to perform the grasping and storing work of the file boxes; however, the pure two-dimensional code positioning scheme cannot judge the occupancy status of the compartments; in addition, the two-dimensional code reading camera cannot obtain the depth information of the scene, and the acquisition of the spatial positions of the file boxes and the compartments is not accurate enough.

[0005] In order to achieve the above purpose, the technical solution of the present invention is as follows:

[0006] The present invention is implemented through the following technical solutions: A method for locating the storage of file boxes based on a depth camera and a vision algorithm,

[0007] A method for locating the storage of file boxes based on a depth camera and a vision algorithm includes

[0008] Collecting a scene image of the file rack through a depth camera;

[0009] Performing semantic segmentation processing on the scene image to obtain a set of pixel points of the background, file boxes, compartment boundaries, and compartment codes in the scene image;

[0010] Find the connected components of the pixel point sets of the file box, bin boundary, and bin code to obtain the connected component sets of the file box, bin boundary, and bin code;

[0011] Pair the connected component sets with each other, and determine the occupancy and idle states of the bins and the scene information on whether the bin positions are available according to the pairing results;

[0012] Obtain the spatial positions of the target file box and the bin code to be picked up or placed, and use the grasping mechanism to grasp or store the target file box according to the scene information.

[0013] Further, the semantic segmentation processing of the scene image includes:

[0014] Perform semantic segmentation, convolution, normalization, and non-linear transformation processing on the scene image;

[0015] Classify each pixel of each scene image into four categories: background, file box, bin boundary, and bin code.

[0016] Further, the segmentation formula for semantic segmentation of the scene image is:

[0017] ConvBN() = BN(Conv()) (1)

[0018] Block = ConvBN3(RELU(ConvBN2(RELU(ConvBN1))) (2)

[0019] P = softmax(Block n (Block n-1 (…(Block1(I))))) (3)

[0020] where P = {P c＝backgroud ,P c＝box ,P c＝edge , P c ＝digit} (4);

[0021] In the above formula (1), ConvBN() represents the cascaded structure of the convolution operation Conv() and the batch normalization BN();

[0022] Formula (2) indicates that each Block is composed of 3 cascaded ConvBN() structures, and RELU() represents the non-linear transformation function RELU;

[0023] Formula (3) indicates that the network structure is composed of multiple cascaded Block(), where softmax() represents the softmax operation on the output of the network, and I represents the file box scene image;

[0024] Equation (4) represents the output of the network structure, where P represents a set of points P with the same number of pixels as the input image pixels. c＝ b ackgroud , P c＝box , P c＝edge , P c＝digit respectively represent the sets of pixels of the background, file box, bin boundary, and bin code in the point set output by the network that belong to the file box scene image.

[0025] Furthermore, obtaining the connected component sets of the file box, bin boundary, and bin code includes:

[0026] By using the prior knowledge of the sizes of the file box, bin code, and bin boundary in the real scene, excluding the connected regions that do not meet the requirements, the connected component sets of the file box, bin boundary, and bin code are obtained.

[0027] Furthermore, pairing the connected component sets with each other and determining the occupancy and idle states of the bins and the scene information of whether the bin positions are available according to the pairing results includes:

[0028] Pair the connected component sets of the file box and the bin code.

[0029] Judge whether the overlap degree of the circumscribed rectangles of the two connected components of the file box and the bin code in the vertical direction exceeds the threshold; if so, the paired bin is in the occupied state; otherwise, the paired bin is in the idle state.

[0030] Pair the connected component sets of the bin boundary and the bin code.

[0031] Judge whether the absolute value of the distance between the circumscribed rectangles of the two connected components of the bin boundary and the bin code in the horizontal direction is less than the threshold; if so, the paired bin position is unavailable; otherwise, the paired bin position is available.

[0032] Furthermore, obtaining the spatial positions of the target file box and the bin code to be picked up and placed includes:

[0033] Each file box surface is provided with a QR code, and the depth camera is used to identify the QR code to obtain the target file box.

[0034] Use the depth camera to obtain the pixel coordinate positions of the bin code to be stored and the center of the connected region of the target file box, combined with the depth information of the depth camera and the internal and external parameter information of the camera.

[0035] Obtain the spatial positions of the target file box and the bin code to be picked up and placed.

[0036] Further, the grasping or storing of the target file box according to the scene information includes:

[0037] When grasping the target file box, the three-axis module is used to control the grasping mechanism to take out the target file box from the bin corresponding to the bin code to be retrieved or placed.

[0038] Or when storing the target file box, the three-axis module is used to control the grasping mechanism to store the target file box in the bin corresponding to the bin code to be retrieved or placed.

[0039] Further, the acquisition of the scene image of the file rack by the depth camera includes: connecting the depth camera to the three-axis module and driving the depth camera to move through the three-axis module to acquire the scene information of the file rack.

[0040] Compared with the prior art, the beneficial effects of the present invention include:

[0041] The file box storage positioning method of the present invention uses a depth camera to acquire the image of the file rack, performs semantic segmentation on the file rack image through a convolutional neural network, determines the positional relationship between the file box, bin boundary, and bin code in the image, thereby obtaining the free and occupied states of the file rack, and obtains the accurate spatial positions of the file box and the bin of the file rack through the depth information captured by the depth camera; making the storage of the file box more accurate, and at the same time, through the recognition of the depth camera and the bin state, further reducing the manpower and improving the efficiency of file box storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The disclosure of the present invention will be described with reference to the accompanying drawings. It should be understood that the drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. In the drawings, the same reference numerals are used to refer to the same components. Among them:

[0043] Figure 1 is the overall flow schematic diagram of a file box storage positioning method based on a depth camera and a vision algorithm of the present invention;

[0044] Figure 2 is the overall structural schematic diagram of the file rack in the embodiment of the present invention;

[0045] Figure 3 is the structural schematic diagram of semantic segmentation and pairing in the embodiment of the present invention;

[0046] The reference numerals in the figure are explained as follows: 1. File box; 2. QR code; 3. Bin boundary; 4. Bin code. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0047] It is easy to understand that according to the technical solution of the present invention, without changing the essential spirit of the present invention, those of ordinary skill in the art can propose various interchangeable structural forms and implementation methods. Therefore, the following specific embodiments and drawings are only exemplary descriptions of the technical solution of the present invention, and should not be regarded as the whole of the present invention or as a limitation or restriction on the technical solution of the present invention.

[0048] The present invention provides a method for positioning the storage of file boxes based on a depth camera and a vision algorithm. The method of the present invention is applicable to storing a file box on a file rack or grasping a corresponding document box from within the file rack and removing the file box from the file rack; the overall structure of the file rack of the present invention is as Figure 2 shown, in which a plurality of bins are provided on the file rack, and bin boundaries 3 are provided on both sides of each bin. Two bin boundaries 3 can form a bin for storing a file box 1. Each bin corresponds to a bin code 4. Exemplarily, each bin is encoded by an Arabic numeral, and each digit encodes a bin to achieve the uniqueness of the bin and the digit; some of the bins store file boxes 1, and a QR code 2 is provided on the surface of the file box 1 therein. The QR code 2 is used to record the information of the corresponding file box 1; the method for storing the file box of the present invention is as Figure 1 shown, and the positioning method includes the following steps:

[0049] Step 1: Collect the scene image of the file rack through a depth camera;

[0050] Specifically, connect the depth camera to a three-axis module, and drive the depth camera to move through the three-axis module to collect the scene information of the file rack;

[0051] Step 2: Perform semantic segmentation processing on the scene image to obtain the pixel point sets of the background, file box, bin boundary, and bin code in the scene image;

[0052] Specifically, the semantic segmentation processing of the scene image includes:

[0053] 2.1. Perform semantic segmentation, convolution, normalization, and non-linear transformation processing on the scene image;

[0054] 2.2. Classify each pixel of each scene image into four categories: background, file box, bin boundary, and bin code; <{

[0055] Exemplarily, use a convolutional neural network based on MobileNet v3 to perform semantic segmentation on the file rack scene as Figure 2 shown, and the segmentation formula for semantic segmentation is:

[0056] ConvBN() = BN(Conv()) (1)

[0057] Block = ConvBN3(RELU(ConvBN2(RELU(ConvBN1)))) (2)

[0058] P = softmax(Block n (Block n-1 (…(Block1(I))))) (3)

[0059] where P = {P c＝backgroud , P c＝box , P c＝edge , P c＝digit} (4)

[0060] In the above formula (1), ConvBN() represents the cascaded structure of the convolution operation Conv() and the batch normalization BN();

[0061] Formula (2) indicates that each Block is composed of the cascading of 3 ConvBN() structures, and RELU() represents the non-linear transformation function RELU;

[0062] Formula (3) indicates that the network structure is composed of the cascading of multiple Block(), where softmax() represents performing the softmax operation on the output of the network, and I represents the image of the archive box scene;

[0063] Formula (4) represents the output of the network structure, where P represents the point set P with the same number of pixels as the input image c＝backgroud , P c＝box , P c＝edge , P c＝digit respectively represent the sets of pixels belonging to the background, archive box, bin boundary, and bin code in the point set of the network output for the archive box scene image;

[0064] The segmented archive box scene image I is subjected to a series of convolution, normalization, and non-linear transformation processes through a neural network, and finally each pixel P x，y is respectively classified into four categories: the background, archive box, bin boundary, and bin code of the archive box scene image; four pixel point sets are output, including the pixel point set P of the background c＝backgroud , the pixel point set P of the archive box c＝box , the pixel point set P of the bin boundary c＝edge , and the pixel point set P of the bin code c＝digit .

[0065] Step 3: Obtain the connected components of the pixel point sets of the archive box, bin boundary, and bin code;

[0066] Specifically, solving the connected components means solving the set of adjacent pixel points with the same pixel value in the pixel set. The solution of the connected components can be achieved through the morphological algorithm Seed Filling. The purpose of using the connected components solution is to obtain, from the pixel set P c＝box , P c＝edge , P c＝digit , the connected component sets B, D, and E of the file box, the bin boundary, and the bin code, respectively. Each connected component in B, D, and E corresponds to each recognized file box, bin boundary, and bin code;

[0067] The obtaining of the connected component sets of the file box, the bin boundary, and the bin code further includes: excluding the connected regions that do not meet the size requirements through the prior knowledge of the sizes of the file box, the bin code, and the bin boundary in the real scene, and obtaining the connected component sets of the file box, the bin boundary, and the bin code.

[0068] Step 4: Pair the connected component sets with each other, and determine the occupancy and idle states of the bins and the scene information of whether the bin positions are available according to the pairing results;

[0069] The pairing of the connected component sets with each other and the determination of the occupancy and idle states of the bins and the scene information of whether the bin positions are available according to the pairing results include:

[0070] 4.1 Pair the connected component sets of the file box and the bin code,

[0071] 4.2 Determine whether the vertical overlap degree of the circumscribed rectangles of the two connected components of the file box and the bin code exceeds the threshold;

[0072] If so, the paired bin is in the occupied state; otherwise, the paired bin is in the idle state;

[0073] 4.3 Pair the connected component sets of the bin boundary and the bin code,

[0074] 4.4 Determine whether the absolute value of the horizontal distance between the circumscribed rectangles of the two connected components of the bin boundary and the bin code is less than the threshold;

[0075] If so, the position of the paired bin is unavailable; otherwise, the position of the paired bin is available.

[0076] Exemplarily,

[0077] Set the collection of file boxes, bin codes, and connected components of bin boundaries as B, E, and D respectively; Matched-BD is the set where the file box and bin code are successfully paired; Matched-ED is the set where the bin boundary and bin code are successfully paired, minRec() represents solving the minimum bounding rectangle of the connected component, and IOU v () represents solving the vertical overlap degree of two bounding rectangles, and dist h () represents solving the horizontal distance between two bounding rectangles.

[0078] When the IOU of the file box and bin code in the vertical direction v exceeds the threshold iou-threshold, pair the bin code and the file box; when the absolute value of the horizontal distance dist h between the bin boundary and the bin code is less than dist-threshold, pair the bin boundary and the bin code, where if dist h is negative, the bin boundary is the left boundary of the current bin code, otherwise, it is the right boundary of the current bin code; one bin code corresponds to one file box and two bin boundaries; the pairing algorithm is as follows:

[0079] Input: B, D, E

[0080] Output: Matched_BD, Matched_ED

[0081]

[0082] As Figure 3 shown, determine the occupied and idle states of the bin and the scene information of whether the bin position is available according to the pairing result; after semantic segmentation of the file rack, obtain the file box segmentation result, bin code segmentation result, and bin boundary segmentation result; after the above pairing, if the file box and bin code are successfully paired, it means that there is a file box in the bin corresponding to the bin code, that is, the bin corresponding to the bin code is in the occupied state; if there is no file box but there is a file code, it means that the bin corresponding to this bin code is in the idle state; judge whether the bin is an available bin by matching the bin boundary and the bin code, if so, it is an available bin, if not, the bin corresponding to the bin code is unavailable.

[0083] Step Five: Obtain the spatial positions of the target file box and the bin code to be picked up or placed, and use the grasping mechanism to grasp or place the target file box according to the scene information.

[0084] Specifically; the obtaining of the spatial positions of the target file box and the bin code to be picked up or placed includes,

[0085] Each file box has a QR code on its surface, and a depth camera is used to identify the QR code to obtain the target file box;

[0086] Use a depth camera to obtain the bin code to be stored and the pixel coordinate position of the center of the connected area of the target file box, and combine the depth information of the depth camera, the internal and external parameter information of the camera;

[0087] Obtain the spatial positions of the target file box and the bin code to be retrieved or placed.

[0088] The grabbing or storing of the target file box by using the grabbing mechanism according to the scene information includes:

[0089] When grabbing the target file box, the grabbing mechanism is controlled by a three-axis module to take out the target file box from the bin corresponding to the bin code to be retrieved or placed;

[0090] Or when storing the target file box, the grabbing mechanism is controlled by a three-axis module to store the target file box in the bin corresponding to the bin code to be retrieved or placed.

[0091] Implementation case

[0092] When storing a file box in a file rack, use a depth camera to collect the scene image of the file rack; perform semantic segmentation processing on the scene image to obtain the pixel point sets of the background, file box, bin boundary, and bin code in the scene image; obtain the connected components of the pixel point sets of the file box, bin boundary, and bin code to get the connected component set of the file box, bin boundary, and bin code; pair the connected component sets with each other, and determine the occupancy and idle status of the bin and the scene information of whether the bin position is available according to the pairing result; obtain the spatial positions of the target file box (obtained by QR code recognition) and the bin code to be stored, and use the grabbing mechanism to store the target file box in the bin corresponding to the bin code to be stored according to the scene information;

[0093] When grabbing a file box from a file case, use a depth camera to collect the scene image of the file rack; perform semantic segmentation processing on the scene image to obtain the pixel point sets of the background, file box, bin boundary, and bin code in the scene image; obtain the connected components of the pixel point sets of the file box, bin boundary, and bin code to get the connected component set of the file box, bin boundary, and bin code; pair the connected component sets with each other, and determine the occupancy and idle status of the bin and the scene information of whether the bin position is available according to the pairing result; obtain the spatial positions of the target file box (obtained by QR code recognition) and the bin code to be retrieved or placed, and use the grabbing mechanism to grab the target file box according to the scene information so that it is removed from the file case.

[0094] The present invention uses a depth camera to collect images of file racks, performs semantic segmentation on the file rack images through a convolutional neural network, determines the positional relationships of various parts such as file boxes, bin boundaries, and bin codes in the images, thereby obtaining the free and occupied states of the file racks, and obtains the accurate spatial positions of the volume file boxes and the bins of the file racks through the depth information captured by the depth camera; making the storage of file boxes more accurate, and at the same time, through the recognition of the depth camera and the bin status, further reducing the manpower and improving the efficiency of file box storage

[0095] The technical scope of the present invention is not limited to the content described above. Those skilled in the art can make various deformations and modifications to the above embodiments without departing from the technical idea of the present invention, and these deformations and modifications should all fall within the protection scope of the present invention.

Claims

1. A positioning method for storing file boxes based on a depth camera and a vision algorithm, characterized in that: Including Collecting the scene image of the archive rack through a depth camera; Performing semantic segmentation processing on the scene image to obtain the pixel point sets of the background, archive box, bin boundary, and bin code in the scene image; Calculating the connected components of the pixel point sets of the archive box, bin boundary, and bin code to obtain the connected component sets of the archive box, bin boundary, and bin code; Pairing between the connected component sets, and determining the occupancy and idle states of the bins and the scene information on whether the bin positions are available according to the pairing results; Obtaining the spatial positions of the target archive box and the bin code to be retrieved or placed, and using a grasping mechanism to grasp or store the target archive box according to the scene information; Among them, the performing semantic segmentation processing on the scene image includes: Performing semantic segmentation, convolution, normalization, and non-linear transformation processing on the scene image; Classifying each pixel of each scene image into four categories: background, archive box, bin boundary, and bin code; The segmentation formula for performing semantic segmentation on the scene image is: ConvBN() = BN(Conv()) (1) Block = ConvBN3(RELU(ConvBN2(RELU(ConvBN1)))) (2) P = softmax(Block n (Block n-1 (…(Block1(I))))) (3) where P = {P c＝backgroud , P c＝box , P c＝edge , P c＝digit}} (4); In formula (1), ConvBN() represents the cascaded structure of the convolution operation Conv() and the batch normalization BN(); Formula (2) indicates that each Block is composed of 3 cascaded ConvBN() structures, and RELU() represents the non-linear transformation function RELU; Formula (3) indicates that the network structure is composed of multiple cascaded Block(), where softmax() represents performing a softmax operation on the output of the network, and I represents the archive box scene image; Equation (4) represents the output of the network structure, where P represents a set of points with the same number as the number of pixels in the input image, P c＝backgroud , P c＝box , P c＝edge , P c＝digit respectively represent the sets of pixel points belonging to the background, file box, bin boundary, and bin code in the file box scene image in the set of points output by the network; Among them, the pairing between the connected component sets and determining the occupancy and idle states of the bins and the scene information on whether the bin positions are available according to the pairing results includes: Pairing the connected component sets of the archive box and the bin code, Judging whether the overlap degree of the circumscribed rectangles of the connected components of the archive box and the bin code in the vertical direction exceeds the threshold; if so, the paired bin is in an occupied state; otherwise, the paired bin is in an idle state; Pairing the connected component sets of the bin boundary and the bin code, Judging whether the absolute value of the distance between the circumscribed rectangles of the connected components of the bin boundary and the bin code in the horizontal direction is less than the threshold; if so, the paired bin position is unavailable; otherwise, the paired bin position is available.

2. The method for positioning the storage of file boxes based on a depth camera and a vision algorithm according to claim 1, wherein: The obtaining the connected component sets of the archive box, bin boundary, and bin code includes: Excluding the connected regions that do not meet the size requirements through the prior knowledge of the sizes of the archive box, bin code, and bin boundary in the real scene to obtain the connected component sets of the archive box, bin boundary, and bin code.

3. A method for positioning the storage of file boxes based on a depth camera and a vision algorithm according to claim 1, characterized in that: The obtaining the spatial positions of the target archive box and the bin code to be retrieved or placed includes: QR codes are provided on the surfaces of each archive box, and the depth camera is used to identify the QR codes to obtain the target archive box; The depth camera is used to obtain the bin code to be stored and the pixel coordinate position of the center of the connected area of the target file box, and combine the depth information of the depth camera and the internal and external parameter information of the camera; Obtain the spatial positions of the target file box and the bin code to be picked up or placed.

4. A method for positioning the storage of file boxes based on a depth camera and a vision algorithm according to claim 1, characterized in that: The grabbing or storing of the target file box by using the grabbing mechanism according to the scene information includes: When grabbing the target file box, the grabbing mechanism is controlled by the three-axis module to take out the target file box from the bin corresponding to the bin code to be picked up or placed; Or when storing the target file box, the grabbing mechanism is controlled by the three-axis module to store the target file box in the bin corresponding to the bin code to be picked up or placed.

5. A method for positioning the storage of file boxes based on a depth camera and a vision algorithm according to any one of claims 1-4, characterized in that: The scene image of the file rack is collected by the depth camera, including: connecting the depth camera to the three-axis module, and driving the depth camera to move through the three-axis module to collect the scene information of the file rack.

Images