A storage location detection method and system based on RGB camera

Abstract

The present invention discloses a storage location detection method and system based on an RGB camera, which comprises at least the following steps: step S1: obtaining a current storage area image captured by an RGB camera according to a control instruction; wherein the storage area is an area composed of multiple storage locations, and the storage location outline is marked by one of the three primary colors of red, blue, and green that is different from the ground color; step S2: performing image processing on the current storage area image according to pre-stored storage area calibration information to map the corresponding storage area calibration information to obtain a map of the storage location to be detected; step S3: inputting the map of the storage location to be detected into a pre-trained network model for processing, obtaining the storage location status and outputting the detection result.

Description

Technical Field

[0001] The present invention relates to the technical field of mobile robots, and in particular to a storage location detection method and system based on an RGB camera. Background Art

[0002] Various mobile robots are currently being used in the intelligent automation industry and are playing a crucial role, particularly in logistics and warehousing. When robots are used to carry goods in a large number of centralized storage locations, there is a pressing challenge: how to improve the efficiency of picking, unloading, and handling, while maximizing storage space utilization and increasing the intelligence, safety, and reliability of robotic cargo handling. To address this issue, robots must know the availability of storage locations before they begin handling goods.

[0003] Therefore, in view of the defects of the existing technology, it is necessary to propose a technical solution to solve the technical problems of the existing technology. It can use RGB cameras to collect images of the warehouse area, realize the identification, coordinate calculation and segmentation of the warehouse locations through autonomous calibration, collect and enhance the data set based on on-site shooting, train the network model, load the network model to infer the warehouse location category, and obtain whether the status of the warehouse location is available. Summary of the Invention

[0004] In view of this, it is indeed necessary to provide a warehouse location detection method and system based on RGB cameras. When the robot starts to handle goods, it can detect whether the warehouse location status is available in real time and provide status input for the next planning of the dispatch end. The warehouse area image is collected by RGB camera, and the contours of the warehouse locations contained in the warehouse area are calibrated after being processed by the calibration module algorithm. The location contour points are extracted and segmented. When the detection task arrives, the calibration information is mapped to the current warehouse area map, the warehouse locations to be detected are segmented, the network model is loaded and reasoned, and the location status is obtained as "occupied" or "free", thereby ensuring the rationality and effectiveness of the subsequent loop of picking and unloading target warehouse location planning, ensuring that the robot avoids "occupying" warehouse locations during the handling task, and increasing the safety and reliability of the robot.

[0005] In order to solve the technical problems existing in the prior art, the technical solutions of the present invention are as follows:

[0006] A storage location detection method based on an RGB camera includes at least the following steps:

[0007] Step S1: According to the control instruction, an image of the current storage area captured by the RGB camera is obtained; wherein the storage area is an area consisting of multiple storage locations, and the storage location outlines are marked with one of the three primary colors of red, blue, and green that are different from the ground color;

[0008] Step S2: performing image processing on the current warehouse area image according to the pre-stored warehouse area calibration information to map the corresponding warehouse area calibration information to obtain a warehouse location map to be inspected;

[0009] Step S3: Input the warehouse location map to be detected into the pre-trained network model for processing, obtain the warehouse location status and output the detection result.

[0010] As a further improvement, in step S2, the reservoir area calibration information is obtained through the following steps:

[0011] Step S21: remove objects placed on the ground where the storage area is located, and use one of the three primary colors of red, blue, and green that are different from the ground color to mark the outline of the storage area so that the outline of the storage area has a certain contrast with the ground; on this basis, start the RGB camera to collect images of the storage area;

[0012] Step S22: pre-processing the storage area image;

[0013] Step S23: Separate the RGB three channels of the warehouse image, use one of the channels as the subsequent data input, perform threshold segmentation on it, highlight the edge pixels, extract the edge points, filter them, remove the mis-extracted and outlier edge points, and then calculate the minimum enclosing rectangle of the warehouse location. Segment all the warehouse areas in the warehouse according to the minimum enclosing rectangle, add the warehouse location ID and associate the corresponding warehouse location outline and background image, reconstruct the warehouse location outline, and complete the warehouse location calibration.

[0014] As a further improvement, in step S3, the pre-trained network model is obtained by the following steps:

[0015] Step S31: constructing a garage “vacant” enhanced dataset and an “occupied” enhanced dataset and adding corresponding labels;

[0016] Step S32: Build a MobileNetV3-Large network model and train the model using the dataset built in step S31;

[0017] Step S33: After training is completed, the network model is output.

[0018] As a further improvement, in step S2, if the calibration information of the current warehouse area is not found, the warehouse location calibration process is first executed to generate the warehouse area calibration information.

[0019] As a further improvement, in step S3, the state of each storage location is classified by a pre-trained network model to obtain a storage location detection result of "free" or "occupied".

[0020] As a further improvement, red floor tape is used to stick the outline of each storage location around the edge of the gray ground storage area. In step S23, the red channel is extracted as subsequent data input.

[0021] As a further improvement, in step S31, videos and images of empty storage locations and cargo are captured using a mobile phone. The videos are then cropped to obtain a preliminary dataset, which is then expanded and enhanced. This includes randomly adjusting the color, angle, and size of the empty storage locations. The "idle" enhanced dataset is randomly combined with cargo images to generate an "occupied" enhanced dataset. Labels are then added to the enhanced "idle" and "occupied" datasets.

[0022] As a further improvement, in step S33, after training the MobileNetV3-Large network model using the data set, a model weight file is generated.

[0023] As a further improvement, step S22 further includes the following steps:

[0024] Histogram equalization is performed on the reservoir map Ω, and the equalization result is filtered. First, a median filter is applied. This traverses the pixels in the image and sorts the grayscale values ​​of all pixels in the four neighborhoods of the center pixel. The median grayscale value of this sorted value is then selected to replace the grayscale value of the center element. Next, a 3*3 mean filter is applied. This traverses the pixels in the image and takes the average of all pixel values ​​in the neighborhood of the center pixel, replacing the center pixel. Finally, a Gaussian filter is applied with a Gaussian kernel size of 3*3. The formula is as follows:

[0025]

[0026] Among them, f(x,y) represents the pixel value at the coordinate (x,y), σ is the standard deviation, and the reservoir area Ω is obtained filter ;

[0027] The storage area image Ω filter Perform channel segmentation to obtain three single-channel grayscale images of R, G, and B. The pixel values ​​are between 0 and 255, with 0 being black and 255 being white. Use the R channel as data input and perform threshold segmentation operation with a threshold of 200. Traverse the R channel image and perform pixel transformation on the R channel library image according to the following formula:

[0028]

[0029] The pixel values ​​of the storage location edges with pixel values ​​greater than 200 are set to 255, and then the pixel values ​​are divided by 255 to obtain the binary storage area map Ω thres .

[0030] As a further improvement, step S23 further includes the following steps:

[0031] Step 3-1: Analyze Ω thres topological structure, find the outer boundary and hole boundaries in the reservoir area map. The outer boundary consists of outer boundary points, and the hole boundaries consist of hole boundary points. The outer boundary points and hole boundary points are defined as follows:

[0032]

[0033] where \(0\leq x \lt m\), \(0\leq y \lt n\), and \(m\) and \(n\) are the number of rows and columns of Ω thres Traverse Ω from left to right and top to bottom thres , set the flag bit NBD to represent the new boundary, and LNBD to represent the parent boundary. Initialize NBD to 1 for each new row traversed;

[0034] Step 3-2: When traversing to the first outer boundary point \((x,y)\), increment the flag bit NBD of \((x,y)\) by 1. Denote the pixel at \((x,y - 1)\) as \((x2,y2)\). Centered at \((x,y)\), rotate clockwise from \((x2,y2)\) to find the first non-zero pixel within the eight-neighborhood of \((x,y)\), denoted as \((x1,y1)\). If there is no non-zero pixel, set the flag bit to -NBD and jump to Step 3-4;

[0035] Step 3-3: Redefine the pixel point \((x1,y1)\) as \((x2,y2)\) and \((x,y)\) as \((x3,y3)\). Starting from \((x2,y2)\), rotate counterclockwise to find the first non-zero pixel point within the eight-neighborhood of \((x3,y3)\), denoted as \((x4,y4)\);

[0036] Step 3-4: If \(f(x3,y3 + 1)\) equals zero, set the value of \(f(x3,y3)\) to -NBD. If \(f(x3,y3 + 1)\) is not zero, set the value of \(f(x3,y3)\) to NBD. Otherwise, do not change the pixel value at \((x3,y3)\);

[0037] Step 3-5: Check if the counterclockwise rotation in Step 3-3 returns to the starting point, i.e., \((x4,y4)=(x,y)\). If it returns to the starting point, it proves that the retrieval of the pixel \((x,y)\) is complete, and jump to Step 3-6. Otherwise, jump to Step 3-3 to continue the counterclockwise rotation to find non-zero elements;

[0038] Step 3-6: If the flag bit at \((x,y)\) is not equal to 1, then set the LNBD at that location to \(|NBD|\). Move the central pixel to the right. Centered at \((x,y + 1)\), if traversing to the right boundary of the reservoir area map, use the left boundary pixel of the next row as the center, and repeat Steps 3-1 to 3-5 until traversing to the point \((m - 1,n - 1)\) of the image to obtain all the contours and contour point sets included in the reservoir area map;

[0039] Step 3-7: In step 3-6, all the contours and contour point sets contained in the storage area are obtained, and further filtering is performed to eliminate invalid storage location contours that do not meet the conditions; a set of contours consisting of n points can be represented as {(x0, y0), (x1, y1), ..., (x n-1 ,y n-1 )}, the contour area S is calculated as follows:

[0040]

[0041] Where × is the cross product. The area of ​​each obtained contour is calculated according to the above formula. A threshold is set based on the storage location area in the application scenario. Contours and contour point sets with areas smaller than the threshold are discarded to obtain contours and contour point sets containing valid storage locations.

[0042] Step 3-8: Create a minimum enclosing rectangle for the location's outline obtained in step 3-7. Based on the principle that the minimum enclosing rectangle of a polygon must be collinear with one of its sides, use the rotating caliper algorithm to enumerate the edges of the location's outline and create enclosing rectangles. Compare the areas of all the resulting enclosing rectangles and select the smallest one as the location's minimum enclosing rectangle. The location's outline must be contained within its minimum enclosing rectangle. Simply map the minimum enclosing rectangle to the warehouse area map to create a rectangular location map that includes the individual locations.

[0043] In step 3-9, the area between the location outline and the rectangular boundary in ω is an invalid region. Traverse the invalid region and change the pixel values ​​within the invalid region to 0 to remove the interference of the invalid region on ω. Perform steps 3-7 to 3-9 for all the locations in the warehouse area map to complete the location segmentation. Using the location ID as the key, the segmented location and the location outline point set and minimum bounding rectangle in step 3-6 as the value, generate a key-value pair and save it locally. Set the calibration completion flag to complete the location calibration.

[0044] Step 3-10: Repeat steps 3-1 to 3-9 for the new uncalibrated storage area to complete the calibration of the storage locations included in all storage areas.

[0045] As a further improvement, step S3 further includes the following steps:

[0046] Step 4-1: Use a mobile phone to capture 500 images of empty storage locations of varying styles and colors at various application scenarios, such as project sites and Cainiao Express Stations, as raw samples for the "vacant" category. Additionally, capture videos of the storage locations for subsequent dataset expansion, and capture 1,000 images of occupied storage locations as the "occupied" category. An additional 500 images of express parcels of various shapes are also captured as object samples.

[0047] Step 4-2: Extract the region of interest from the bin video in Step 4-1. First, load the video stream. Select two points A(r1, c1) and B(r2, c2) in the video by mouse, and generate a rectangular region of interest with A and B as the diagonals.

[0048] Step 4-3: Read the video stream frame by frame, convert the video frame to a Mat-format grayscale image, and denote the sum of the pixel values in the j-th column of the region of interest in the i-th frame as S ij If there are N (c1 < N < c2) columns where the absolute value of the difference between the pixel sum of the current column and the pixel sum of the same column in the previous frame is greater than the threshold T ij , as shown in the following formula:

[0049]

[0050]

[0051] where M is the total number of frames in the video stream, and the threshold T ij = 0.8 * S ij If n is greater than N, it proves that the image within the region of interest in the current frame has changed significantly compared to the previous frame, and then save the region of interest as a sample.

[0052] Step 4-4: Augment the "idle" dataset. Perform the following operations on the 500 copies of empty bins obtained in Step 4-1:

[0053] (1) Randomly change the color of the bin;

[0054] (2) Randomly generate an angle between 0 - 180°, and rotate the copy by this angle.

[0055] (3) Randomly generate a scale factor between 0.1 - 10, and scale the copy.

[0056] (4) Randomly generate salt-and-pepper noise and add it to the copy.

[0057] (5) Binarize the image, binarize the copy;

[0058] Add the copies obtained from the above 5 steps to the "idle" bin dataset, and expand it to 1000 to obtain an enhanced "idle" bin dataset.

[0059] Step 4-5: Perform several operations to enhance the "occupied" bin dataset in Step 4-1, the enhanced "idle" bin dataset obtained in Step 4-4, and the 500 object samples obtained in Step 4-1:

[0060] (1) Sequentially extract n samples from the 500 object samples, where n is a randomly generated number between 1 - 30, and then sequentially extract 10 samples from the enhanced "idle" bin dataset.

[0061] (2) Perform operations on n object samples according to the five methods in step 4-4.

[0062] (3) Randomly overlay the results in (2) onto the 10 enhanced “vacant” storage location samples in step (1) to obtain “occupied” storage location samples.

[0063] After the above steps, 3158 occupied storage locations with different object colors, different storage location floor colors, and objects of different shapes are obtained as the enhanced "occupied" storage location dataset.

[0064] Step 4-6: Merge the enhanced “idle” dataset and the enhanced “occupied” dataset obtained in steps 4-4 and 4-5 to obtain a dataset containing 4158 storage location images.

[0065] In steps 4-7, the resulting dataset was annotated according to its true classification using the labImg tool. The annotation information was stored in XML format and the annotated dataset was divided into a training set to test set ratio of 8:2. A virtual environment for model training was set up on the GPU server. After training, the training set was fed into the MobileNetV3-Large network for model training, resulting in a network model suitable for inventory location detection.

[0066] The present invention also discloses a storage location detection system based on RGB camera, which at least includes a mobile robot, an RGB camera and a control unit, wherein:

[0067] The RGB camera is connected to the control unit and is used to capture images of the warehouse area and output them to the control unit. The warehouse area is an area consisting of multiple storage locations, which are used by robots to pick up and place goods. The storage location outlines are marked with one of the three primary colors of red, blue, or green, which is different from the ground color.

[0068] The mobile robot is connected to the control unit and is used to plan the target storage location according to the storage location detection results fed back by the control unit, thereby avoiding "occupying" the storage location and moving autonomously to realize the transportation and unloading functions of the goods;

[0069] The control unit is used to perform storage location detection operations according to the control instructions of the mobile robot and output the results to provide the mobile robot with current storage location status information.

[0070] As a further improvement, the control unit includes at least a video stream preprocessing module, a storage location calibration module, and a storage location detection module, wherein:

[0071] The video stream and processing module is used to perform histogram equalization to increase the contrast between the storage area edge and the ground, median filtering to remove salt and pepper noise above the storage area, and mean filtering and Gaussian filtering to smooth the storage area image to reduce Gaussian noise interference;

[0072] The storage location calibration module is used to separate the R, B, and G channels according to the storage area map, use a certain channel as data input, perform threshold segmentation, highlight edge pixels, extract edge points, and remove erroneous and outlier edge points through filtering. Then, the minimum enclosing rectangle of the storage location is calculated, and all storage areas in the storage area are segmented according to the minimum enclosing rectangle. The storage location ID is added and associated with the corresponding storage location outline and background map, and the storage location outline is reconstructed to achieve storage location calibration.

[0073] The storage location detection module includes the creation and enhancement of a data set, which is fed into the MobileNetV3-Large network for training to obtain a reasonable network model, classify the storage location map to be detected, and obtain the final state of the storage location.

[0074] As a further improvement, the RGB camera is a monocular RGB camera with wireless transmission function. The camera video can be transmitted wirelessly to other receiving ends. The camera is installed just above the center of the storage area, looking down vertically at the storage area.

[0075] Compared with the existing technology, the present invention uses a monocular RGB camera to realize one-click autonomous calibration of a large number of storage locations, including storage location identification, contour point calculation and segmentation. The calibration results are accurate, fast and do not require human intervention. A lightweight neural network model is used to detect storage location status, which has the advantages of high accuracy, low parameter count, low computational complexity and simple deployment. It can quickly provide storage location status for robots during cargo handling, meet the real-time requirements of robots, greatly improve the rationality and safety of robot cargo handling storage location planning, and improve the stability and reliability of the robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 This is a flowchart of the storage location detection method based on RGB camera of the present invention.

[0077] Figure 2 This is a flow chart of a storage location calibration and detection method based on an RGB camera in a preferred embodiment of the present invention.

[0078] Figure 3 This is the flow chart of the storage location calibration in the present invention.

[0079] Figure 4 This is a flow chart of storage location detection in the present invention.

[0080] Figure 5 This is a functional block diagram of the storage location detection system based on RGB camera of the present invention.

[0081] The following specific embodiments will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0082] The technical solution provided by the present invention will be further described below with reference to the accompanying drawings.

[0083] See also Figure 1 , shown is a flowchart of the storage location detection based on RGB camera of the present invention, which at least includes the following steps:

[0084] Step S1: According to the control instruction, an image of the current storage area captured by the RGB camera is obtained; wherein the storage area is an area consisting of multiple storage locations, and the storage location outlines are marked with one of the three primary colors of red, blue, and green that are different from the ground color;

[0085] Step S2: performing image processing on the current warehouse area image according to the pre-stored warehouse area calibration information to map the corresponding warehouse area calibration information to obtain a warehouse location map to be inspected;

[0086] Step S3: Input the warehouse location map to be detected into the pre-trained network model for processing, obtain the warehouse location status and output the detection result.

[0087] In the above technical solution, since one of the three primary colors of red, blue and green, which is different from the ground color, is used for marking, it is possible to easily identify and segment storage locations, which greatly simplifies the difficulty of image processing.

[0088] Furthermore, in step S2, the reservoir area calibration information is obtained by the following steps:

[0089] Step S21: remove objects placed on the ground where the storage area is located, and use one of the three primary colors of red, blue, and green that are different from the ground color to mark the outline of the storage area so that the outline of the storage area has a certain contrast with the ground; on this basis, start the RGB camera to collect images of the storage area;

[0090] Step S22: pre-processing the storage area image;

[0091] Step S23: Separate the RGB three channels of the warehouse image, use one of the channels as the subsequent data input, perform threshold segmentation on it, highlight the edge pixels, extract the edge points, filter them, remove the mis-extracted and outlier edge points, and then calculate the minimum enclosing rectangle of the warehouse location. Segment all the warehouse areas in the warehouse according to the minimum enclosing rectangle, add the warehouse location ID and associate the corresponding warehouse location outline and background image, reconstruct the warehouse location outline, and complete the warehouse location calibration.

[0092] In step S3, the pre-trained network model is obtained by the following steps:

[0093] Step S31: constructing a garage “vacant” enhanced dataset and an “occupied” enhanced dataset and adding corresponding labels;

[0094] Step S32: Build a MobileNetV3-Large network model and train the model using the dataset built in step S31;

[0095] Step S33: After training is completed, the network model is output.

[0096] See also Figure 2 , which shows a preferred embodiment of the present invention, wherein the storage location calibration and detection method based on RGB camera specifically includes the following steps:

[0097] (1) Use one of the three primary colors of red, blue, and green to stick to the edge of the storage location, start the camera, collect images of the storage area, analyze the camera video stream, create a new buffer folder, initialize the storage location calibration and detection state variables, issue instructions according to the visual interface, and enter the calibration or detection process ready state; the specific process is as follows:

[0098] Step 1-1: Remove objects placed on the ground in the storage area, and use red floor tape to stick the outline of each storage location around the edge of the gray ground storage area to create a certain contrast between the storage location and the outline.

[0099] Step 1-2: Start the top-mounted camera, configure the IP address, and sense the warehouse area vertically downward. Use the camera's built-in wireless transmission function to transmit the warehouse area video stream to the control unit via the RTSP address. The address is as follows:

[0100] rtsp: / / 192.168.1.33 / 2

[0101] Using the uniqueness of the fourth segment of the IP address, we assign a unique ID to the storage area, ID 33 (see above). We create a new storage area buffer folder, ID 33, to store the calibration result image. We create a video stream parsing thread, looping to capture and decode the camera data from the RTSP address, obtaining an 8-bit unsigned 3-channel RGB storage area image and storing it in Mat format, denoted as Ω.

[0102] (2) Preprocessing of the reservoir area image includes using histogram equalization to increase the contrast between the storage area edge and the ground, median filtering to remove the salt and pepper noise above the reservoir area, and mean filtering and Gaussian filtering to smooth the reservoir area image and reduce Gaussian noise interference. The specific process is as follows:

[0103] Step 2-1. Perform histogram equalization on the Mat-type reservoir map Ω obtained in Step 1-2. Filter the equalized result. First, use a median filter to traverse the pixels in the image and sort the grayscale values ​​of all pixels in the four neighborhoods of the center pixel. Then, select the median grayscale value of this sorted value to replace the grayscale value of the center element. Next, perform a 3*3 mean filter. Similarly, traverse the pixels in the image and average the values ​​of all pixels in the neighborhood of the center pixel, replacing the center pixel. Finally, perform a Gaussian filter with a Gaussian kernel size of 3*3. The formula is as follows:

[0104]

[0105] Among them, f(x,y) represents the pixel value at the coordinate (x,y), σ is the standard deviation, and the reservoir area Ω is obtained filter .

[0106] Step 2-2: Respond to commands issued by the visual interface. For calibration commands, reset the Start Detection, Calibration Complete, and Detection Complete flags, set the Start Calibration flag, and save the library area ID and cache folder path from Steps 1-2. For detection commands, reset the Start Calibration, Calibration Complete, and Detection Complete flags, set the Start Detection flag, and check whether calibration has been completed for the library area. If not, terminate the response and wait for the calibration command. Otherwise, continue the process.

[0107] Step 2-3: In step (1), the warehouse location calibration command is obtained, the calibration thread enters the running state, and reads steps 1-3 to obtain the filtered warehouse area image.

[0108] Step 2-4: Transform the storage area image Ω filter Perform channel segmentation to obtain three single-channel grayscale images of R, G, and B. The pixel values ​​are between 0 and 255, with 0 being black and 255 being white. Use the R channel as data input and perform threshold segmentation operation with a threshold of 200. Traverse the R channel image and perform pixel transformation on the R channel library image according to the following formula:

[0109]

[0110] The pixel values ​​of the storage location edges with pixel values ​​greater than 200 are set to 255, and then the pixel values ​​are divided by 255 to obtain the binary storage area map Ω thres .

[0111] (3) Separate the RGB three channels of the reservoir area image, use the R channel as the subsequent data input, perform threshold segmentation on it to highlight the edge pixels, then extract the edge points, and after filtering to remove mis-extracted and outlier edge points, calculate the minimum bounding rectangle of the storage location, segment all storage locations in the reservoir area according to the minimum bounding rectangle, add the storage location ID and associate the corresponding storage location contour and background image, and reconstruct the storage location contour to complete the storage location calibration; as Figure 3 shown, the specific process is as follows:

[0112] Step 3-1, Analyze the thres topological structure, find the outer boundary and hole boundary in the reservoir area map. The outer boundary is composed of outer boundary points, and the hole boundary is composed of hole boundary points. The definitions of outer boundary points and hole boundary points are as follows:

[0113]

[0114] where 0 ≤ x < m, 0 ≤ y < n, and m and n are the number of rows and columns of thres Ω. Traverse Ω from left to right and top to bottom thres , set the flag NBD to represent the new boundary and LNBD to represent the parent boundary. Initialize NBD to 1 every time a new row is traversed.

[0115] Step 3-2, When traversing to the first outer boundary point (x, y), increment the flag NBD of (x, y) by 1. Denote the pixel at (x, y - 1) as (x2, y2). Taking (x, y) as the center, rotate clockwise from (x2, y2) to find the first non-zero pixel in the eight-neighborhood of (x, y), denoted as (x1, y1). If there is no non-zero pixel, set the flag to -NBD and jump to Step 3-4.

[0116] Step 3-3, Re-denote the pixel point (x1, y1) as (x2, y2), and (x, y) as (x3, y3). Starting from (x2, y2), rotate counterclockwise to find the first non-zero pixel point in the eight-neighborhood of (x3, y3), denoted as (x4, y4)

[0117] Step 3-4, If f(x3, y3 + 1) is equal to zero, set the value of f(x3, y3) to -NBD. If f(x3, y3 + 1) is not zero, set the value of f(x3, y3) to NBD. Otherwise, do not change the pixel value at (x3, y3).

[0118] Step 3-5, Check whether the counterclockwise rotation in Step 3-3 returns to the starting point, that is, (x4, y4) = (x, y). If it returns to the starting point, it proves that the retrieval of the pixel (x, y) is completed, and jump to Step 3-6. Otherwise, jump to Step 3-3 to continue the counterclockwise rotation to find non-zero elements.

[0119] Step 3-6: If the flag bit at (x, y) is not equal to 1, then the LNBD at that location is set to |NBD|, and the center pixel is moved to (x, y+1) as the center. If the right boundary of the reservoir area map is traversed, the left boundary pixel point of the next row is taken as the center, and steps 3-1 to 3-5 are repeated until the traversal reaches the image point (m-1, n-1), and all contours and contour point sets contained in the reservoir area map are obtained.

[0120] Step 3-7: In step 3-6, all the contours and contour point sets contained in the storage area are obtained. It is inevitable that there are contours that are extracted by mistake. Therefore, further filtering is required to eliminate invalid storage location contours that do not meet the conditions. For a set of contours consisting of n points, it can be expressed as {(x0,y0),(x1,y1),...,(x n-1 ,y n-1 )}, the contour area S is calculated as follows:

[0121]

[0122] Where × is the cross product. The area of ​​each obtained contour is calculated using the above formula. A threshold is set based on the storage location area in the application scenario of this invention. Contours and contour point sets with areas smaller than the threshold are discarded to obtain contours and contour point sets containing valid storage locations.

[0123] Step 3-8: In practice, some storage locations may not be strictly rectangular. To address this problem, a minimum enclosing rectangle (MBR) is constructed for the location outline obtained in Step 3-7. Based on the principle that the minimum enclosing rectangle of a polygon is necessarily collinear with one of its sides, a rotating caliper algorithm is used to enumerate the edges of the location's outline and construct the circumscribed rectangles. The areas of all the resulting circumscribed rectangles are compared, and the smallest one is selected as the location's MBR. The location's outline is always contained within its MBR. By simply mapping the MBR onto the storage area map, a rectangular location map ω can be created that encompasses the individual locations.

[0124] In step 3-9, the area between the location outline and the rectangular boundary in ω is an invalid region. Traverse the invalid region and change the pixel values ​​within the invalid region to 0 to remove the interference of the invalid region on ω. Perform steps 3-7 to 3-9 for all the locations in the warehouse area map to complete the location segmentation. Using the location ID in step 1-4 as the key, the segmented location and the location outline point set and minimum bounding rectangle in step 3-6 as the value, generate a key-value pair and save it locally. Set the calibration completion flag to complete the location calibration.

[0125] Step 3-10: Repeat steps 3-1 to 3-9 for the new uncalibrated storage area to complete the calibration of the storage locations included in all storage areas.

[0126] (4) Shoot videos and images of empty storage spaces and goods with a mobile phone, then cut the videos to obtain a preliminary dataset, and expand and enhance it, including randomly adjusting the color, angle, and size of the empty storage spaces to generate an "idle" enhanced dataset, and randomly combining the "idle" enhanced dataset with the goods images to generate an "occupied" enhanced dataset. Add labels to the enhanced "idle" and "occupied" datasets, and use the MobileNetV3-Large network model to train the datasets to generate model weight files.

[0127] Step 4-1: In different application scenarios such as project sites and Cainiao stations, use a mobile phone to shoot 500 images of empty storage spaces with different styles and colors as the original samples of the "idle" category. In addition, shoot videos of storage spaces for subsequent expansion of the dataset, and at the same time shoot 1000 images of storage spaces occupied by objects as the "occupied" category. Additionally, shoot 500 express parcels of various shapes as object samples.

[0128] Step 4-2:截取感兴趣区 from the video of the storage space in Step 4-1. First, load the video stream, and click two points A(r1, c1) and B(r2, c2) on the video with the mouse, and generate a rectangular region of interest with A and B as the diagonals.

[0129] Step 4-3: Read the video stream frame by frame, convert the video frame to a Mat format grayscale image, and use S to represent the sum of the pixel values of the j-th column of the region of interest in the i-th frame. If there are N (c1 < N < c2) columns where the absolute value of the difference between the pixel sum of the current column and the pixel sum of the same column in the previous frame is greater than the threshold T, as shown in the following formula: ij where M is the total number of frames in the video stream, and the threshold T ij = 0.8 * S

[0130]

[0131]

[0132] If n is greater than N, it proves that the image within the region of interest in the current frame has changed significantly compared to the previous frame, then save the region of interest as a sample. ij = 0.8 * S ij If n is greater than N, it proves that the image within the region of interest in the current frame has changed significantly compared to the previous frame, then save the region of interest as a sample.

[0133] Step 4-4: Expand the "idle" dataset and perform the following operations on the 500 copies of the empty storage spaces obtained in Step 4-1:

[0134] (1) Randomly transform the color of the storage space;

[0135] (2) Randomly generate an angle between 0 - 180°, and rotate the copy according to this angle.

[0136] (3) Randomly generate a scale factor between 0.1 and 10 to scale the replicas.

[0137] (4) Randomly generate salt and pepper noise and add it to the replica.

[0138] (5) Binarize the image and binarize the copy;

[0139] Add the copies obtained in the above 5 steps to the "free" storage location dataset, expand it to 1000, and obtain the enhanced "free" storage location dataset.

[0140] Step 4-5: Perform several operations to enhance the "occupied" location dataset from step 4-1, the enhanced "vacant" location dataset from step 4-4, and the 500 object samples from step 4-1:

[0141] (1) Select n objects from the 500 object samples, where n is a randomly generated number between 1 and 30, and then select 10 objects from the enhanced “free” location dataset.

[0142] (2) Perform operations on n object samples according to the five methods in step 4-4.

[0143] (3) Randomly overlay the results in (2) onto the 10 enhanced “vacant” storage location samples in step (1) to obtain “occupied” storage location samples.

[0144] After the above steps, 3158 occupied storage locations with different object colors, different storage location floor colors, and objects of different shapes are obtained as the enhanced "occupied" storage location dataset.

[0145] Step 4-6: Merge the enhanced “idle” dataset and the enhanced “occupied” dataset obtained in steps 4-4 and 4-5 to obtain a dataset containing 4158 storage location images.

[0146] In steps 4-7, the resulting dataset was annotated according to its true classification using the labImg tool. The annotation information was stored in XML format and the annotated dataset was divided into a training set to test set ratio of 8:2. A virtual environment for model training was set up on the GPU server. After training, the training set was fed into the MobileNetV3-Large network for model training, resulting in a network model suitable for inventory location detection.

[0147] (5) Start the detection process, segment the current warehouse area image, obtain the warehouse location map contained in the warehouse area, load the trained model weight file, deploy the network model on the controller GPU, reason about the warehouse location map, realize the warehouse location status classification, and obtain the warehouse location detection result "free" or "occupied"; Figure 4 The specific process is as follows:

[0148] Step 5-1: Receive the detection command for storage area k, and search for k in the key-value pair in step 3-9. If k is not found, it proves that the storage area calibration process has not been completed for this storage location, and the storage location detection process is ended and the flag is reset.

[0149] Step 5-2: Read the image Ω of the warehouse area k currently captured by the camera according to the method described in step 1-2, and use k as the key to search in the key-value pair of the calibration result obtained in step (4), and obtain all the warehouse location IDs, warehouse location minimum bounding rectangles, and warehouse location outlines contained in Ω, and map them onto Ω. According to the mapped minimum bounding rectangle, cut out all the warehouse locations contained in the current warehouse area k to be detected, and set the pixel value of the invalid area in the minimum bounding rectangle to 0 according to the method described in step 3-9, remove the interference of this area on the detection process, and obtain the warehouse location map to be detected.

[0150] Step 5-3: Input the warehouse location map to be detected obtained in step 5-2 into the network model trained in step 4-7 for inference, so as to obtain the status of the warehouse location and realize the detection of the warehouse location.

[0151] The present invention also provides a storage location calibration and detection system based on an RGB camera, which includes a mobile robot, an RGB camera, a control unit, and a storage area with storage locations; the storage area is an area composed of multiple storage locations, and the storage locations are used for robots to pick up and place goods. The boundaries of this area are pasted with red floor tape.

[0152] The mobile robot is an industrial AGV, which can plan the target storage location according to the storage location detection results provided by the system, move autonomously, avoid "occupying" the storage location, and realize the transportation and unloading functions of goods.

[0153] The RGB camera is a monocular RGB camera with a wireless transmission function. The camera video can be transmitted wirelessly to other receiving ends. The camera is installed just above the center of the storage area and looks down vertically at the storage area.

[0154] The control unit includes a video stream preprocessing module, a storage location calibration module, and a storage location detection module;

[0155] The video stream and processing module includes histogram equalization to increase the contrast between the edge of the storage area and the ground, median filtering to remove salt and pepper noise above the storage area, mean filtering and Gaussian filtering to smooth the storage area image and reduce Gaussian noise interference.

[0156] The storage location calibration module separates the R, B, and G channels according to the storage area map, uses the R channel as data input, performs threshold segmentation, highlights edge pixels, extracts edge points, and removes mis-extracted and outlier edge points through filtering. It then calculates the minimum enclosing rectangle of the storage location, segments all storage areas in the storage area according to the minimum enclosing rectangle, adds the storage location ID, associates the corresponding storage location outline and background map, reconstructs the storage location outline, and realizes storage location calibration.

[0157] The storage location detection module includes the creation and enhancement of a data set, which is fed into the MobileNetV3-Large network for training to obtain a reasonable network model, classify the storage location map to be detected, and obtain the final state of the storage location.

[0158] See also Figure 5 , shown is a principle block diagram of the storage location detection system based on RGB camera of the present invention, the system includes at least a mobile robot, an RGB camera and a control unit, wherein,

[0159] The RGB camera is connected to the control unit and is used to capture images of the warehouse area and output them to the control unit. The warehouse area is an area consisting of multiple storage locations, which are used by robots to pick up and place goods. The storage location outlines are marked with one of the three primary colors of red, blue, or green, which is different from the ground color.

[0160] The mobile robot is connected to the control unit and is used to plan the target storage location according to the storage location detection results fed back by the control unit, thereby avoiding "occupying" the storage location and moving autonomously to realize the transportation and unloading functions of the goods;

[0161] The control unit is used to perform storage location detection operations according to the control instructions of the mobile robot and output the results to provide the mobile robot with current storage location status information.

[0162] In the above technical solution, the RGB camera is a monocular RGB camera with wireless transmission function. The camera video can be transmitted wirelessly to other receiving ends. The camera is installed just above the center of the storage area and looks down vertically at the storage area.

[0163] The control unit includes at least a video stream preprocessing module, a storage location calibration module, and a storage location detection module, wherein:

[0164] The video stream and processing module is used to perform histogram equalization to increase the contrast between the storage area edge and the ground, median filtering to remove salt and pepper noise above the storage area, and mean filtering and Gaussian filtering to smooth the storage area image to reduce Gaussian noise interference;

[0165] The storage location calibration module is used to separate the R, B, and G channels according to the storage area map, use a certain channel as data input, perform threshold segmentation, highlight edge pixels, extract edge points, and remove erroneous and outlier edge points through filtering. Then, the minimum enclosing rectangle of the storage location is calculated, and all storage areas in the storage area are segmented according to the minimum enclosing rectangle. The storage location ID is added and associated with the corresponding storage location outline and background map, and the storage location outline is reconstructed to achieve storage location calibration.

[0166] The storage location detection module includes the creation and enhancement of a data set, which is fed into the MobileNetV3-Large network for training to obtain a reasonable network model, classify the storage location map to be detected, and obtain the final state of the storage location.

[0167] The video stream preprocessing module, the storage location calibration module and the storage location detection module execute the relevant algorithms described in the above-mentioned storage location detection method based on RGB camera, which will not be repeated here.

[0168] The above embodiments are only intended to help understand the method and core concept of the present invention. It should be noted that, without departing from the principles of the present invention, a number of improvements and modifications may be made to the present invention by those skilled in the art, and such improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0169] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A storage location detection method based on RGB camera, characterized in that: At least the following steps are included: Step S1: According to the control instruction, an image of the current storage area captured by the RGB camera is obtained; wherein the storage area is an area consisting of multiple storage locations, and the storage location outlines are marked with one of the three primary colors of red, blue, and green that are different from the ground color; Step S2: performing image processing on the current warehouse area image according to the pre-stored warehouse area calibration information to map the corresponding warehouse area calibration information to obtain a warehouse location map to be inspected; Step S3: Input the warehouse location map to be tested into the pre-trained network model for processing, obtain the warehouse location status and output the test results; In step S2, the reservoir area calibration information is obtained through the following steps: Step S21: remove objects above the ground where the storage area is located, and use one of the three primary colors of red, blue, and green that is different from the ground color to mark the outline of the storage area so that the outline of the storage area has a contrast with the ground; then start the RGB camera to capture an image of the storage area; Step S22: pre-processing the storage area image; Step S23: Separate the RGB channels of the warehouse image, use one of the channels as the subsequent data input, perform threshold segmentation on it, highlight edge pixels, extract edge points, filter them to remove erroneous and outlier edge points, and then calculate the minimum enclosing rectangle of the warehouse location. Segment all warehouse areas in the warehouse according to the minimum enclosing rectangle, add the warehouse location ID and associate the corresponding warehouse location outline with the background image, reconstruct the warehouse location outline, and complete the warehouse location calibration; In step S3, the pre-trained network model is obtained by the following steps: Step S31: Constructing a garage "vacant" enhanced dataset and an "occupied" enhanced dataset and adding corresponding labels; Step S32: Build a MobileNetV3-Large network model and train the model using the dataset built in step S31; Step S33: After training is completed, the network model is output.

2. The storage location detection method based on RGB camera according to claim 1, characterized in that: In step S2, if the calibration information of the current warehouse area is not found, the warehouse location calibration process is first executed to generate the warehouse area calibration information.

3. The storage location detection method based on RGB camera according to claim 1, characterized in that: In step S3, the state of each storage location is classified by the pre-trained network model to obtain the storage location detection result "free" or "occupied".

4. The storage location detection method based on RGB camera according to claim 1, characterized in that: Red floor tape is used to stick the outline of each storage location around the edge of each storage location in the gray ground storage area. In step S23, the red channel is extracted as subsequent data input.

5. The storage location detection method based on RGB camera according to claim 1, characterized in that: In step S31, a mobile phone is used to capture videos and images of empty storage locations and cargo. The videos are then cropped to generate a preliminary dataset, which is then expanded and enhanced. This includes randomly adjusting the color, angle, and size of the empty storage locations. The "idle" enhanced dataset is randomly combined with cargo images to generate an "occupied" enhanced dataset. Labels are then added to the enhanced "idle" and "occupied" datasets.

6. The storage location detection method based on RGB camera according to claim 1, characterized in that: Step S33: After training the MobileNetV3-Large network model using the data set, a model weight file is generated.

7. The storage location detection system based on RGB camera is characterized by: The system comprises at least a mobile robot, an RGB camera and a control unit, wherein: The RGB camera is connected to the control unit and is used to capture images of the warehouse area and output them to the control unit. The warehouse area is an area consisting of multiple storage locations, which are used by robots to pick up and place goods. The storage location outlines are marked with one of the three primary colors of red, blue, or green, which is different from the ground color. The mobile robot is connected to the control unit and is used to plan the target storage location according to the storage location detection results fed back by the control unit, thereby avoiding "occupying" the storage location and autonomously moving to realize the transportation and unloading functions of the goods; The control unit is used to perform storage location detection operations according to the control instructions of the mobile robot and output the results to provide the mobile robot with current storage location status information; The control unit at least includes a video stream preprocessing module, a storage location calibration module, and a storage location detection module, wherein: The video stream and processing module is used to perform histogram equalization to increase the contrast between the storage area edge and the ground, median filtering to remove salt and pepper noise above the storage area, and mean filtering and Gaussian filtering to smooth the storage area image to reduce Gaussian noise interference; The storage location calibration module is used to separate the R, G, and B channels according to the storage area map, use a certain channel as data input, perform threshold segmentation, highlight edge pixels, extract edge points, and remove erroneous and outlier edge points through filtering. Then, the minimum enclosing rectangle of the storage location is calculated, and all storage areas in the storage area are segmented according to the minimum enclosing rectangle. The storage location ID is added and associated with the corresponding storage location outline and background map, and the storage location outline is reconstructed to achieve storage location calibration. The storage location detection module includes the creation and enhancement of a data set, which is fed into the MobileNetV3-Large network for training to obtain a reasonable network model, classify the storage location map to be detected, and obtain the final state of the storage location.

Images