Three-dimensional storage real-time monitoring method and device

Abstract

The invention discloses a three-dimensional storage real-time monitoring method and device, and relates to the technical field of storage logistics monitoring, and the method comprises the following steps: obtaining an RGB image and a depth image containing a storage target; performing space alignment on the RGB image and the depth image; inputting the aligned RGB image and depth image into a pre-trained YOLOv5 network, and determining a target category in the image and a two-dimensional bounding box of each target; combining the detected coordinates of the two-dimensional bounding box of the target with the depth information in the depth image, and calculating the position information of the target in the three-dimensional space; searching a matched setting frame area from a database according to the position information of the target in the three-dimensional space; and according to the intersection-to-union ratio of the target two-dimensional boundary frame and the set frame area, whether the placement of the target is compliant is judged, and an alarm signal is sent to the storage target which is not compliant in placement.

Description

Technical Field

[0001] The present invention relates to the technical field of warehousing logistics monitoring, and in particular to a three-dimensional warehousing real-time monitoring method and device. Background Art

[0002] Existing warehouse monitoring systems generally lack multi-target detection and classification capabilities, particularly in the classification and location of cargo, pallets, and transport equipment. Furthermore, these traditional systems lack effective real-time monitoring of cargo status, leading to safety risks and inefficiencies in warehouse operations.

[0003] Traditional warehouse management usually relies on manual operations. Although it can basically meet the storage needs of goods, it has a significant gap in efficiency, safety and cost control compared with modern automated warehousing systems. This model cannot adapt to the real-time monitoring and intelligent management needs of unmanned warehousing environments.

[0004] In view of this, the present invention proposes a three-dimensional warehouse real-time monitoring method and device. Summary of the Invention

[0005] The purpose of the present invention is to provide a three-dimensional warehouse real-time monitoring method and device, which can monitor the goods, shelves, pallets, transportation equipment and working areas in the warehouse environment in real time.

[0006] In a first aspect, the present invention provides a three-dimensional warehouse real-time monitoring method, comprising the following steps:

[0007] Step S1: Acquire an RGB image and a depth image containing a storage target;

[0008] Step S2: spatially aligning the RGB image with the depth image;

[0009] Step S3: Input the aligned RGB image and depth image into the pre-trained YOLOv5 network to determine the object category and the two-dimensional bounding box of each object in the image;

[0010] Step S4: combining the coordinates of the detected two-dimensional bounding box of the target with the depth information in the depth image to calculate the position information of the target in three-dimensional space;

[0011] Step S5: searching a matching setting frame area from a database according to the position information of the target in the three-dimensional space;

[0012] Step S6: Determine whether the placement of the target is compliant based on the intersection-over-union ratio of the target two-dimensional bounding box and the set box area, and issue an alarm signal for storage targets that are not placed in compliance.

[0013] As a preferred technical solution of the first aspect of the present invention, the logic for obtaining the object category and the two-dimensional bounding box of each object in the image is:

[0014] Determine the corresponding point of each pixel of the depth image in the RGB image;

[0015] For each pixel in the RGB image, the corresponding depth information is interpolated through the four nearest corresponding points around it;

[0016] Extracting color information of each pixel in the aligned RGB image, and generating a first intermediate value based on the trained first parameter combination;

[0017] Superimposing the first intermediate value with the depth information of each pixel in the aligned depth image to form a second intermediate value;

[0018] For the second intermediate value, information about the target category and the target bounding box in each preset grid unit is obtained by calculation based on the trained second parameter combination.

[0019] As a preferred technical solution of the first aspect of the present invention, step S4 includes:

[0020] Determine the coordinates of the center point of the target's two-dimensional bounding box;

[0021] Find the depth information at the center point coordinates in the aligned depth image;

[0022] The three-dimensional space coordinates are calculated according to the center point coordinates and the depth information as the position information of the target.

[0023] As a preferred technical solution of the first aspect of the present invention, step S6 includes:

[0024] Determine the intersection area and union area of ​​the two-dimensional bounding box of the target and the set box;

[0025] Calculating an intersection-over-union ratio based on the intersection area and the union area;

[0026] Whether the placement of the target is compliant is determined based on the intersection-over-union ratio and a preset threshold value. When the intersection-over-union ratio is lower than the threshold value, the placement of the target is not compliant; otherwise, the placement of the target is compliant.

[0027] In a second aspect, the present invention provides a three-dimensional warehouse real-time monitoring device, based on the implementation of the first aspect, comprising:

[0028] Image acquisition module, used to obtain RGB images and depth images containing storage targets;

[0029] An image alignment module, configured to spatially align the RGB image with the depth image;

[0030] The object recognition module is used to input the aligned RGB image and depth image into the pre-trained YOLOv5 network to determine the object category and the two-dimensional bounding box of each object in the image;

[0031] The target position information calculation module is used to combine the coordinates of the two-dimensional bounding box of the detected target with the depth information in the depth image to calculate the position information of the target in three-dimensional space;

[0032] A setting frame acquisition module, configured to search for a corresponding setting frame according to the position information of the target in the three-dimensional space;

[0033] The target pose estimation module is used to determine whether the placement of the target is compliant based on the setting box information and the two-dimensional bounding box of the target, and to issue an alarm signal when the placement of the target is not compliant.

[0034] As a preferred technical solution of the second aspect of the present invention, the target category and the two-dimensional bounding box of each target in the image are obtained based on the image alignment module and the target recognition module;

[0035] The image alignment module includes a projection calculation unit and an interpolation calculation unit:

[0036] A projection calculation unit, used to determine the corresponding point of each pixel of the depth image in the RGB image;

[0037] The interpolation calculation unit is used to interpolate the corresponding depth information for each pixel in the RGB image through the four nearest corresponding points around it.

[0038] The target recognition module includes a first intermediate value calculation unit and a second intermediate value calculation unit:

[0039] a first intermediate value calculation unit, configured to extract color information of each pixel in the aligned RGB image and generate a first intermediate value based on a trained first parameter combination;

[0040] a second intermediate value calculation unit, configured to superimpose the first intermediate value with the depth information of each pixel in the aligned depth image to form a second intermediate value;

[0041] The target recognition unit is used to calculate the second intermediate value based on the trained second parameter combination to obtain the target category and target bounding box information in each preset grid unit.

[0042] As a preferred technical solution of the second aspect of the present invention, the target position information calculation module includes:

[0043] A center point coordinate determination unit, used to determine the coordinates of the center point of the two-dimensional bounding box of the target;

[0044] a depth information determining unit, configured to find the depth information at the coordinates of the center point in the aligned depth image;

[0045] The position information determining unit is used to calculate the three-dimensional space coordinates according to the center point coordinates and the depth information as the position information of the target.

[0046] As a preferred technical solution of the second aspect of the present invention, the target pose estimation module includes:

[0047] An intersection and union region determining unit, configured to determine an intersection region and a union region between a two-dimensional bounding box of the target and the set box;

[0048] an intersection-over-union ratio calculation unit, configured to calculate an intersection-over-union ratio based on the intersection area and the union area;

[0049] The compliance judgment unit is used to judge whether the placement of the target is compliant based on the intersection-over-union ratio and a preset threshold value. When the intersection-over-union ratio is lower than the threshold value, the placement of the target is not compliant; otherwise, the placement of the target is compliant.

[0050] In a third aspect, the present invention provides a computer device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the three-dimensional warehouse real-time monitoring method as described above.

[0051] In a fourth aspect, a readable storage medium stores a computer program, and when the computer program is executed by a processor, it is used to implement the three-dimensional warehouse real-time monitoring method as described above.

[0052] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0053] The present invention can monitor goods, shelves, pallets, transportation equipment and operation areas in a warehousing environment in real time, thereby improving the recognition accuracy of warehousing targets and reducing safety hazards in warehousing operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments described in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0055] Figure 1 This is a flow chart of the three-dimensional warehouse real-time monitoring method of the present invention.

[0056] Figure 2 This is a structural framework diagram of the three-dimensional warehouse real-time monitoring system of the present invention;

[0057] Figure 3 It is a schematic diagram of the structure of the computer equipment of the terminal or server of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the implementation of this application clearer, the technical solutions in the implementation of this application will be described in more detail below in conjunction with the drawings in the implementation of this application. In the drawings, the same or similar numbers throughout represent the same or similar elements or elements with the same or similar functions. The described implementation is a part of the implementation of this application, not all of the implementations. The implementation described below with reference to the drawings is exemplary and is intended to be used to explain this application, and should not be understood as a limitation on this application. Based on the implementation in this application, all other implementations obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. The implementation of this application is described in detail below in conjunction with the drawings.

[0059] Example 1

[0060] See also Figure 1 As shown, this embodiment provides a three-dimensional warehouse real-time monitoring method, including the following steps:

[0061] Step S1: Acquire an RGB image and a depth image containing a storage target.

[0062] This step obtains an RGB image through an RGB camera and a depth image through a depth camera. Common depth cameras include binocular vision cameras, speckle structured light cameras, and ToF cameras.

[0063] Step S2: spatially align the RGB image with the depth image.

[0064] The purpose of spatial alignment is to map each pixel in the depth image D to the coordinate system of the RGB image so that the pixels of the two correspond one to one. In this way, each pixel in the RGB image not only has the original RGB color information, but also has depth information. This embodiment uses the RGB image as a reference to adjust the depth image, specifically including the following steps:

[0065] Step S21: determining the corresponding point of each pixel of the depth image in the RGB image;

[0066] Step S22: For each pixel in the RGB image, interpolate the corresponding depth information through the four nearest corresponding points around it.

[0067] In step S21 of this embodiment, the coordinates corresponding to each pixel point in the depth image D are mapped to the RGB image, that is, a plurality of corresponding points with coordinate and depth information are formed on the RGB image. The coordinates of these corresponding points may be inconsistent with the coordinates of the pixel points of the RGB image obtained in step S1. Therefore, in step S22, the depth information of the pixel points of the RGB image is calculated by interpolation, thereby completing the processing of the RGB image, so that the RGB image has both color information and depth information. Compared with the existing warehouse monitoring that usually uses RGB or depth information separately, this embodiment combines the two through spatial alignment and interpolation technology (steps S21-S22), thereby solving the spatial inconsistency of the two data sources and enhancing the accuracy of three-dimensional positioning.

[0068] For example, the resolution of the depth image D is WD×HD, and the coordinates of each pixel in the depth image D are (X D ,Y D ), the depth value is Z D The resolution of the RGB image is WI×HI, and the coordinates of each pixel are (X I ,Y I ). Assume that the camera intrinsic parameter matrices of the depth image D and the RGB image are both K, and the rotation and translation matrices of the two images are recorded as [R|T].

[0069] In step S21, for each pixel point (X D ,Y D ), the corresponding three-dimensional space point (X D ,Y D ,Z D ) can be calculated using the camera model:

[0070]

[0071] Then the three-dimensional space point (X D ,Y D ,Z D ) is projected into the pixel coordinate system of the RGB image and the coordinates of the corresponding points (X I ,Y I ):

[0072]

[0073] As mentioned above, due to the different resolutions of the depth image D and the RGB image, or due to the difference in viewing angles, the coordinates of some pixel points in the RGB image may be inconsistent with the coordinates (xI, yI) of the corresponding points calculated above, that is, there is no corresponding depth value. Therefore, in step S22, these gaps are filled by interpolation methods. This application uses a bilinear interpolation method. For each pixel point (xI, yI) in the RGB image,I ,Y I ), find the four nearest corresponding points around it (X I ,Y I ), here we use To express it, you can use the depth values ​​of these four points Calculate each pixel (X in the original RGB image by bilinear interpolation I ,Y I )’s depth value D′(X I ,Y I ):

[0074]

[0075] Among them, Interp is the bilinear interpolation function.

[0076] The above generates a mixed image of RGB image and depth image. In an alternative embodiment, they can also be represented separately. That is, based on the above principle, a new depth image D' is interpolated from the original depth image D according to the pixel coordinates of the RGB image. The algorithm generates a depth image D' aligned with the RGB image, with a resolution of WI×HI and a depth value of each pixel D'(X I , Y I ).

[0077] It is further explained that the depth interpolation based on the four nearest neighbor points (step S22) is adopted instead of global interpolation or direct mapping, which reduces the amount of calculation while avoiding the distortion of depth information. It is particularly suitable for regularly arranged shelf structures in a warehousing environment. This optimization is scene-specific.

[0078] Step S3: Input the aligned RGB image and depth image into the pre-trained YOLOv5 network to determine the object category and the two-dimensional bounding box of each object in the image.

[0079] This application uses a pre-trained YOLOv5 network to identify targets in input RGB images and depth images to determine the type of target and its boundaries.

[0080] In some optional embodiments, step S3 further includes:

[0081] Step S31: extracting color information of each pixel in the aligned RGB image, and generating a first intermediate value based on the trained first parameter combination;

[0082] Step S32: superimposing the first intermediate value with the depth information of each pixel in the aligned depth image to form a second intermediate value;

[0083] Step S33: For the second intermediate value, obtain the preset target category and target bounding box information within each grid unit based on the trained second parameter combination calculation.

[0084] In step S31, the application first reads the aligned RGB image. The backbone network (Backbone) of YOLOv5 uses the CSPDarknet53 structure to extract the multi-scale feature map. Based on the trained parameters, the YOLOv5 network calculates the features of the RGB image and generates a first intermediate value FRGB. The first intermediate value FRGB is a feature representation of the RGB image and contains information such as the color and texture of the target. Then in step S32, since the pixels are aligned in advance, the first intermediate value of each pixel can be superimposed with the depth information to generate a second intermediate value FRGBD.

[0085] Finally, in step S33, the YOLOv5 network divides the input image into S×S grid cells, each of which is responsible for detecting the target whose center point falls within the cell. For each grid cell (i, j), the YOLOv5 network calculates the coordinates (x i,j ,y i,j ,w i,j , h i，j ) and the target category probability c i,j The specific formula is: i,j =σ(t x )+c x ,y i,j =σ(t y )+c y , Where: (t x , t y ,t w ,t h ) is the offset of the network prediction, σ is the Sigmoid function, which is used to limit the predicted value to the range [0,1], (c x , c y ) is the coordinate of the upper left corner of the grid cell, (p w , p h ) is the preset width and height of the anchor box. For each grid unit (i, j), the YOLOv5 network outputs the category probability c of the target i，j , according to the category probability c i，j The category with the highest probability is selected as the category of the target in the grid cell. Finally, the bounding boxes of the targets of the same category in the grid cells are fused to generate the final target detection result, that is, the optimal two-dimensional bounding box (xdet, ydet, wdet, hdet) of the target and the target category information.

[0086] It should also be noted that the YOLOv5 network of the present application adopts a non-maximum suppression algorithm to remove boxes with excessive overlap. Through the superposition of bimodal data (steps S31-S33), the color information and depth information are input into the YOLOv5 network, which makes up for the shortcomings of a single modality in target classification (such as distinguishing between pallets and goods) and occlusion scenarios, and is an atypical improvement to YOLOv5. The specific process includes: while outputting the target category, the YOLOv5 network outputs the category probability of the target as the confidence level, and sorts the bounding boxes corresponding to each target from high to low according to the confidence level. Select the bounding box with the highest confidence level, and remove other bounding boxes with its IoU (intersection over union) exceeding the preset threshold. Repeat the above process until all bounding boxes are processed, and the remaining targets and bounding boxes are used as input for subsequent bounding box fusion.

[0087] Step S4: Combine the coordinates of the detected two-dimensional bounding box of the target with the depth information in the depth image to calculate the position information of the target in the three-dimensional space.

[0088] In some optional embodiments, step S4 further includes:

[0089] Step S41, determining the coordinates of the center point of the two-dimensional bounding box of the target;

[0090] Step S42: searching for depth information at the center point coordinates in the aligned depth image;

[0091] Step S43: Calculate the three-dimensional space coordinates according to the center point coordinates and the depth information as the position information of the target.

[0092] In this embodiment, the center point (x c ,y c ), in step S42, the corresponding depth value Zc is obtained from the aligned depth image D′: Z c =D′(x c ,y c In step S43, the center point of the two-dimensional bounding box of the target (x c ,y c ) and the depth value Zc are converted into three-dimensional space coordinates (X c ,Y c , Z c ):

[0093]

[0094] Step S4 obtains the three-dimensional spatial coordinates of the center point of the target and uses this as the location information of the target to determine whether the target has been put into storage and whether the location after storage is accurate. For example, if it is stipulated that the goods A needs to be placed on shelf M, then the location information of the goods A calculated in step S4 is compared with the spatial range of the preset shelf M to determine whether the goods A has been put into storage on the correct shelf.

[0095] Step S5: searching for a corresponding setting box according to the position information of the target in the three-dimensional space.

[0096] The target location information calculated in step S4, in addition to determining whether the target is in storage, can also search the database for the setting box data at that location based on the specific target location information. The so-called setting box refers to the area on the specific shelf in the warehouse used to place goods. This area usually refers to a range that can stably store and correctly place goods, such as a certain area in the middle of a panel on a shelf. The setting box can be rectangular, circular or other shapes. Taking a rectangle as an example, its coordinates are expressed as (xref, yref, wref, href), where: (xref, yref) is the coordinate of the upper left corner of the setting box. Wref and href are the width and height of the setting box, respectively. The database stores a large amount of setting box information in the warehouse. Once the target location information is determined, the only setting box that matches it can be queried.

[0097] Step S6: determining whether the placement of the target is compliant based on the setting frame information and the two-dimensional bounding box of the target, and issuing an alarm signal when the placement of the target is not compliant.

[0098] In some optional implementations, step S6 further includes:

[0099] Step S61: determining the intersection area and union area of ​​the target's two-dimensional bounding box and the set box;

[0100] Step S62: calculating an intersection-to-union ratio based on the intersection area and the union area;

[0101] Step S63: judging whether the placement of the target is compliant based on the IoU ratio and a preset threshold value; when the IoU ratio is lower than the threshold value, the placement of the target is not compliant; otherwise, the placement of the target is compliant.

[0102] In this embodiment, the intersection area and the union area are calculated in step S61, where:

[0103] Coordinate of the upper left corner of the intersection area: x inter =max(x ref ,x det ), y inter =max(y ref ,ydet );

[0104] Lower right corner coordinates of the intersection area: x inter_end = min(x ref + w ref , x det + w det ), y inter_end = min(y ref + h ref , y det + h det );

[0105] Thus, calculate the width of the intersection area: w inter = x inter_end - x inter [[ID=३३]]), and calculate the height of the intersection area: h inter = y inter_end - y inter .

[0106] Finally, calculate the area of the intersection area: Area of Intersection = w inter × h inter .

[0107] After that, the area of the union area can be calculated as:

[0108] Area of Union = w ref × h ref + w det × h det - Area of Intersection.

[0109] In step S62, calculate the Intersection over Union IoU:

[0110] In step S63, set an Intersection over Union threshold IoUth, and determine whether the IoU between the detection box and the set box is lower than the threshold: If IoU < IoUth, it is considered that the goods placement is non-compliant.

[0111] It should be noted that: migrating IoU (step S6) from the traditional object detection field to 3D warehouse compliance monitoring, dynamically determining whether the goods cross the boundary in combination with the set box (such as the preset area of the shelf), solves the problem of low efficiency of traditional manual rule judgment. Step S6 can detect whether the target placement is compliant in real time during the process of the operator placing the goods, so as to send an alarm signal to prompt the operator to make modifications.

[0112] Example 2

[0113] Such as Figure 2As shown, the parts not described in detail in this embodiment are as shown in Example 1. This embodiment provides a three-dimensional warehouse real-time monitoring device, including an image acquisition module, an image alignment module, a target recognition module, a target position information calculation module, a setting frame acquisition module and a target pose estimation module. The modules are connected to each other via wired and / or wireless connections:

[0114] Image acquisition module, used to obtain RGB images and depth images containing storage targets;

[0115] An image alignment module, configured to spatially align the RGB image with the depth image;

[0116] The object recognition module is used to input the aligned RGB image and depth image into the pre-trained YOLOv5 network to determine the object category and the two-dimensional bounding box of each object in the image;

[0117] The target position information calculation module is used to combine the coordinates of the two-dimensional bounding box of the detected target with the depth information in the depth image to calculate the position information of the target in three-dimensional space;

[0118] A setting frame acquisition module, configured to search for a corresponding setting frame according to the position information of the target in the three-dimensional space;

[0119] The target pose estimation module is used to determine whether the placement of the target is compliant based on the setting box information and the two-dimensional bounding box of the target, and to issue an alarm signal when the placement of the target is not compliant.

[0120] In some optional implementations, the image alignment module includes:

[0121] A projection calculation unit, used to determine the corresponding point of each pixel of the depth image in the RGB image;

[0122] The interpolation calculation unit is used to interpolate the corresponding depth information for each pixel in the RGB image through the four nearest corresponding points around it.

[0123] In some optional embodiments, the target recognition module includes:

[0124] a first intermediate value calculation unit, configured to extract color information of each pixel in the aligned RGB image and generate a first intermediate value based on a trained first parameter combination;

[0125] a second intermediate value calculation unit, configured to superimpose the first intermediate value with the depth information of each pixel in the aligned depth image to form a second intermediate value;

[0126] The target recognition unit is used to calculate the second intermediate value based on the trained second parameter combination to obtain the target category and target bounding box information in each preset grid unit.

[0127] In some optional implementations, the target position information calculation module includes:

[0128] A center point coordinate determination unit, used to determine the coordinates of the center point of the two-dimensional bounding box of the target;

[0129] a depth information determining unit, configured to find the depth information at the coordinates of the center point in the aligned depth image;

[0130] The position information determining unit is used to calculate the three-dimensional space coordinates according to the center point coordinates and the depth information as the position information of the target.

[0131] In some optional implementations, the target pose estimation module includes:

[0132] An intersection and union region determining unit, configured to determine an intersection region and a union region between a two-dimensional bounding box of the target and the set box;

[0133] an intersection-over-union ratio calculation unit, configured to calculate an intersection-over-union ratio based on the intersection area and the union area;

[0134] The compliance judgment unit is used to judge whether the placement of the target is compliant based on the intersection-over-union ratio and a preset threshold value. When the intersection-over-union ratio is lower than the threshold value, the placement of the target is not compliant; otherwise, the placement of the target is compliant.

[0135] Example 3

[0136] The parts not described in detail in this embodiment are as shown in Example 1. This embodiment provides a computer device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the three-dimensional warehouse real-time monitoring method as described above.

[0137] Example 4

[0138] As described in Example 1, this embodiment provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program is used to implement the above-described method for real-time monitoring of three-dimensional warehouses. The computer-readable storage medium may be included in the apparatus described in the above embodiments, or it may exist independently and not be incorporated into the apparatus. The computer-readable storage medium carries one or more programs, and when executed by the apparatus, the one or more programs process data according to the above-described method.

[0139] The computer program for running the three-dimensional warehouse real-time monitoring method of the present application can be set on the mobile robot chip, and can also be set on a computer device remotely connected to the mobile robot. When it is installed on the remote computer device, refer to Figure 3 , which shows a structural diagram of a computer device 400 suitable for implementing the embodiments of the present application. Figure 2 The computer device shown is only an example and should not limit the functions and scope of use of the embodiments of the present application.

[0140] like Figure 3 As shown, computer device 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage portion 408 into a random access memory (RAM) 403. Various programs and data required for the operation of device 400 are also stored in RAM 403. CPU 401, ROM 402, and RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to bus 404.

[0141] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, and the like; an output section 407 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 408 including a hard disk; and a communication section 409 including a network interface card such as a LAN card or a modem. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. Removable media 411, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 410 as needed, so that computer programs read therefrom can be installed into the storage section 408 as needed.

[0142] In particular, according to the embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the central processing unit (CPU) 401, the above-mentioned functions defined in the method of the present application are executed. It should be noted that the computer storage medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wire, optical cable, RF, or any suitable combination thereof.

[0143] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the module, program segment or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0144] The modules or units described in the embodiments of the present application may be implemented in software or hardware. The modules or units described may also be provided in a processor, and the names of these modules or units do not, in certain circumstances, limit the modules or units themselves.

[0145] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A three-dimensional warehouse real-time monitoring method, characterized in that: The following steps are involved: Step S1: Acquire an RGB image and a depth image containing a storage target; Step S2: spatially aligning the RGB image with the depth image; Step S3: Input the aligned RGB image and depth image into the pre-trained YOLOv5 network to determine the object category and the two-dimensional bounding box of each object in the image; Step S4: combining the coordinates of the detected two-dimensional bounding box of the target with the depth information in the depth image to calculate the position information of the target in three-dimensional space; Step S5: searching a matching setting frame area from a database according to the position information of the target in the three-dimensional space; Step S6: Determine whether the placement of the target is compliant based on the intersection-over-union ratio of the target two-dimensional bounding box and the set box area, and issue an alarm signal for storage targets that are not placed in compliance.

2. The three-dimensional warehouse real-time monitoring method according to claim 1, characterized in that: The logic for obtaining the object categories and the two-dimensional bounding boxes of each object in the image is: Determine the corresponding point of each pixel of the depth image in the RGB image; For each pixel in the RGB image, the corresponding depth information is interpolated through the four nearest corresponding points around it; Extracting color information of each pixel in the aligned RGB image, and generating a first intermediate value based on the trained first parameter combination; Superimposing the first intermediate value with the depth information of each pixel in the aligned depth image to form a second intermediate value; For the second intermediate value, information about the target category and the target bounding box in each preset grid unit is obtained by calculation based on the trained second parameter combination.

3. The three-dimensional warehouse real-time monitoring method according to claim 2, characterized in that: Step S4 includes: Determine the coordinates of the center point of the target's two-dimensional bounding box; Find the depth information at the center point coordinates in the aligned depth image; The three-dimensional space coordinates are calculated according to the center point coordinates and the depth information as the position information of the target.

4. The three-dimensional warehouse real-time monitoring method according to claim 3, characterized in that: Step S6 includes: Determine the intersection area and union area of ​​the two-dimensional bounding box of the target and the set box; Calculating an intersection-over-union ratio based on the intersection area and the union area; Whether the placement of the target is compliant is determined based on the intersection-over-union ratio and a preset threshold value. When the intersection-over-union ratio is lower than the threshold value, the placement of the target is not compliant; otherwise, the placement of the target is compliant.

5. A three-dimensional warehouse real-time monitoring device, based on the implementation of the three-dimensional warehouse real-time monitoring method according to any one of claims 1 to 5, comprising an image acquisition module, an image alignment module, a target recognition module, a target position information calculation module, a setting frame acquisition module, and a target pose estimation module: Image acquisition module, used to obtain RGB images and depth images containing storage targets; An image alignment module, configured to spatially align the RGB image with the depth image; The object recognition module is used to input the aligned RGB image and depth image into the pre-trained YOLOv5 network to determine the object category and the two-dimensional bounding box of each object in the image; The target position information calculation module is used to combine the coordinates of the two-dimensional bounding box of the detected target with the depth information in the depth image to calculate the position information of the target in three-dimensional space; A setting frame acquisition module, configured to search for a corresponding setting frame according to the position information of the target in the three-dimensional space; The target pose estimation module is used to determine whether the placement of the target is compliant based on the setting box information and the two-dimensional bounding box of the target, and to issue an alarm signal when the placement of the target is not compliant.

6. The three-dimensional warehouse real-time monitoring device according to claim 5, characterized in that: Obtain the target category and the two-dimensional bounding box of each target in the image based on the image alignment module and the target recognition module; The image alignment module includes a projection calculation unit and an interpolation calculation unit; A projection calculation unit, used to determine the corresponding point of each pixel of the depth image in the RGB image; The interpolation calculation unit is used to interpolate the corresponding depth information for each pixel in the RGB image through the four nearest corresponding points around it. The target recognition module includes a first intermediate value calculation unit and a second intermediate value calculation unit; a first intermediate value calculation unit, configured to extract color information of each pixel in the aligned RGB image and generate a first intermediate value based on a trained first parameter combination; a second intermediate value calculation unit, configured to superimpose the first intermediate value with the depth information of each pixel in the aligned depth image to form a second intermediate value; The target recognition unit is used to calculate the second intermediate value based on the trained second parameter combination to obtain the target category and target bounding box information in each preset grid unit.

7. The three-dimensional warehouse real-time monitoring device according to claim 6, characterized in that: The target position information calculation module includes: A center point coordinate determination unit, used to determine the coordinates of the center point of the two-dimensional bounding box of the target; a depth information determining unit, configured to find the depth information at the coordinates of the center point in the aligned depth image; The position information determining unit is used to calculate the three-dimensional space coordinates according to the center point coordinates and the depth information as the position information of the target.

8. The three-dimensional warehouse real-time monitoring device according to claim 7, characterized in that: The target pose estimation module includes: An intersection and union region determining unit, configured to determine an intersection region and a union region between a two-dimensional bounding box of the target and the set box; an intersection-over-union ratio calculation unit, configured to calculate an intersection-over-union ratio based on the intersection area and the union area; The compliance judgment unit is used to judge whether the placement of the target is compliant based on the intersection-over-union ratio and a preset threshold value. When the intersection-over-union ratio is lower than the threshold value, the placement of the target is not compliant; otherwise, the placement of the target is compliant.

9. A computer device, characterized in that: It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the three-dimensional warehouse real-time monitoring method according to any one of claims 1 to 4.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which, when executed by a processor, is used to implement the three-dimensional warehouse real-time monitoring method according to any one of claims 1 to 4.

Images