Single-camera mirror-assisted bin object dual-channel classification method and system

Abstract

The application discloses a kind of based on single camera mirror image auxiliary warehouse body object dual-channel classification method and system.The method includes: single camera is arranged at the entrance of warehouse body directly above, and mirror surface reflection element is arranged in one side inside warehouse body;Collect original image, and divide out the first image area without mirror surface reflection element and the second image area containing only mirror surface reflection element;The first image area without mirror surface reflection element and the second image area containing only mirror surface reflection element are respectively input into the two branches of dual-flow feature fusion network, and object first feature vector and object second feature vector are extracted and fused, to generate fusion feature vector;Finally, input classifier outputs classification result.The application uses single camera to cooperate mirror surface reflection element to obtain dual-view image, combines dual-channel feature fusion network, reduces hardware cost while improving the classification accuracy of occluded object, and is suitable for intelligent recycling, automatic sorting and other warehouse body scenes.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, specifically to a dual-channel classification method and system for objects in a warehouse based on single-camera mirroring assistance. Background Technology

[0002] In fields such as intelligent recycling and automated sorting, there is often a need for rapid classification and identification of objects (such as beverage bottles, bottle caps, and packaging boxes) within the storage area. Existing visual recognition solutions mainly fall into two categories: One type uses a single camera positioned directly above the cargo compartment (e.g.) Figure 1 As shown in the image, the identification is performed by acquiring an image of the top of the object. This solution has low hardware costs, but due to the single perspective, it is difficult to obtain side information of the object, resulting in an inability to accurately distinguish planar objects, difficulty in identifying bottle cap types, and significant errors in estimating dimensions such as bottle length.

[0003] Another type uses two or more cameras to shoot from multiple directions (such as...) Figure 2 As shown in the figure, the images are then stitched together using a fusion algorithm. While this approach can obtain more complete object information, it increases hardware costs, complicates the layout, and makes the design of multi-view data fusion algorithms difficult, resulting in poor real-time performance.

[0004] In addition, in the aforementioned single-camera system, when an object moves inside the cabin, some areas may be continuously obscured by the object itself or the cabin structure, and existing technologies lack effective solutions.

[0005] Therefore, there is an urgent need in this field for a high-precision method and system for classifying objects in warehouses that can control hardware costs, acquire sufficient information from multiple perspectives, and is unaffected by occlusion. Summary of the Invention

[0006] To address the problems in existing technologies, such as incomplete object information due to the limited field of view of a single camera, high cost and complex fusion of multi-camera solutions, and occlusion caused by object movement, this invention provides a dual-channel classification method and system for warehouse objects based on single-camera mirroring assistance, in order to solve the aforementioned technical deficiencies.

[0007] This invention proposes a dual-channel classification method for objects in a warehouse based on single-camera mirroring, comprising the following steps: S1. Place a single camera directly above the entrance at the top of the cabin, and place a mirror reflector on one side inside the cabin, so that the camera's original field of view and the mirrored virtual field of view formed by the reflection of the mirror reflector can each cover the entire interior area of ​​the cabin from their respective directions, thus obtaining a dual-channel coverage optical configuration. S2. Under the dual-channel coverage optical configuration, the original image of the object to be classified inside the chamber is acquired, and based on the preset segmentation algorithm, the first image region without specular reflection elements and the second image region containing only specular reflection elements are segmented from the original image. S3. Input the first image region without specular reflection elements and the second image region containing only specular reflection elements into the two feature extraction branches of the dual-stream feature fusion network respectively, extract the first feature vector and the second feature vector of the object, and fuse the first feature vector and the second feature vector of the object to generate a fused feature vector. S4. Input the fused feature vector into the classifier, perform object classification, and obtain and output the classification result.

[0008] Preferably, the mirror reflection element is a plane mirror, and step S1 specifically includes the following sub-steps: S11. Determine the installation angle θ of the plane mirror so that it satisfies the following relationship: ; Where DC is the length of the compartment, FA is the vertical height of the camera from the bottom of the compartment, and DA is the height of the compartment entrance; S12. Determine the length EC of the plane mirror such that it satisfies the following relationship: ; Where EC is the length of the plane mirror and θ is the installation angle.

[0009] Preferably, in step S2, based on a preset segmentation algorithm, a first image region without specular reflection elements and a second image region containing only specular reflection elements are segmented from the original image, specifically including the following sub-steps: S21. During the initialization calibration phase and when there is no object to be tested in the chamber, continuously acquire multiple frames of background images and take the average value to obtain the calibration background image, and convert the calibration background image into a grayscale image. S22. Perform edge detection on the grayscale image to determine the fixed coordinate range of the second image region containing only the specular reflection element in the original image; wherein, the edge detection includes: using the Canny operator to capture the contour of the specular reflection element, connecting the broken edges through morphological closing operation to form a closed contour, selecting the largest closed contour on one side of the image where the specular reflection element is located to calculate the bounding rectangle, and obtaining the fixed coordinate range. S23. In actual operation, based on a fixed coordinate range, the second image region containing only specular reflection elements and the first image region without specular reflection elements are directly cropped from the acquired original image.

[0010] Preferably, before inputting the first image region without specular reflection elements and the second image region containing only specular reflection elements into the dual-stream feature fusion network, the method further includes performing a horizontal flipping operation on the second image region containing only specular reflection elements, so that the orientation of the object in the flipped second image region containing only specular reflection elements is consistent with the actual orientation of the object in the first image region without specular reflection elements.

[0011] Preferably, in step S3, the first image region without specular reflection elements and the second image region containing only specular reflection elements are respectively input into the dual-stream feature fusion network, specifically including the following sub-steps: S31. Input the first image region without specular reflection elements into the first feature extraction branch to obtain the first feature vector of the object; S32. Input the second image region containing only the specular reflection element into the second feature extraction branch to obtain the second feature vector of the object; The first feature extraction branch and the second feature extraction branch share the same backbone network and network weights. The backbone network is a pre-trained ResNet-50 convolutional neural network. Furthermore, the two-stream feature fusion network also includes a fully connected layer connected after feature fusion, which acts as a classifier to output classification results based on the fused feature vector.

[0012] In a further preferred embodiment, the fusion of the first feature vector and the second feature vector of the object in step S3 is specifically achieved by fusing the first feature vector and the second feature vector of the object through vector concatenation or element-wise addition to generate a fused feature vector; wherein, vector concatenation is to connect the first feature vector and the second feature vector of the object end to end, and element-wise addition to the vectors is to add the first feature vector and the second feature vector of the object element by element.

[0013] Preferably, after step S2 and before step S3, the following steps are also included: The first image region without specular reflection elements and the second image region containing only specular reflection elements are preprocessed by resizing and normalizing to obtain the preprocessed first image region without specular reflection elements and the preprocessed second image region containing only specular reflection elements; wherein, resizing means adjusting the first image region without specular reflection elements and the second image region containing only specular reflection elements to the same preset input size, and normalization means scaling the image pixel values ​​to the range of [0,1]; Furthermore, in step S3, the first image region without specular reflection elements and the second image region containing only specular reflection elements are respectively input into the dual-stream feature fusion network. Specifically, the preprocessed first image region without specular reflection elements and the preprocessed second image region containing only specular reflection elements are respectively input into the dual-stream feature fusion network.

[0014] This invention also proposes a dual-channel classification system for cargo objects based on single-camera mirroring, used to implement any of the methods described above. The system includes: The optical imaging module includes a single camera and a specular reflector. The camera is positioned directly above the entrance at the top of the cabin, and the specular reflector is positioned on one side inside the cabin. The optical imaging module is configured to provide both the camera's original field of view and the mirrored virtual field of view formed by reflection through the specular reflector to independently cover the optical environment of the entire interior area of ​​the cabin from their respective directions. The image acquisition and segmentation module is configured to acquire the original image of the interior of the container containing the object in the optical environment provided by the optical imaging module, and to segment the original image into a first image region without specular reflection elements and a second image region containing only specular reflection elements based on a preset segmentation algorithm. The dual-stream feature fusion module has a built-in dual-stream feature fusion network. It is configured to input the first image region without specular reflection elements and the second image region containing only specular reflection elements into the two feature extraction branches of the dual-stream feature fusion network, respectively, to extract the first feature vector and the second feature vector of the object, and then fuse the first feature vector and the second feature vector of the object to generate a fused feature vector. The classification output module has a built-in classifier and is configured to input the fused feature vector into the classifier, perform object classification, and output the classification result.

[0015] The present invention also proposes a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, it implements the steps of the dual-channel classification method for cargo objects based on single-camera image assistance as described above.

[0016] The present invention also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the dual-channel classification method for cargo objects based on a single-camera image assisted as described above.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) A single camera with a mirror reflector is used instead of a multi-camera solution. No additional image acquisition equipment or complex multi-view calibration and fusion algorithms are required. This significantly reduces the system hardware cost while acquiring dual-view information.

[0018] (2) Using mirror reflection elements to expand the field of view is beneficial to the accurate extraction of AI features in the future.

[0019] (3) The mirror image of the object from the other side is obtained by reflecting the mirror reflection element. When one view of the object is blocked, the other view can still provide effective local features. The dual-channel feature fusion network can integrate the information from the two viewpoints, enhance the semantic understanding of the occluded area, and improve the classification robustness.

[0020] (4) A fixed coordinate range is obtained by background modeling (multi-frame averaging, edge detection, morphological closing operation, and maximum contour extraction) during the initialization and calibration stage. The clipping is done directly at runtime without the need to calculate the mirror area in real time. The segmentation speed is fast and the positioning is accurate.

[0021] (5) The features of the real object image and the mirror image are extracted by a weight-sharing dual-stream ResNet-50 network, and fused by vector concatenation or summation. This enables the network to automatically learn the complementary relationship between the features of the two perspectives. Compared with a single perspective or simple image concatenation, it can obtain more discriminative fused features.

[0022] (6) From optical configuration, image acquisition, mirror segmentation, geometric alignment, preprocessing to dual-stream feature fusion and classification, a complete automated solution is formed, which can be directly deployed in scenarios such as intelligent recycling terminals and automatic sorting equipment. Attached Figure Description

[0023] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments, taken with reference to the accompanying drawings: Figure 1 This is a schematic diagram of a single-camera deployment scheme in existing technology; Figure 2 This is a schematic diagram of a dual-camera deployment scheme in existing technology; Figure 3 This is a flowchart of the dual-channel classification method for warehouse objects based on single-camera mirroring assistance according to the present invention; Figure 4 This is a geometric derivation diagram of the plane mirror installation angle and field of view boundary of the present invention; Figure 5 This is a schematic diagram of the field of view coverage when a convex mirror is used as a reflecting element in this invention; Figure 6 This is a schematic diagram of the dual-channel classification system for warehouse objects based on single-camera mirroring assistance of the present invention; Figure 7 This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present invention. Detailed Implementation

[0024] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0025] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0026] Figure 3 The flowchart illustrates a dual-channel classification method for cargo space objects based on single-camera mirroring assistance provided by this invention. (Reference) Figure 3 The method includes the following steps S1 to S4. Figure 4 This is a geometric derivation diagram of the installation angle of a plane mirror and the boundary of its field of view. Figure 5 This is a schematic diagram of the field of view coverage when a convex mirror is used as a reflective element in this invention. The following is a detailed description of each step.

[0027] S1. A single camera is positioned directly above the entrance at the top of the cabin, and a mirrored reflective element is placed on one side inside the cabin. This allows both the camera's original field of view and the mirrored virtual field of view formed by reflection from the mirrored reflective element to independently cover the entire interior area of ​​the cabin from their respective directions, achieving a dual-channel coverage optical configuration. It should be noted that the mirrored reflective element described in this invention is used to expand the camera's field of view through reflection, and the mirrored reflective element can be any of a plane mirror, a convex mirror, or a prism group.

[0028] like Figure 4 As shown, when the specular reflector is a plane mirror, let the length of the recognition chamber be AB (or denoted as DC, AB=DC), the inlet height be DA, and the outlet height be CB, where DA=CB=H. The camera F is positioned directly above the inlet A of the recognition chamber, with a vertical height of FA from the bottom surface of the chamber. The specular reflector EC is positioned inside the chamber on the right side, with a length of EC and an angle θ with the horizontal plane.

[0029] In order for both the camera's original field of view and the mirrored virtual field of view formed by reflection through the plane mirror to independently cover the entire internal area of ​​the cabin (from entrance point D to exit point B), the installation angle θ and length EC of the plane mirror need to be determined according to the following geometric relationship.

[0030] First, determine the installation angle θ of the plane mirror so that it satisfies the following relationship:

[0031] Where DC is the length of the compartment, FA is the vertical height of the camera from the bottom of the compartment, and DA is the height of the compartment entrance.

[0032] Secondly, determine the length EC of the plane mirror so that it satisfies the following relationship:

[0033] Where EC is the length of the plane mirror and θ is the installation angle.

[0034] Furthermore, to ensure the camera can capture the entire recognition chamber, a camera with a sufficiently wide angle of view needs to be selected. The required minimum wide angle of view, ∠DFE, can be calculated using the following formula:

[0035] When selecting a camera, the wide-angle angle should be greater than or equal to ∠DFE.

[0036] The width of the identification bin and the width of the plane mirror are determined based on the maximum size of the recyclable material. That is, the width of the plane mirror should not be less than the width of the bin to ensure that there are no blind spots in the reflected field of view.

[0037] When the mirror reflection element is a convex mirror, it must be ensured that both the camera's original field of view and the mirrored virtual field of view formed by the convex mirror can independently cover the entire recognition chamber from their respective directions. The size and curvature of the convex mirror are determined by the actual application scenario and there is no fixed angular relationship. Furthermore, the virtual point formed by the camera in the convex mirror also needs to be able to cover the entire recognition chamber, that is, both the mirrored virtual field of view and the original field of view can independently ensure coverage without blind spots from their respective directions. The size of the convex mirror is determined by its curvature; the larger the curvature, the wider the field of view. In other embodiments of the present invention, when the mirror reflection element is a prism group, the reflection or refraction characteristics of the prism can also be used to expand the mirrored virtual field of view, and its specific arrangement can be adjusted according to the optical design.

[0038] S2. Under the dual-channel coverage optical configuration, acquire the original image of the interior of the container containing the object, and based on the preset segmentation algorithm, segment the original image into a first image region without specular reflection elements and a second image region containing only specular reflection elements.

[0039] This step employs a segmentation algorithm that combines initialization calibration with direct runtime pruning, specifically including the following sub-steps S21 to S23.

[0040] S21. During the initialization and calibration phase, when there is no object to be tested inside the chamber, continuously acquire multiple frames of background images and take the average value to obtain the calibration background image, and convert the calibration background image into a grayscale image.

[0041] During the initial calibration phase, the positions of the camera and the specular reflector are fixed. At this time, there is no object to be measured inside the chamber, only the background and the specular reflector (and its reflected background). Three to five consecutive background images are acquired, and the pixel average of each frame is calculated to obtain a stable calibration background image. To improve the robustness of edge detection, this calibration background image is converted to a grayscale image; alternatively, it can be converted to the HSV color space (e.g., extracting the V channel). Both methods are acceptable. This embodiment uses a grayscale image as an example for explanation.

[0042] S22. Perform edge detection on the grayscale image to obtain the mirror boundary contour of the specular reflection element, and determine the fixed coordinate range of the second image region containing only the specular reflection element in the original image based on the mirror boundary contour; wherein, the edge detection includes: using the Canny operator to capture the contour of the specular reflection element, connecting the broken edges through morphological closing operation to form a closed contour, selecting the largest closed contour on one side of the image where the specular reflection element is located to calculate the circumscribed rectangle, and obtaining the fixed coordinate range.

[0043] Since there are usually detectable boundary features between the specular reflective element and the background, such as color differences, edge lines, or texture breaks, the specular contour can be extracted through edge detection. The Canny operator is used to perform edge detection on the grayscale image obtained in S21, resulting in a preliminary binary image of the specular reflective element's edge. The Canny operator can use an adaptive threshold or a fixed threshold set empirically. Since edges may be broken, a morphological closing operation (dilation followed by erosion) is used to connect adjacent edges into a closed contour. From the contour set, the closed contour with the largest area located on one side of the image containing the specular reflective element (e.g., the right side of the image containing the specular reflective element) is selected; this contour is the specular boundary of the specular reflective element. The bounding rectangle of this contour is calculated, and optionally extended outwards by a few pixels as a safety margin, thus obtaining a fixed coordinate range in the original image for the second image region containing only the specular reflective element, for example, represented by pixel coordinates (x_start, y_start, x_end, y_end). The fixed coordinate range is the region of interest (ROI) of the second image region containing only specular reflective elements; correspondingly, the left region of the original image (0, 0, x_start, Image_Height) is the first image region that does not contain specular reflective elements.

[0044] In other embodiments of the present invention, the edge detection operator is not limited to the Canny operator, but may also use the Sobel operator, the Laplacian operator, or a deep learning-based edge detection method; the morphological closing operation may also be replaced by other image processing methods that connect broken edges (such as dilation and erosion combination).

[0045] S23. In actual operation, based on a fixed coordinate range, the second image region containing only specular reflection elements and the first image region without specular reflection elements are directly cropped from the acquired original image.

[0046] After completing the S22 calibration, each time a new original image is acquired, edge detection is not repeated. Instead, the second image region containing only specular reflective elements and the first image region without specular reflective elements are directly cropped from the original image based on the two regions of interest (ROIs). This runtime direct cropping method can significantly improve processing speed.

[0047] In a specific embodiment, the method further includes performing a geometric transformation on the second image region containing only specular reflective elements, so that its spatial orientation is consistent with that of the first image region without specular reflective elements. Specifically, a flipping operation (i.e., left-right mirroring) is performed on the second image region containing only specular reflective elements, so that the orientation of objects in the flipped second image region containing only specular reflective elements is consistent with the actual orientation of objects in the first image region without specular reflective elements.

[0048] S3. Input the first image region without specular reflection elements and the second image region containing only specular reflection elements into the two feature extraction branches of the dual-stream feature fusion network, respectively, to extract the first feature vector and the second feature vector of the object, and fuse the first feature vector and the second feature vector of the object to generate a fused feature vector.

[0049] After step S2 and before step S3, the following preprocessing steps are also performed: The first image region without specular reflection elements and the second image region containing only specular reflection elements are preprocessed by resizing and normalizing to obtain the preprocessed first image region without specular reflection elements and the preprocessed second image region containing only specular reflection elements; wherein, resizing means adjusting the image to a preset input size, and normalization means scaling the image pixel values ​​to the range of [0,1].

[0050] Specifically, to meet the input image size requirements of the dual-stream feature fusion network, the first image region without specular reflection elements and the second image region containing only specular reflection elements are adjusted to the same input size preset by the network (e.g., 224×224 pixels). At the same time, the pixel values ​​of the image are linearly scaled from [0,255] to the [0,1] range to accelerate network convergence.

[0051] Then, perform the following sub-steps of step S3: S31. Input the first image region without specular reflection elements into the first feature extraction branch to obtain the first feature vector of the object.

[0052] The first feature extraction branch uses a pre-trained ResNet-50 convolutional neural network as its backbone. The pre-processed first image region, excluding specular reflection elements, is input into this branch. Through forward propagation of the network, it outputs a high-dimensional first feature vector of the object. This step utilizes transfer learning techniques, loading weights pre-trained on large datasets (such as ImageNet), to achieve high recognition accuracy even with small sample sizes.

[0053] S32. Input the second image region containing only the specular reflection element into the second feature extraction branch to obtain the second feature vector of the object.

[0054] The second feature extraction branch shares the same backbone network and network weights as the first feature extraction branch. The backbone network is a pre-trained ResNet-50 convolutional neural network. The second image region containing only specular reflection elements is input into this branch to extract the object's second feature vector. Because the two branches share weights, the network is able to learn common features between the real-world view and the mirrored view.

[0055] In other embodiments of the present invention, the backbone network can be replaced by SwingTransformer, VisionTransformer (ViT) or other equivalent deep neural network models to achieve the extraction and fusion of dual-view features.

[0056] Extracting the first feature vector of the object and the object's second eigenvector Next, the two need to be merged. The specific fusion method is as follows: the first feature vector of the object and the second feature vector of the object are merged by concatenating vectors or adding vectors element by element to generate a fused feature vector; where concatenating vectors means connecting the first feature vector of the object and the second feature vector of the object end to end, and adding vectors element by element means adding the first feature vector of the object and the second feature vector of the object element by element.

[0057] Vector concatenation (Concat): ... and Connecting the first and last ends, the resulting new vector has the dimension of the sum of the dimensions of the two vectors, i.e. .

[0058] Adding vectors element by element: and Adding corresponding elements together results in a new vector with the same dimensions. .

[0059] S4. Input the fused feature vector into the classifier, perform object classification, and obtain and output the classification result.

[0060] In a two-stream feature fusion network, a fully connected layer (i.e., a classifier) ​​is connected after the feature fusion layer. The fused feature vector generated by S3 is then processed. The fully connected layer is input, undergoes fully connected operations and a softmax activation function, and outputs a probability distribution. The category corresponding to the highest probability is the classification result of the object (e.g., "plastic bottle", "aluminum can", "glass bottle"). Finally, the system outputs this classification result for subsequent sorting, counting, or other processing.

[0061] Further reference Figure 6 As an implementation of the above method, this invention also proposes a dual-channel classification system for cargo objects based on single-camera mirroring assistance. This system can be applied to various electronic devices. The dual-channel classification system 600 for cargo objects based on single-camera mirroring assistance includes the following modules: The optical imaging module 610 includes a single camera and a specular reflector. The camera is positioned directly above the entrance at the top of the cabin, and the specular reflector is positioned on one side inside the cabin. The optical imaging module is configured to provide both the camera's original field of view and the mirrored virtual field of view formed by reflection through the specular reflector to independently cover the optical environment of the entire interior area of ​​the cabin from their respective directions. The image acquisition and segmentation module 620 is configured to acquire the original image of the interior of the container containing the object in the optical environment provided by the optical imaging module, and to segment the original image into a first image region without specular reflection elements and a second image region containing only specular reflection elements based on a preset segmentation algorithm. The dual-stream feature fusion module 630 has a built-in dual-stream feature fusion network. It is configured to input a first image region without specular reflection elements and a second image region containing only specular reflection elements into two feature extraction branches of the dual-stream feature fusion network, respectively, to extract the first feature vector and the second feature vector of the object, and then fuse the first feature vector and the second feature vector of the object to generate a fused feature vector. The classification output module 640 has a built-in classifier and is configured to input the fused feature vector into the classifier, perform object classification, and output the classification result.

[0062] The present invention also proposes a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, it implements the steps of the dual-channel classification method for cargo objects based on single-camera image assistance as described above.

[0063] The present invention also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the dual-channel classification method for cargo objects based on a single-camera image assisted as described above.

[0064] The following is for reference. Figure 7 It shows a schematic diagram of the structure of a computer system 700 suitable for implementing terminal devices or servers in the embodiments of this application. Figure 7 The terminal device or server shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0065] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 702 or programs loaded from storage section 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the system 700. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0066] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a liquid crystal display (LCD) and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card and a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0067] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing unit (CPU) 701, it performs the functions defined in the methods of this application. It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable medium or any combination thereof. The computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0068] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0069] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0070] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention is not limited to the specific combination of the above-described technical features, but also includes other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A dual-channel classification method for objects in a warehouse based on single-camera mirroring assistance, characterized in that, Includes the following steps: S1. Place a single camera directly above the entrance at the top of the cabin, and place a mirror reflection element on one side inside the cabin, so that the original field of view of the camera and the mirror virtual field of view formed by the reflection of the mirror reflection element can each cover the entire interior area of ​​the cabin from their respective directions, thus obtaining a dual-channel coverage optical configuration. S2. Under the dual-channel coverage optical configuration, the original image of the object to be classified inside the chamber is acquired, and based on the preset segmentation algorithm, a first image region without specular reflection elements and a second image region containing only specular reflection elements are segmented from the original image. S3. Input the first image region without specular reflection elements and the second image region containing only specular reflection elements into the two feature extraction branches of the dual-stream feature fusion network respectively to extract the first feature vector and the second feature vector of the object, and fuse the first feature vector and the second feature vector of the object to generate a fused feature vector. S4. Input the fused feature vector into the classifier, perform object classification, and obtain and output the classification result.

2. The dual-channel classification method for warehouse objects based on single-camera mirroring assistance according to claim 1, characterized in that, The mirror reflection element is a plane mirror, and step S1 specifically includes the following sub-steps: S11. Determine the installation angle θ of the plane mirror so that it satisfies the following relationship: ； Where DC is the length of the compartment, FA is the vertical height of the camera from the bottom of the compartment, and DA is the height of the compartment entrance; S12. Determine the length EC of the plane mirror to satisfy the following relationship: ； Where EC is the length of the plane mirror, and θ is the installation angle.

3. The dual-channel classification method for warehouse objects based on single-camera mirroring assistance according to claim 1, characterized in that, In step S2, based on a preset segmentation algorithm, a first image region without specular reflection elements and a second image region containing only specular reflection elements are segmented from the original image. This specifically includes the following sub-steps: S21. During the initialization calibration phase and when there is no object to be tested in the chamber, multiple frames of background images are continuously acquired and the average value is taken to obtain a calibration background image, and the calibration background image is converted into a grayscale image. S22. Perform edge detection on the grayscale image to determine the fixed coordinate range of the second image region containing only the specular reflection element in the original image; wherein, the edge detection includes: using the Canny operator to capture the outline of the specular reflection element, connecting the broken edges through morphological closing operation to form a closed outline, selecting the largest closed outline on one side of the image where the specular reflection element is located to calculate the circumscribed rectangle, and obtaining the fixed coordinate range; S23. In actual operation, based on the fixed coordinate range, the second image region containing only the specular reflection element and the first image region without the specular reflection element are directly cropped from the acquired original image.

4. The dual-channel classification method for warehouse objects based on single-camera mirroring assistance according to claim 1, characterized in that, Before inputting the first image region without specular reflection elements and the second image region containing only specular reflection elements into the dual-stream feature fusion network, the method further includes performing a horizontal flipping operation on the second image region containing only specular reflection elements, so that the orientation of the object in the flipped second image region containing only specular reflection elements is consistent with the actual orientation of the object in the first image region without specular reflection elements.

5. The dual-channel classification method for warehouse objects based on single-camera mirroring assistance according to claim 1, characterized in that, In step S3, the first image region without specular reflection elements and the second image region containing only specular reflection elements are respectively input into the dual-stream feature fusion network, specifically including the following sub-steps: S31. Input the first image region without specular reflection elements into the first feature extraction branch to obtain the first feature vector of the object; S32. Input the second image region containing only the specular reflection element into the second feature extraction branch to obtain the second feature vector of the object; The first feature extraction branch and the second feature extraction branch share the same backbone network and network weights. The backbone network is a pre-trained ResNet-50 convolutional neural network. Furthermore, the two-stream feature fusion network also includes a fully connected layer connected after feature fusion, which serves as the classifier for outputting classification results based on the fused feature vector.

6. The dual-channel classification method for warehouse objects based on single-camera mirroring assistance according to claim 5, characterized in that, In step S3, the first feature vector of the object and the second feature vector of the object are fused together by concatenating the vectors or adding the vectors element by element to generate the fused feature vector. The vector concatenation is to connect the first feature vector of the object and the second feature vector of the object end to end, and the element-by-element addition is to add the first feature vector of the object and the second feature vector of the object element by element.

7. The dual-channel classification method for warehouse objects based on single-camera mirroring assistance according to claim 1, characterized in that, The following steps are included after step S2 and before step S3: The first image region without specular reflection elements and the second image region containing only specular reflection elements are preprocessed with size adjustment and normalization to obtain the preprocessed first image region without specular reflection elements and the preprocessed second image region containing only specular reflection elements; wherein, the size adjustment is to adjust the first image region without specular reflection elements and the second image region containing only specular reflection elements to the same preset input size, and the normalization is to scale the image pixel values ​​to the [0,1] range; Furthermore, in step S3, the first image region without specular reflection elements and the second image region containing only specular reflection elements are respectively input into the dual-stream feature fusion network. Specifically, the preprocessed first image region without specular reflection elements and the preprocessed second image region containing only specular reflection elements are respectively input into the dual-stream feature fusion network.

8. A dual-channel classification system for cargo containers based on single-camera mirroring assistance, used to implement the method as described in any one of claims 1 to 7, characterized in that, The system includes: An optical imaging module includes a single camera and a specular reflector. The camera is positioned directly above the entrance at the top of the cabin, and the specular reflector is positioned on one side inside the cabin. The optical imaging module is configured to provide both the original field of view of the camera and the mirrored virtual field of view formed by reflection through the specular reflector, each of which can independently cover the optical environment of the entire interior area of ​​the cabin from their respective directions. The image acquisition and segmentation module is configured to acquire an original image of the interior of the container containing the object under the optical environment provided by the optical imaging module, and to segment a first image region without specular reflection elements and a second image region containing only specular reflection elements from the original image based on a preset segmentation algorithm. The dual-stream feature fusion module has a built-in dual-stream feature fusion network. It is configured to input the first image region without specular reflection elements and the second image region containing only specular reflection elements into the two feature extraction branches of the dual-stream feature fusion network, respectively, to extract the first feature vector and the second feature vector of the object, and then fuse the first feature vector and the second feature vector of the object to generate a fused feature vector. The classification output module has a built-in classifier and is configured to input the fused feature vector into the classifier, perform object classification, and output the classification result.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the dual-channel classification method for cargo objects based on single-camera image assistance as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the dual-channel classification method for cargo objects based on single-camera image assistance as described in any one of claims 1 to 7.

Images