A vault automation method, system, device and medium

Abstract

This application relates to a method, system, apparatus, and medium for vault automation. The method includes: in response to a vault scheduling command, acquiring two-dimensional and three-dimensional images of the corresponding scheduling scenario using a camera; performing box recognition based on the three-dimensional image to obtain a region recognition result for the cash box; converting the region recognition result onto a two-dimensional image using the camera's intrinsic and extrinsic parameters to obtain a two-dimensional mask region; performing color recognition based on the two-dimensional mask region and the two-dimensional image to obtain a color recognition result for the two-dimensional mask region; and determining the target cash box corresponding to the vault scheduling command from the scheduling scenario based on the region recognition result and the color recognition result. This application achieves color recognition of the box point cloud region based on the color features of the 2D image, in addition to recognizing the box point cloud region based on the 3D image, thus improving the accuracy of cash box recognition in vault automation and solving the problem of how to improve the automation accuracy of intelligent vaults.

Description

Technical Field

[0001] The present application relates to the field of warehouse automation technology, and in particular to a vault automation method, system, device and medium. Background Art

[0002] With the development of science and technology, the methods of vault management are also changing with each passing day. The previous manual management methods now have problems such as low management efficiency, high labor intensity, and inaccurate information processing. The existing intelligent vault solutions, such as the patent with application number 202111285025.7, disclose an automatic sorting system for an intelligent vault, including a transport robot, an initial sorting robot, a secondary sorting conveyor line, a manual sorting conveyor line, a secondary sorting robot, and an empty cash box shelving robot. Although it improves the management efficiency of the vault operation center and reduces manual labor, its sorting robot only relies on ordinary 3D visual recognition components to identify the location of the cash box for grabbing the cash box. It can be seen that it is difficult to achieve refined identification and grabbing of the cash box, which can easily lead to errors in the entry and exit of the cash box and information processing errors in the intelligent vault.

[0003] Currently, no effective solution has been proposed for the problem of how to improve the automation accuracy of intelligent safes in related technologies. Summary of the invention

[0004] The embodiments of the present application provide a safe automation method, system, device and medium to at least solve the problem of how to improve the automation accuracy of an intelligent safe in the related art.

[0005] In a first aspect, an embodiment of the present application provides a vault automation method, the method comprising:

[0006] In response to the vault dispatch instruction, acquiring a two-dimensional image and a three-dimensional image of a dispatch scene corresponding to the vault dispatch instruction through a camera;

[0007] Based on the three-dimensional image, box recognition is performed from the three-dimensional image by a box recognition algorithm to obtain a region recognition result of the cash box, wherein the region recognition result includes a three-dimensional mask region of the cash box;

[0008] The region recognition result is converted to the two-dimensional image by using the camera intrinsic parameters and camera extrinsic parameters of the camera to obtain a two-dimensional mask region;

[0009] Based on the two-dimensional mask area and the two-dimensional image, color recognition is performed by a color recognition algorithm to obtain a color recognition result of the two-dimensional mask area;

[0010] Based on the area recognition result and the color recognition result, a target cash box corresponding to the vault scheduling instruction is determined from the scheduling scenario.

[0011] In some embodiments, obtaining a two-dimensional image and a three-dimensional image of a dispatching scene corresponding to the vault dispatching instruction by a camera includes:

[0012] Acquire a two-dimensional image of the dispatching scene corresponding to the vault dispatching instruction by a 2D camera;

[0013] A three-dimensional image of the scheduling scene corresponding to the vault scheduling instruction is obtained through a 3D camera.

[0014] In some embodiments, the region recognition result is converted to the two-dimensional image through the camera intrinsic parameters and camera extrinsic parameters of the camera, and the two-dimensional mask region is obtained, including:

[0015] The region recognition result is converted into a world coordinate system by using the camera intrinsic parameters and camera extrinsic parameters of the 3D camera;

[0016] The region recognition result converted to the world coordinate system is converted to the two-dimensional image through the camera intrinsic parameters and camera extrinsic parameters of the 2D camera to obtain a two-dimensional mask region.

[0017] In some embodiments, based on the three-dimensional image, box recognition is performed from the three-dimensional image by a box detection algorithm, and the area recognition result of the cash box is obtained, including:

[0018] Based on the three-dimensional image and the ROI area, extracting regional point cloud data corresponding to the ROI area;

[0019] Based on the regional point cloud data, box recognition is performed in the ROI area using a box detection algorithm to obtain a regional recognition result of the cash box.

[0020] In some embodiments, the method comprises:

[0021] Acquire a depth image of a dispatching scene corresponding to the vault dispatching instruction through a camera;

[0022] Based on the depth image, a conversion ratio between a pixel coordinate system and a world coordinate system of an object in the scheduling scene is calculated.

[0023] In some embodiments, based on the three-dimensional image and the ROI area, extracting regional point cloud data corresponding to the ROI area includes:

[0024] Based on the three-dimensional image and the ROI area, extracting regional point cloud data corresponding to the ROI area;

[0025] Based on the conversion ratio, the regional point cloud data corresponding to the ROI area is preprocessed to obtain the regional point cloud data including the complete box surface of the cash box.

[0026] In some embodiments, after determining the target cash box corresponding to the vault dispatch instruction from the dispatch scenario based on the area recognition result and the color recognition result, the method includes:

[0027] The target cash box is moved to an automatic guided vehicle by a robotic arm to execute the vault dispatch instruction.

[0028] In a second aspect, an embodiment of the present application provides a vault automation system, the system being used to execute the method described in any one of the first aspects above, the system comprising an image acquisition module, a first recognition module, a conversion module, a second recognition module and a determination module;

[0029] The image acquisition module is used to respond to the vault dispatch instruction and acquire a two-dimensional image and a three-dimensional image of the dispatch scene corresponding to the vault dispatch instruction through a camera;

[0030] The first recognition module is used to perform box recognition from the three-dimensional image by a box recognition algorithm according to the three-dimensional image, and obtain a region recognition result of the cash box, wherein the region recognition result includes a three-dimensional mask region of the cash box;

[0031] The conversion module is used to convert the region recognition result to the two-dimensional image through the camera intrinsic parameters and camera extrinsic parameters of the camera to obtain a two-dimensional mask region;

[0032] The second recognition module is used to perform color recognition by a color recognition algorithm according to the two-dimensional mask area and the two-dimensional image to obtain a color recognition result of the two-dimensional mask area;

[0033] The determination module is used to determine the target cash box corresponding to the vault scheduling instruction from the scheduling scene according to the area recognition result and the color recognition result.

[0034] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the computer program.

[0035] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0036] Compared with the related art, the embodiments of the present application provide a safe automation method, system, device and medium, wherein the method responds to the safe scheduling instruction, obtains a two-dimensional image and a three-dimensional image in a scheduling scene corresponding to the safe scheduling instruction through a camera; based on the three-dimensional image, a box recognition algorithm is used to perform box recognition from the three-dimensional image to obtain a region recognition result of the cash box, wherein the region recognition result includes a three-dimensional mask area of ​​the cash box; the region recognition result is converted to the two-dimensional image through the camera intrinsic parameter and the camera extrinsic parameter of the camera to obtain a two-dimensional mask area; based on the two-dimensional mask area and the two-dimensional image, color recognition is performed through a color recognition algorithm to obtain a color recognition result of the two-dimensional mask area; based on the region recognition result and the color recognition result, a target cash box corresponding to the safe scheduling instruction is determined from the scheduling scene, thereby realizing the recognition of the cash box combining 3D and 2D images in the safe automation, and cleverly performing color recognition on the box point cloud area based on the 3D image based on the color characteristics of the 2D image, thereby improving the recognition accuracy of the cash box in the safe automation, and solving the problem of how to improve the automation accuracy of the intelligent safe. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0038] Figure 1 is a flowchart of the steps of the vault automation method according to an embodiment of the present application;

[0039] Figure 2 is a schematic diagram of preprocessing regional point cloud data corresponding to the ROI area according to an embodiment of the present application;

[0040] Figure 3 is a structural block diagram of a vault automation system according to an embodiment of the present application;

[0041] Figure 4 It is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0043] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0044] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0045] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantity limitation, and may indicate the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0046] The present application embodiment provides a vault automation method, Figure 1 is a flowchart of the steps of the vault automation method according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0047] Step S102, in response to the vault dispatch instruction, acquiring a two-dimensional image and a three-dimensional image of a dispatch scene corresponding to the vault dispatch instruction through a camera;

[0048] Step S102 specifically includes the following steps:

[0049] Step S1021, obtaining a two-dimensional image of a dispatching scene corresponding to the vault dispatching instruction through a 2D camera;

[0050] Step S1022, obtaining a three-dimensional image of a scheduling scene corresponding to the vault scheduling instruction through a 3D camera.

[0051] It should be noted that a 2D camera is a device that can capture two-dimensional images of the surface of an object. These two-dimensional images contain information in the X and Y axis directions, that is, the length and width of the object. A 2D camera mainly includes a lens for focusing light onto the sensor; an image sensor (CMOS) for converting light into electrical signals; and an image processing unit for processing electrical signals to generate images.

[0052] A 3D camera is a device that can capture the three-dimensional information of an object, that is, the length and width in the X and Y axis directions, and the depth information in the Z axis direction, that is, a 3D camera can obtain the spatial position and shape of an object. A 3D camera can obtain the three-dimensional information of an object through a variety of technologies, such as: laser triangulation - calculating the distance by projecting a laser beam and measuring the angle change of the reflected light; structured light method - projecting a known pattern of light onto an object, and then analyzing the deformed pattern to calculate the three-dimensional shape of the object; TOF (Time of Flight) method - calculating the distance by measuring the time from the emission to the return of light; binocular stereo vision - using two cameras to simulate the stereoscopic vision of the human eye and calculating the depth through parallax.

[0053] In addition, the three-dimensional image acquired in the embodiment of the present application does not contain color information, while the two-dimensional image contains color information.

[0054] Step S104, based on the three-dimensional image, a box recognition algorithm is used to perform box recognition from the three-dimensional image to obtain a region recognition result of the cash box, wherein the region recognition result includes a three-dimensional mask region of the cash box;

[0055] Specifically, step S104 is based on the three-dimensional image and a box recognition algorithm is used to perform box recognition from the three-dimensional image to obtain a regional recognition result of the cash box. The box recognition algorithm includes a computer vision algorithm, a feature extraction and classification algorithm, and a deep learning algorithm.

[0056] It should be noted that traditional computer vision algorithms, such as: edge detection - use Canny edge detection, Sobel operator, etc. to find the image edges in the three-dimensional image, and then find pairs of parallel line segments to identify rectangles (assuming a money box rectangle); contour detection - use the contour detection function in OpenCV and other libraries to find closed contours in the three-dimensional image, and filter out the contours of the rectangular shape; Hough transform - use Hough line transform to detect straight lines in the three-dimensional image, and then identify rectangles.

[0057] Traditional feature extraction and classification algorithms, such as: SIFT (Scale-Invariant Feature Transform) - uses scale-invariant feature transforms to identify key points in 3D images and uses these key points to locate boxes; SURF (Speeded Up Robust Features) - similar to SIFT, but more efficient, used to quickly locate and describe features; HOG (Histogram of Oriented Gradients) - uses gradient directional histograms to describe the shape of local objects in 3D images, which can be used with support vector machines (SVM) or other classifiers to identify boxes.

[0058] Deep learning algorithms, such as: Convolutional Neural Networks (CNN) - use convolutional layers to automatically learn feature representations, such as using fully convolutional networks (FCN) to semantically segment 3D images to identify boxes; YOLO (You Only Look Once) object detection framework - can simultaneously predict bounding boxes and category probabilities as a 3D image passes through the network; Faster R-CNN object detection - first generate candidate regions (Region Proposals) based on the 3D image, and then classify and regress each candidate region; Mask R-CNN - expands the functionality of Faster R-CNN, not only for object detection, but also for generating segmentation masks for each object.

[0059] It should be further explained that in step S104, in the area recognition based on the three-dimensional image, only shapes that meet the requirements will be recognized from the three-dimensional image according to the set parameters, such as rectangles, squares, diamonds, etc. Preferably, the box shape of the cash box in the embodiment of the present application is a rectangle.

[0060] Step S106, converting the region recognition result to a two-dimensional image through the camera intrinsic parameters and camera extrinsic parameters to obtain a two-dimensional mask region;

[0061] Step S106 specifically includes the following steps:

[0062] Step S1061, converting the region recognition result into a world coordinate system through the camera intrinsic parameters and camera extrinsic parameters of the 3D camera;

[0063] Step S1062 , converting the region recognition result converted to the world coordinate system to the two-dimensional image through the camera intrinsic parameters and camera extrinsic parameters of the 2D camera to obtain a two-dimensional mask region.

[0064] It should be noted that, in the embodiment of the present application, the cash box recognition of the safe based on the three-dimensional image and the two-dimensional image is considered that the resolution of the two-dimensional image and the three-dimensional image is different, and the area recognition result of the three-dimensional image cannot be directly used on the two-dimensional image. Therefore, the three-dimensional mask area of ​​the three-dimensional image is first transferred to the world coordinate system through the internal and external references of the 3D camera. Since the 2D and 3D cameras will be calibrated before shooting to obtain the relative position relationship between the two in the world coordinate system, the three-dimensional mask area of ​​the 3D image converted to the world coordinate system can be transferred to the two-dimensional image through the internal and external references of the 2D camera. In other words, in step S106, the area recognition result based on the three-dimensional image is converted to the two-dimensional image through the camera internal and external references of the 3D camera and the 2D camera in turn to obtain the two-dimensional mask area, which can frame the recognition range (i.e., the two-dimensional mask area) for the color recognition based on the two-dimensional image, eliminate the interference of irrelevant background on the two-dimensional image, and improve the recognition accuracy and efficiency.

[0065] Step S108, performing color recognition by a color recognition algorithm based on the two-dimensional mask area and the two-dimensional image to obtain a color recognition result of the two-dimensional mask area;

[0066] Specifically, step S108 performs color recognition based on the two-dimensional mask area and the two-dimensional image through a color recognition algorithm to obtain a color recognition result of the two-dimensional mask area. The color recognition algorithm includes a computer vision algorithm, a feature extraction and classification algorithm, and a deep learning algorithm.

[0067] It should be noted that the color recognition algorithm used in step S108 is similar to the box recognition algorithm in step S104, both of which include traditional computer vision algorithms (such as color histogram, color space conversion, color clustering, threshold segmentation, etc.), feature extraction and classification algorithms (such as color features, texture features, etc.) and deep learning algorithms (such as convolutional neural networks, transfer learning, target detection framework, etc.), which will not be elaborated here.

[0068] It should be further explained that step S108 frames the recognition range on the two-dimensional image based on the two-dimensional mask area, and then performs color recognition through a color recognition algorithm, thereby eliminating interference from irrelevant background on the two-dimensional image and improving recognition accuracy and efficiency.

[0069] Step S110, based on the area recognition result and the color recognition result, determine the target cash box corresponding to the vault scheduling instruction from the scheduling scene.

[0070] It should be noted that the area recognition result gives the length, width, height and world coordinates of the four corners of the box, and the color recognition result returns the color and coordinates of each box, and finally only classifies them into target colors and non-target colors. Based on the area recognition results and color recognition results, the target cash box corresponding to the vault scheduling instruction is determined from the scheduling scene, which improves the recognition accuracy of cash boxes in vault automation.

[0071] Through steps S102 to S108 in the embodiment of the present application, cash box recognition in safe automation is achieved by combining 3D and 2D images. On the basis of the box point cloud area recognition based on the 3D image, color recognition is cleverly performed on the box point cloud area based on the color features of the 2D image, thereby improving the recognition accuracy of cash boxes in safe automation and solving the problem of how to improve the automation accuracy of intelligent safes.

[0072] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0073] In some of the embodiments, step S104 in the above embodiments optionally extracts regional point cloud data corresponding to the ROI area based on the three-dimensional image and the ROI area; based on the regional point cloud data, box recognition is performed in the ROI area through a box detection algorithm to obtain a regional recognition result of the cash box.

[0074] It should be noted that in the process of regional recognition of cash boxes based on three-dimensional images, the corresponding regional point cloud data is first extracted from the original three-dimensional image based on the ROI area, and then regional recognition is performed based on the regional point cloud data. This can reduce unnecessary computing burden and improve processing efficiency by limiting specific areas in the image, and by excluding background noise and other irrelevant objects, it also helps to improve the accuracy of target detection and recognition.

[0075] Furthermore, the regional point cloud data corresponding to the ROI region extracted based on the three-dimensional image and the ROI region is preferably:

[0076] Step 1: obtain a depth image of the scheduling scene corresponding to the vault scheduling instruction through a camera; based on the depth image, calculate the conversion ratio between the pixel coordinate system and the world coordinate system of the objects in the scheduling scene.

[0077] Step 2: Based on the three-dimensional image and the ROI area, extract the regional point cloud data corresponding to the ROI area; based on the conversion ratio, pre-process the regional point cloud data corresponding to the ROI area to obtain the regional point cloud data of the complete box surface including the cash box.

[0078] It should be noted that if the cash box is not facing the camera, that is, the surface of the box is tilted at a certain angle to the camera, then its highest plane in the image cannot fully display the entire surface of the box. Figure 2 is a schematic diagram of preprocessing the regional point cloud data corresponding to the ROI area according to an embodiment of the present application, such as Figure 2 As shown, the conversion ratio obtained in the preferred steps 1 and 2 to pre-process the regional point cloud data can be to find the highest layer of the regional point cloud data and retain the regional point cloud data within the upper and lower parts, the highest layer refers to the highest area in the world coordinate system after the regional point cloud data is converted by the conversion ratio, and the area within the upper and lower parts refers to the area above and below the highest area. At the same time, although the three-dimensional image obtained in step S102 also has depth information, the depth information in the depth map is usually easier to process than that in the three-dimensional image, because the depth map provides depth information in a two-dimensional format without complex 3D data processing, so obtaining the depth map in step 1 can improve the subsequent processing efficiency.

[0079] The present application embodiment provides a vault automation system, Figure 3 is a structural block diagram of a vault automation system according to an embodiment of the present application, such as Figure 3 As shown, the system includes an image acquisition module, a first recognition module, a conversion module, a second recognition module and a determination module;

[0080] An image acquisition module is used to respond to the vault dispatch instruction and acquire a two-dimensional image and a three-dimensional image of the dispatch scene corresponding to the vault dispatch instruction through a camera;

[0081] A first recognition module is used to perform box recognition from the three-dimensional image by a box recognition algorithm according to the three-dimensional image, and obtain a region recognition result of the cash box, wherein the region recognition result includes a three-dimensional mask region of the cash box;

[0082] A conversion module, used to convert the region recognition result to a two-dimensional image through the camera intrinsic parameters and camera extrinsic parameters of the camera to obtain a two-dimensional mask region;

[0083] A second recognition module is used to perform color recognition by a color recognition algorithm according to the two-dimensional mask area and the two-dimensional image to obtain a color recognition result of the two-dimensional mask area;

[0084] The determination module is used to determine the target cash box corresponding to the vault scheduling instruction from the scheduling scene according to the area recognition result and the color recognition result.

[0085] Through the image acquisition module, the first recognition module, the conversion module, the second recognition module and the determination module in the embodiments of the present application, the cash box recognition combining 3D and 2D images in the safe automation is realized. On the basis of the box point cloud area recognition based on the 3D image, the color of the box point cloud area is cleverly recognized based on the color features of the 2D image, thereby improving the recognition accuracy of the cash box in the safe automation and solving the problem of how to improve the automation accuracy of the intelligent safe.

[0086] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0087] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0088] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0089] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0090] In addition, in combination with the safe automation method in the above embodiment, the embodiment of the present application can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, any safe automation method in the above embodiment is implemented.

[0091] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a vault automation method is implemented. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball, or touchpad provided on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0092] In one embodiment, Figure 4 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Figure 4 As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in Figure 4 As shown. The electronic device includes a processor, a network interface, an internal memory and a non-volatile memory connected through an internal bus, wherein the non-volatile memory stores an operating system, a computer program and a database. The processor is used to provide computing and control capabilities, the network interface is used to communicate with an external terminal through a network connection, the internal memory is used to provide an environment for the operation of the operating system and the computer program, the computer program is executed by the processor to implement a vault automation method, and the database is used to store data.

[0093] Those skilled in the art will understand that Figure 4 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0094] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0095] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0096] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A vault automation method, characterized in that: The method comprises: In response to the vault dispatch instruction, a 2D camera is used to obtain a 2D image of a dispatch scene corresponding to the vault dispatch instruction; and a 3D camera is used to obtain a 3D image of the dispatch scene corresponding to the vault dispatch instruction; Based on the three-dimensional image and the ROI area, extracting regional point cloud data corresponding to the ROI area; based on the regional point cloud data, performing box recognition in the ROI area by a box recognition algorithm to obtain a regional recognition result of the cash box, wherein the regional recognition result includes a three-dimensional mask area of ​​the cash box; The region recognition result is converted into a world coordinate system by using the camera intrinsic parameters and camera extrinsic parameters of the 3D camera; the region recognition result converted into the world coordinate system is converted into the two-dimensional image by using the camera intrinsic parameters and camera extrinsic parameters of the 2D camera to obtain a two-dimensional mask region; Based on the two-dimensional mask area and the two-dimensional image, color recognition is performed by a color recognition algorithm to obtain a color recognition result of the two-dimensional mask area; Based on the area recognition result and the color recognition result, a target cash box corresponding to the vault scheduling instruction is determined from the scheduling scenario.

2. The method according to claim 1, characterized in that The method comprises: Acquire a depth image of a dispatching scene corresponding to the vault dispatching instruction through a camera; Based on the depth image, a conversion ratio between a pixel coordinate system and a world coordinate system of an object in the scheduling scene is calculated.

3. The method according to claim 2, characterized in that Based on the three-dimensional image and the ROI area, extracting regional point cloud data corresponding to the ROI area includes: Based on the three-dimensional image and the ROI area, extracting regional point cloud data corresponding to the ROI area; Based on the conversion ratio, the regional point cloud data corresponding to the ROI area is preprocessed to obtain the regional point cloud data including the complete box surface of the cash box.

4. The method according to claim 1, characterized in that: After determining the target cash box corresponding to the vault dispatch instruction from the dispatch scenario based on the area recognition result and the color recognition result, the method includes: The target cash box is moved to an automatic guided vehicle by a robotic arm to execute the vault dispatch instruction.

5. A vault automation system, characterized in that: The system is used to perform the method according to any one of claims 1 to 4, and the system comprises an image acquisition module, a first recognition module, a conversion module, a second recognition module and a determination module; The image acquisition module is used to respond to the vault dispatch instruction and acquire a two-dimensional image of the dispatch scene corresponding to the vault dispatch instruction through a 2D camera; and acquire a three-dimensional image of the dispatch scene corresponding to the vault dispatch instruction through a 3D camera; The first recognition module is used to extract regional point cloud data corresponding to the ROI area according to the three-dimensional image and the ROI area; Based on the regional point cloud data, box recognition is performed in the ROI region by a box recognition algorithm to obtain a regional recognition result of the cash box, wherein the regional recognition result includes a three-dimensional mask area of ​​the cash box; The conversion module is used to convert the region recognition result into a world coordinate system through the camera intrinsic parameters and camera extrinsic parameters of the 3D camera; and convert the region recognition result converted into the world coordinate system into the two-dimensional image through the camera intrinsic parameters and camera extrinsic parameters of the 2D camera to obtain a two-dimensional mask region; The second recognition module is used to perform color recognition by a color recognition algorithm according to the two-dimensional mask area and the two-dimensional image to obtain a color recognition result of the two-dimensional mask area; The determination module is used to determine the target cash box corresponding to the vault scheduling instruction from the scheduling scene according to the area recognition result and the color recognition result.

6. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images