Warehouse management method, device, computer equipment, storage medium

Abstract

This application relates to a warehouse management method, device, computer device, storage medium, and computer program product. The method includes: obtaining a face image of a person entering or leaving the warehouse, and processing the face image through a face recognition model to obtain a face recognition result; obtaining the status of an emergency fire door, where the status of the emergency fire door is divided into an open state and a closed state; obtaining monitoring images of a plurality of perspectives of an object to be managed in the warehouse, and processing the monitoring images of the plurality of perspectives of the object to be managed through an object trajectory tracking model to obtain the movement trajectory of the object to be managed; obtaining the characteristic information of the warehouse, integrating the characteristic information of the warehouse, the face recognition result, the status of the emergency fire door, and the movement trajectory of the object to be managed to obtain warehouse integration data, and generating a prompt message according to the warehouse integration data. Using this method can improve the utilization efficiency of warehouse monitoring video data and improve the safety factor of warehouse management.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a warehouse management method, device, computer device, and storage medium. Background Art

[0002] Machine vision refers to using a computer to simulate the visual function of a human, extracting information from the images of objective things, processing and mainly understanding it, and finally achieving the purpose of actual detection, measurement, and control. The safety of goods and personnel is one of the goals of warehouse safety management, and real-time and intelligent monitoring is an important technical means to effectively ensure the safety of warehouse goods. The traditional monitoring method is to use sensing technologies such as RFID to monitor the inbound and outbound situations of goods, and at the same time use cameras to monitor the warehouse, record the video under the operating state of the warehouse, and conduct monitoring and archiving.

[0003] Due to the limited transmission distance of RFID signals, a large number of sensors need to be arranged for large-scale monitoring and management. On the one hand, this leads to an increase in costs. On the other hand, it complicates the storage warehouse safety management system and improves the maintenance difficulty. At the same time, the method of using sensing technologies such as RFID cannot monitor human behaviors and the movement trajectories of people and goods, making the objects of monitoring and management incomplete and the monitoring scope not wide enough. For the videos captured by cameras, manual viewing and monitoring are required, which is time-consuming and laborious. Due to the influence of some factors, such as fatigue or personnel configuration reasons, etc., it is often necessary for monitoring personnel to view multiple fields of view. Sometimes, the monitoring personnel will miss important safety clues, which may bring serious consequences. If abnormal behaviors cannot be detected in a timely manner, the video data will be saved as evidence, making the value of the data unable to be effectively utilized. Because from the perspective of the timeliness of data value, the value of newly generated data will be higher, and as time goes by, the value of the data shows a decaying trend.

[0004] Traditional warehouse management technologies have the disadvantages of difficult maintenance of sensing technologies, incomplete monitoring objects, narrow monitoring scope, and low utilization efficiency of monitoring video data values, resulting in a low safety factor for warehouse management. Summary of the Invention

[0005] Based on this, in view of the above technical problems, it is necessary to provide a warehouse management method, device, computer device, computer-readable storage medium, and computer program product that can improve the safety factor of warehouse management.

[0006] In a first aspect, the present application provides a warehouse management method. The method includes:

[0007] Obtain the face images of the personnel entering and leaving the warehouse, and process the face images through a face recognition model to obtain a face recognition result;

[0008] Obtain the status of the emergency fire door, and the status of the emergency fire door is divided into the open state and the closed state;

[0009] Obtain the monitoring images of the objects to be managed in the warehouse from multiple perspectives, and process the monitoring images of the objects to be managed from multiple perspectives through the object trajectory tracking model to obtain the movement trajectories of the objects to be managed;

[0010] Obtain the characteristic information of the warehouse, integrate the characteristic information of the warehouse, the face recognition result, the status of the emergency fire door and the movement trajectory of the objects to be managed to obtain the integrated data of the warehouse, and generate a prompt message according to the integrated data of the warehouse.

[0011] In one embodiment, process the face image through the face recognition model to obtain the face recognition result, including:

[0012] Extract the face features of the face image through the face recognition model, and match and compare the face features of the face image with the preset authorized face features in the authorized face feature library;

[0013] If the face features of the face image can match the preset authorized face features in the authorized face feature library, obtain the face recognition result that the person is authorized;

[0014] If the face features of the face image cannot match the preset authorized face features in the authorized face feature library, obtain the face recognition result that the person is not authorized.

[0015] In one embodiment, obtain the monitoring images of the objects to be managed in the warehouse from multiple perspectives, and process the monitoring images of the objects to be managed from multiple perspectives through the object trajectory tracking model to obtain the movement trajectories of the objects to be managed, including:

[0016] Obtain the objects to be managed through the object detection model, and obtain the monitoring images of the objects to be managed in the warehouse from multiple perspectives;

[0017] Process the monitoring image of each perspective of the object to be managed through the object trajectory tracking model to obtain the trajectory position of the object to be managed in the world coordinate system under each perspective;

[0018] Calculate the similarity of the trajectory positions of the objects to be managed in the world coordinate system under each perspective to obtain the movement trajectories of the objects to be managed.

[0019] In one embodiment, obtain the characteristic information of the warehouse, integrate the characteristic information of the warehouse, the face recognition result, the status of the emergency fire door and the movement trajectory of the objects to be managed to obtain the integrated data of the warehouse, including:

[0020] Obtain the outbound quantity and inbound quantity of the objects to be managed according to the movement trajectories of the objects to be managed;

[0021] Determine the quantity of the object to be managed in the warehouse according to the outbound quantity and inbound quantity of the object to be managed;

[0022] Obtain the characteristic information of the warehouse, and integrate the characteristic information of the warehouse, the face recognition result, the status of the emergency fire door, the movement trajectory of the object to be managed, and the quantity of the object to be managed in the warehouse to obtain the integrated warehouse data.

[0023] In one embodiment, determining the outbound quantity and inbound quantity of the object to be managed according to the movement trajectory of the object to be managed includes:

[0024] Obtain the stacking layer number and the top layer quantity of the object to be managed, and obtain the stacking quantity of the object to be managed according to the stacking layer number and the top layer quantity of the object to be managed;

[0025] Determine the outbound quantity and inbound quantity of the object to be managed according to the movement trajectory of the object to be managed and the stacking quantity.

[0026] In one embodiment, the method further includes:

[0027] Obtain the personnel image in the warehouse according to the preset image size;

[0028] Detect the personnel image through the feature detection model. If it is detected that the personnel in the personnel image carry goods, generate an abnormal behavior warning message.

[0029] In a second aspect, the present application further provides a warehouse management device. The device includes:

[0030] A personnel detection module, configured to obtain the face image of the personnel entering and leaving the warehouse, and process the face image through a face recognition model to obtain a face recognition result;

[0031] A fire door detection module, configured to obtain the status of the emergency fire door, and the status of the emergency fire door is divided into an open state and a closed state;

[0032] A trajectory detection module, configured to obtain the monitoring images of multiple perspectives of the object to be managed in the warehouse, and process the monitoring images of multiple perspectives of the object to be managed through an object trajectory tracking model to obtain the movement trajectory of the object to be managed;

[0033] An information integration module, configured to obtain the characteristic information of the warehouse, integrate the characteristic information of the warehouse, the face recognition result, the status of the emergency fire door, and the movement trajectory of the object to be managed to obtain the integrated warehouse data, and generate a prompt message according to the integrated warehouse data.

[0034] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0035] Obtain the face images of the personnel entering and leaving the warehouse, and process the face images through a face recognition model to obtain the face recognition results;

[0036] Obtain the status of the emergency fire door, and the status of the emergency fire door is divided into an open state and a closed state;

[0037] Obtain the monitoring images of the managed objects in the warehouse from multiple perspectives, and process the monitoring images of the managed objects from multiple perspectives through an object trajectory tracking model to obtain the movement trajectories of the managed objects;

[0038] Obtain the characteristic information of the warehouse, integrate the characteristic information of the warehouse, the face recognition results, the status of the emergency fire door, and the movement trajectories of the managed objects to obtain the integrated data of the warehouse, and generate a prompt message according to the integrated data of the warehouse.

[0039] In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium has a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0040] Obtain the face images of the personnel entering and leaving the warehouse, and process the face images through a face recognition model to obtain the face recognition results;

[0041] Obtain the status of the emergency fire door, and the status of the emergency fire door is divided into an open state and a closed state;

[0042] Obtain the monitoring images of the managed objects in the warehouse from multiple perspectives, and process the monitoring images of the managed objects from multiple perspectives through an object trajectory tracking model to obtain the movement trajectories of the managed objects;

[0043] Obtain the characteristic information of the warehouse, integrate the characteristic information of the warehouse, the face recognition results, the status of the emergency fire door, and the movement trajectories of the managed objects to obtain the integrated data of the warehouse, and generate a prompt message according to the integrated data of the warehouse.

[0044] In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0045] Obtain the face images of the personnel entering and leaving the warehouse, and process the face images through a face recognition model to obtain the face recognition results;

[0046] Obtain the status of the emergency fire door, and the status of the emergency fire door is divided into an open state and a closed state;

[0047] Obtain the monitoring images of the managed objects in the warehouse from multiple perspectives, and process the monitoring images of the managed objects from multiple perspectives through an object trajectory tracking model to obtain the movement trajectories of the managed objects;

[0048] Obtain the characteristic information of the warehouse, integrate the characteristic information of the warehouse, the face recognition result, the status of the emergency fire door, and the movement trajectory of the objects to be managed to obtain the integrated warehouse data, and generate a prompt message according to the integrated warehouse data.

[0049] The above warehouse management method, device, computer equipment, storage medium, and computer program product obtain the face images of the personnel entering and leaving the warehouse, and process the face images through a face recognition model to obtain the face recognition result; obtain the status of the emergency fire door, and the status of the emergency fire door is divided into an open state and a closed state; obtain the surveillance images of multiple perspectives of the objects to be managed in the warehouse, and process the surveillance images of multiple perspectives of the objects to be managed through an object trajectory tracking model to obtain the movement trajectory of the objects to be managed; obtain the characteristic information of the warehouse, integrate the characteristic information of the warehouse, the face recognition result, the status of the emergency fire door, and the movement trajectory of the objects to be managed to obtain the integrated warehouse data, and generate a prompt message according to the integrated warehouse data. It can improve the utilization efficiency of the warehouse surveillance video data and improve the safety factor of warehouse management. Description of the Drawings

[0050] Figure 1 It is a schematic flowchart of the warehouse management method in an embodiment;

[0051] Figure 2 It is a schematic flowchart of the object detection step in an embodiment;

[0052] Figure 3 It is an algorithm flowchart of the object trajectory tracking model in an embodiment;

[0053] Figure 4 It is a schematic diagram of the effect of the object trajectory tracking model in an embodiment;

[0054] Figure 5 It is a schematic flowchart of the warehouse management method in an embodiment;

[0055] Figure 6 It is a structural block diagram of the warehouse management device in another embodiment;

[0056] Figure 7 It is an internal structural diagram of a computer device in an embodiment. Detailed Embodiments

[0057] In order to make the purpose, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] In one embodiment, as Figure 1As shown, a warehouse management method is provided. In this embodiment, taking the application of this method to a terminal as an example, it can be understood that this method can also be applied to a server, or to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0059] Step 102: Obtain the face images of the personnel entering and leaving the warehouse, and process the face images through a face recognition model to obtain a face recognition result.

[0060] Specifically, a camera or other image acquisition device is pre-installed at the gate position of the warehouse, so that the camera and other image acquisition devices can capture the faces of the personnel entering and leaving the warehouse gate. The face images of the personnel entering and leaving the warehouse are obtained through the camera, and face recognition is performed on the face images using a face recognition model. For example, a common industrial face recognition algorithm network model (insightFace) is used to process the face images to obtain a face recognition result.

[0061] Step 104: Obtain the status of the emergency fire door. The status of the emergency fire door is divided into an open state and a closed state.

[0062] Specifically, a camera or other image acquisition device is pre-installed at the position of the emergency fire door of the warehouse, so that the camera and other image acquisition devices can capture the emergency fire door. The real-time image of the emergency fire door is obtained through the camera, and the status detection is performed on the real-time image of the emergency fire door using a target detection algorithm. For example, an end-to-end target detection algorithm (yolo5, the network structure is shown in "yolo5-network structure.png") is used to detect the status of the emergency fire door to obtain the status of the emergency fire door.

[0063] Step 106: Obtain the monitoring images of multiple perspectives of the objects to be managed in the warehouse, and process the monitoring images of multiple perspectives of the objects to be managed through an object trajectory tracking model to obtain the movement trajectories of the objects to be managed.

[0064] Specifically, multiple cameras or other image acquisition devices are pre-installed at multiple preset positions in the warehouse. The monitoring images of multiple perspectives are obtained through the multiple cameras, and target detection is performed on all the monitoring images using a target detection algorithm. For example, an end-to-end target detection algorithm (yolo5) is used to detect and identify the monitoring images, and one or more objects to be managed in each monitoring image are identified. The objects to be managed can be objects such as forklifts, pallets, goods, and personnel.

[0065] Further, use the object trajectory tracking model to process each surveillance image. For example, use the multi-view multi-target tracking algorithm (MVMT tracker) to obtain the motion trajectory of the managed object in the world coordinate system based on the position of the managed object in each surveillance image. Similarly, obtain the motion trajectories of each managed object in the world coordinate system.

[0066] Step 108: Obtain the characteristic information of the warehouse, integrate the characteristic information of the warehouse, the face recognition result, the status of the emergency fire door, and the motion trajectories of the managed objects to obtain the integrated warehouse data, and generate a prompt message according to the integrated warehouse data.

[0067] Specifically, obtain the characteristic information of the warehouse according to information such as the warehouse ID, the IP address of the cameras in the warehouse, and the image capture time in the warehouse. Integrate the characteristic information of a warehouse, the face recognition result, the status of the emergency fire door, and the motion trajectories of the managed objects to obtain the integrated warehouse data of the warehouse, and generate a prompt message according to the integrated warehouse data, so as to monitor and manage the managed objects such as personnel, goods, and forklifts in the warehouse.

[0068] In the above warehouse management method, obtain the face images of the personnel entering and leaving the warehouse, and process the face images through the face recognition model to obtain the face recognition result; obtain the status of the emergency fire door, and the status of the emergency fire door is divided into the open state and the closed state; obtain the surveillance images of multiple views of the managed objects in the warehouse, and process the surveillance images of multiple views of the managed objects through the object trajectory tracking model to obtain the motion trajectories of the managed objects; obtain the characteristic information of the warehouse, integrate the characteristic information of the warehouse, the face recognition result, the status of the emergency fire door, and the motion trajectories of the managed objects to obtain the integrated warehouse data, and generate a prompt message according to the integrated warehouse data. It can improve the utilization efficiency of the warehouse surveillance video data and improve the safety factor of warehouse management.

[0069] In one embodiment, processing the face image through the face recognition model to obtain the face recognition result includes: extracting the face features of the face image through the face recognition model, and matching and comparing the face features of the face image with the preset authorized face features in the authorized face feature library; if the face features of the face image can match the preset authorized face features in the authorized face feature library, then obtain the face recognition result that the personnel are authorized; if the face features of the face image cannot match the preset authorized face features in the authorized face feature library, then obtain the face recognition result that the personnel are not authorized.

[0070] Specifically, the face recognition model is used in the inbound / outbound stage to detect whether the operating personnel are warehouse managers and ensure that no other personnel enter the warehouse. The front camera on the inbound door is used to obtain the image information in front of the warehouse door, and the AI algorithm is used to identify whether the current operating personnel meet the permission to enter the warehouse. When the warehouse manager who meets the inbound permission enters the warehouse, there is no alarm. When the warehouse manager who does not meet the inbound permission enters the warehouse and when non-warehouse management personnel enter the warehouse, an alarm is triggered. Before use, the warehouse first provides the face photos of the staff who can enter the warehouse. The common industrial face recognition algorithm network model (insightFace) is used to extract the face feature vectors and construct the warehouse white list (i.e., the face feature library), without storing the face photos. During use, Opencv is used to pull the RTSP video stream of the camera. For each frame in the video stream, dlib is used to detect the face position in the image, and then crop and correct the alignment so that the face position fills the cropped area. The feature extraction model is used to extract the face features, and then compare them with the face feature library to output the face recognition result. When performing face comparison, the 1:N mode is adopted, that is, the face in the image captured by the camera is detected, and the corresponding face information feature vector is cropped and then compared with the N data in the face feature library to determine whether the person has the permission to enter the warehouse. If multiple faces are detected in one image, it is necessary to compare with the feature library multiple times. After obtaining the result, the face recognition model can store or send the picture of the current frame, the acquisition time, the position of each face in the picture, and the comparison result of each face in the form of a dictionary to the specified end.

[0071] In this embodiment, the face features of the face image are extracted through the face recognition model, and the face features of the face image are matched and compared with the preset authorized face features in the authorized face feature library; if the face features of the face image can match the preset authorized face features in the authorized face feature library, the face recognition result of the authorized personnel is obtained; if the face features of the face image cannot match the preset authorized face features in the authorized face feature library, the face recognition result of the unauthorized personnel is obtained. It can accurately judge whether the current inbound / outbound personnel are authorized personnel.

[0072] In one embodiment, multiple perspectives of surveillance images of the managed objects in the warehouse are obtained, and the motion trajectories of the managed objects are obtained by processing the multiple perspectives of surveillance images of the managed objects through the object trajectory tracking model, including: obtaining the managed objects through the object detection model and obtaining multiple perspectives of surveillance images of the managed objects in the warehouse; processing each perspective of the surveillance images of the managed objects through the object trajectory tracking model to obtain the trajectory positions of the managed objects in the world coordinate system for each perspective; calculating the similarity of the trajectory positions of the managed objects in the world coordinate system for each perspective to obtain the motion trajectories of the managed objects.

[0073] Specifically, as Figure 2 shown, the object detection model is used to identify managed objects such as forklifts, pallets, goods, and personnel during the inbound and outbound stages, and to determine whether a forklift enters or exits the warehouse, whether the forklift carries a pallet when entering or exiting the warehouse, and whether there is goods on the pallet when the forklift enters or exits the warehouse, providing a basis for the determination of abnormal behaviors. The end-to-end object detection model (yolo5) is used for detection and identification. The outputs of the model are of four categories: forklift, stacking, person, and tray (representing forklift, stacking, person, and pallet respectively). The output categories can be increased or modified according to the different warehouses and the goods stored in them. The model uses time as the only identifier, with a single-frame image as the input and structured data as the output. The fields in the output structured data include whether there is a forklift, whether there is a pallet, and whether there is goods. By integrating the results of the fields, it can be determined whether a forklift enters or exits the warehouse, whether the forklift carries a pallet when entering or exiting the warehouse, and whether there is goods on the pallet when the forklift enters or exits the warehouse, and it is stored or sent to the specified end in the form of a dictionary.

[0074] Furthermore, the object trajectory tracking model is based on the object detection model. In the warehouse, cameras in front of the door, opposite the door, on top of the door, and on the side of the door will monitor the area at the warehouse door in real time. The video frame of each perspective is used as the input of the object detection model. If an object (forklift, stacking, person, and pallet) appears in the current frame (i.e., the target category), the object detection model will output the coordinates of the corresponding object in the frame (i.e., the target position). As Figure 3 shown, the dashed part in the figure is the processing stage of the object detection model, which mainly includes feature extraction and two functional branches of the model: bounding box regression and class classification. Among them, bounding box regression is mainly the position of the target in the image, which is composed of a quadruple, and class classification is mainly the determination of the target object category, and the target five-tuple information under each perspective is obtained through post-processing. After obtaining the target category and target position under each perspective, the multi-view multi-target tracking model (MVMT tracker) is used to extract the features of the target categories (forklift, stacking, person, and pallet) detected in each perspective, perform cross-view target association, and finally update the feature table, assign IDs, and update the bounding boxes. According to the results of the multi-view multi-target tracking model (MVMT tracker), the trajectory positions of the targets (forklift, stacking, person, and pallet) under each perspective in the world coordinate system are obtained, as Figure 4As shown, it can be the visualization results of the trajectories of the corresponding targets (personnel) from three perspectives. Next, reassign the IDs of the targets (forklift, stacker, person, and pallet). By calculating the similarity of the trajectories from each perspective, the similarity calculation mainly uses the Euclidean distance and the cosine distance. The trajectories are fused based on the numerical differences and directional differences, and then the target IDs are reassigned. After the target ID reassignment, the motion trajectories of the targets (forklift, stacker, person, and pallet) in the 3D space can be visualized, and the motion trajectories of the targets (forklift, stacker, person, and pallet) can be followed to determine the entry and exit directions, that is, whether it is inbound or outbound.

[0075] In this embodiment, the objects to be managed are obtained through an object detection model, and monitoring images of multiple perspectives of the objects to be managed in the warehouse are obtained; the monitoring images of each perspective of the objects to be managed are processed through an object trajectory tracking model to obtain the trajectory positions of the objects to be managed in the world coordinate system for each perspective; the similarity of the trajectory positions of the objects to be managed in the world coordinate system for each perspective is calculated to obtain the motion trajectories of the objects to be managed. The motion trajectories of each object to be managed can be monitored and recorded.

[0076] In one embodiment, the characteristic information of the warehouse is obtained, and the characteristic information of the warehouse, the face recognition result, the status of the emergency fire door, and the motion trajectories of the objects to be managed are integrated to obtain the integrated warehouse data, including: obtaining the outbound quantity and inbound quantity of the objects to be managed according to the motion trajectories of the objects to be managed; determining the quantity of the objects to be managed in the warehouse according to the outbound quantity and inbound quantity of the objects to be managed; obtaining the characteristic information of the warehouse, and integrating the characteristic information of the warehouse, the face recognition result, the status of the emergency fire door, the motion trajectories of the objects to be managed, and the quantity of the objects to be managed in the warehouse to obtain the integrated warehouse data.

[0077] Specifically, obtain the outbound quantity and inbound quantity of personnel and the outbound quantity and inbound quantity of goods according to the motion trajectories of personnel and goods, and determine the characteristic information of the warehouse according to information such as the warehouse ID, the IP of the cameras in the warehouse, and the image shooting time in the warehouse, so as to give information such as the entry and exit of personnel / forklifts / stackers / pallets from the warehouse and their motion trajectories when entering and exiting the warehouse, determine whether a person has permission when entering and exiting the warehouse, whether a person carries goods when entering and exiting, how many people are in the warehouse and their in-warehouse time, how many goods each stacker has when entering and exiting the warehouse, the status of the emergency fire door, etc. When a set rule is triggered, an alarm is issued. For events that require an alarm, the relevant information and images can be saved and uploaded to the cloud server for evidence storage, and finally reviewed and determined manually, greatly reducing the workload of review.

[0078] Further, when the object to be managed is a person, based on the object detection model and the object trajectory tracking model, by judging the situation of the staff entering and leaving the warehouse, calculate the number of staff currently staying in the warehouse and the total number of people entering and leaving the warehouse on the same day. At the same time, combined with the face recognition model, the staying time of the staff entering the warehouse in the warehouse can be calculated. In a specific warehouse scenario (such as a cold chain warehouse), when the staying time of the staff in the warehouse exceeds a certain time limit, an alarm can be issued to ensure the safety of the staff.

[0079] In this embodiment, the outbound quantity and inbound quantity of the object to be managed are obtained according to the movement trajectory of the object to be managed; the quantity of the object to be managed inside the warehouse is determined according to the outbound quantity and inbound quantity of the object to be managed; the characteristic information of the warehouse is obtained, and the characteristic information of the warehouse, the face recognition result, the status of the emergency fire door, the movement trajectory of the object to be managed, and the quantity of the object to be managed inside the warehouse are integrated to obtain the integrated warehouse data. It can improve the utilization efficiency of the warehouse monitoring video data and the safety factor of warehouse management.

[0080] In one embodiment, determining the outbound quantity and inbound quantity of the object to be managed according to the movement trajectory of the object to be managed includes: obtaining the stacking layer number and the top layer quantity of the object to be managed, and obtaining the stacking quantity of the object to be managed according to the stacking layer number and the top layer quantity of the object to be managed; determining the outbound quantity and inbound quantity of the object to be managed according to the movement trajectory and the stacking quantity of the object to be managed.

[0081] Specifically, when the object to be managed is a cargo box, the estimation of the stacking quantity of the cargo box is mainly to count the number of cargo boxes on the stack during the inbound / outbound stage. To realize the estimation of the quantity of stacked cargo boxes, cameras at the front and top of the warehouse door are used to monitor and shoot the two directions (front and top) of the entire stack when the forklift passes by respectively. According to the monitored video stream, the visual algorithm is used to estimate the number of cargo boxes on the entire stack. The calculation formula is N = n * f. In the formula, N represents the number of cargo boxes on the stack, n represents the number of cargo boxes on the top surface, and f represents the number of stacking layers on the front. To estimate the number of cargo boxes on the top surface of the stack and the number of cargo boxes on the front of the stack, the object detection model (yolo5) is used to locate and identify each cargo box on the top surface and the front of the stack, and the number of cargo boxes on the top surface and the number of cargo boxes on the front are obtained through category counting. In the actual use process, first, video frames are extracted from the video stream, and then the object detection model is used to identify the stack, and the stack is cropped from the video frame to ensure that the area to be detected occupies the main part of the image. Then, the preprocessed image is input into the cargo box object detection model to obtain the target category and its position information. Finally, the outbound quantity and inbound quantity of the cargo boxes are obtained through category counting.

[0082] In this embodiment, by obtaining the stacking layers and the top layer quantity of the object to be managed, and obtaining the stacking quantity of the object to be managed according to the stacking layers and the top layer quantity of the object to be managed; determining the outbound quantity and inbound quantity of the object to be managed according to the movement trajectory and the stacking quantity of the object to be managed. It is possible to perform real-time statistics on the inbound and outbound quantities of the managed images.

[0083] In one embodiment, the method further includes: obtaining a personnel image in the warehouse according to a preset image size; detecting the personnel image through a feature detection model, and if it is detected that the personnel in the personnel image carry goods, generating an abnormal behavior warning message.

[0084] Specifically, in the warehouse, the camera will monitor the warehouse in real time, and its video is used as the input of the object detection model. If there are people in the current picture, it will output the coordinates of the corresponding personnel in the picture, crop the personnel area accordingly, and save the current complete picture, marking the position where the personnel are located; then send the cropped personnel image into the feature detection model, and the feature detection model will identify whether the current personnel carry goods. If they do, it will give a warning, otherwise it will not give a warning. The feature detection model is a binary classification model, using resnet101 as the backbone network. For example, when the object detection model detects that there are people in the monitoring video frame, the picture of the person is cropped from the video frame, and the cropped picture is resized to 128*128 size and then input into the feature detection model. The feature detection model outputs whether the person holds goods, and an alarm is given when the person enters or exits the warehouse door with goods in their arms.

[0085] In one embodiment, as Figure 5 shown, taking a warehouse management method applied to a warehouse management system composed of a face recognition algorithm module, an object detection algorithm module, an emergency fire door detection algorithm module, an object trajectory tracking algorithm module, an abnormal behavior detection algorithm module, a personnel counting algorithm module, a stacking box counting algorithm module, and an information integration and rule judgment algorithm module as an example, the method specifically includes:

[0086] Obtain the video of normal warehouse operations through four cameras at the warehouse door and one camera at the emergency fire door. The face recognition algorithm module, the object detection algorithm module, the emergency fire door detection algorithm module, the object trajectory tracking algorithm module, the abnormal behavior detection algorithm module, the personnel counting algorithm module, and the stacking box counting algorithm module process each frame of the video, and the information integration and rule judgment algorithm module integrates the results to achieve the purpose of warehouse safety management. Each algorithm module is separately deployed using docker containers, and the algorithm modules transmit information through json files to reduce the coupling between the algorithm modules and ensure the stable operation of the entire warehouse management system.

[0087] The face recognition algorithm module is used in the inbound / outbound stage to detect whether the operating personnel are warehouse managers and ensure that no other personnel enter the warehouse. The front camera on the inbound door is used to obtain the image information in front of the warehouse door, and an AI algorithm is used to identify whether the current operating personnel have the permission to enter the warehouse. When a warehouse manager who meets the inbound permission enters the warehouse, there is no alarm. When a warehouse manager who does not meet the inbound permission enters the warehouse or a non-warehouse manager enters the warehouse, an alarm is triggered. Before use, the warehouse first provides the face photos of the staff who can enter the warehouse. The common industrial face recognition algorithm network model (insightFace) is used to extract the face feature vectors and construct the warehouse white list (i.e., the face feature library), without storing the face photos. During use, Opencv is used to pull the RTSP video stream of the camera. For each frame in the video stream, dlib is used to detect the face position in the image, and then it is cropped, corrected and aligned so that the face position fills the cropped area. The feature extraction model is used to extract the face features, and then compare them with the face feature library to output the face recognition result. When performing face comparison, the 1:N mode is adopted, that is, the image captured by the camera is subjected to face detection, and the feature vector of the corresponding face information is cropped and then compared with the N data in the face feature library to determine whether the person has the permission to enter the warehouse. If multiple faces are detected in an image, it needs to be compared with the feature library multiple times. After obtaining the result, the face recognition algorithm module sends the picture of the current frame, the acquisition time, the position of each face in the picture, and the comparison result of each face to the information integration and rule judgment algorithm module in the form of a dictionary.

[0088] The emergency fire door detection algorithm module is mainly used to identify the open or closed state of the emergency fire door. According to the warehouse management rules, the emergency door should usually be in the closed state to ensure that the goods in the warehouse are stored normally. If the emergency door is opened, an alarm needs to be triggered. In the warehouse, the image information of the emergency door is obtained in real time through the camera, and an algorithm is used to judge the open or closed state of the emergency door. When it is found that the emergency door is open, an alarm notification is sent. The end-to-end object detection algorithm (yolo5) is used to detect the state of the emergency door. The output results of the model have two categories, namely open door and closed door. During use, Opencv is used to pull the RTSP video stream of the camera, and each frame in the video stream is input into the network. The network outputs the position information of the emergency door and classifies the state of the emergency door at the same time. After obtaining the result, the emergency fire door detection algorithm module sends the picture of the current frame, the acquisition time, the position of the emergency fire door in the picture, and the open / closed state of the door to the information integration and rule judgment algorithm module in the form of a dictionary.

[0089] The object detection algorithm module is used to identify objects such as forklifts, pallets, goods, and people during the inbound and outbound stages, and to determine whether a forklift enters or exits the warehouse, whether the forklift carries a pallet when entering or exiting the warehouse, and whether there are goods on the pallet when the forklift enters or exits the warehouse, providing a basis for the determination of abnormal behaviors. The end-to-end object detection algorithm (yolo5) is used for detection and recognition. The output of this model has four categories: forklift, stacking, person, and tray (representing forklift, stacking, person, and pallet respectively). The output categories can be increased or modified according to the different warehouses and the goods stored in them. The algorithm uses time as the only identifier, with a single-frame image as the input and structured data as the output. The fields in the output structured data include whether there is a forklift, whether there is a pallet, and whether there is goods. By integrating the results of the fields, it can be determined whether a forklift enters or exits the warehouse, whether the forklift carries a pallet when entering or exiting the warehouse, and whether there are goods on the pallet when the forklift enters or exits the warehouse, and it is sent to the information integration and rule judgment algorithm module in the form of a dictionary.

[0090] The object trajectory tracking algorithm module is based on the object detection algorithm module. In the warehouse, cameras in front of the door, opposite the door, on top of the door, and on the side of the door will monitor the area at the warehouse door in real time. The video image of each perspective is used as the input of the object detection algorithm module. If an object (forklift, stacking, person, and pallet) appears in the current image (i.e., the target category), the object detection algorithm module will output the coordinates of the corresponding object in the image (i.e., the target position). After obtaining the target category and target position in each perspective, the multi-view multi-object tracking algorithm (MVMT tracker) is used to extract the features of the target categories (forklift, stacking, person, and pallet) detected in each perspective, perform cross-view target association, and finally update the feature table, assign IDs, and update the bounding boxes. According to the results of the multi-view multi-object tracking algorithm (MVMT tracker), the trajectory positions of the targets (forklift, stacking, person, and pallet) in each perspective in the world coordinate system are obtained. Next, the IDs of the targets (forklift, stacking, person, and pallet) are re-assigned. By calculating the similarity of the trajectories in each perspective, the similarity calculation mainly uses the Euclidean distance and the cosine distance, and the trajectories are fused based on the numerical differences and direction differences, and then the target IDs are re-assigned. After the target ID re-assignment, the motion trajectories of the targets (forklift, stacking, person, and pallet) in 3D space can be visualized, the motion trajectories of the targets (forklift, stacking, person, and pallet) can be followed, and the entry and exit directions, i.e., inbound or outbound, can be determined.

[0091] The abnormal behavior detection algorithm module is based on the object detection algorithm module. When the detector detects a person in the video frame, the image of the person is cropped from the video frame. After resizing the cropped image to a size of 128*128, it is input into the abnormal behavior detection algorithm module. The model outputs whether the person is holding goods, and an alarm is triggered when the person enters or exits the warehouse door while holding goods. The abnormal detection behavior algorithm is a binary classification model that uses resnet101 as the backbone network. In the warehouse, the camera monitors the warehouse in real time, and its video is used as the input of the person detection model. If there is a person in the current frame, the coordinates of the corresponding person in the frame will be output, and the person area will be cropped accordingly, and the current complete image will be saved, marking the position of the person; then the cropped person image is sent into the abnormal behavior recognition model, which will identify whether the current person is carrying goods. If carrying, a warning will be issued, otherwise no warning will be given.

[0092] The personnel counting algorithm module is based on the object detection algorithm module and the object trajectory tracking algorithm module. By judging the entry and exit of the staff in the warehouse, it calculates the number of staff staying in the warehouse currently and the total number of people entering and leaving the warehouse on the same day. At the same time, combined with the face recognition algorithm, it can calculate the time that the staff entering the warehouse stays in the warehouse. In a specific warehouse scenario (such as a cold chain warehouse), when the staff stays in the warehouse for more than a certain time limit, an alarm can be triggered to ensure the safety of the staff.

[0093] The stack box counting algorithm module is based on the object detection algorithm module and the object trajectory tracking algorithm module, and counts the number of boxes on the stack during the inbound / outbound stage. To estimate the number of boxes on the stack, cameras at the front and top of the warehouse door are used, which are responsible for monitoring and shooting the two directions (front and top) of the entire stack when the forklift passes by. According to the monitored video stream, the visual algorithm is used to estimate the number of boxes on the entire stack. The calculation formula is N = n * f. In the formula, N represents the number of boxes on the stack, n represents the number of boxes on the top surface, and f represents the number of layers of the front stack. To estimate the number of boxes on the top surface of the stack and the number of layers of the front stack of the boxes, the object detection algorithm (yolo5) is used to locate and identify each box on the top surface and the front of the stack, and the number of boxes on the top surface and the number of layers of the front boxes are obtained through category counting. In the actual use process, first, video frames are extracted from the video stream, and then the object detection algorithm module is used to identify the stack, and the stack is cropped from the video frame to ensure that the area to be detected occupies the main part of the image. Then the preprocessed image is input into the box target detector to obtain the target category and its position information, and finally the number of boxes is obtained through category counting.

[0094] After summarizing the results of other algorithm modules, the information integration and rule judgment algorithm module uses the warehouse ID, camera IP, and the time of video frame reception as the unique identifier to perform information fusion, and gives information such as the entry and exit of personnel / forklifts / stacks / pallets from the warehouse and their movement trajectories in and out of the warehouse, determines whether a person has permission when entering or leaving the warehouse, whether a person is carrying goods when entering or leaving, how many people are in the warehouse and their in-warehouse time, how many goods are on each stack when a stack enters or leaves the warehouse, and the status of the emergency fire door. When a set rule is triggered, an alarm is issued. For events that require an alarm, the information integration and rule judgment algorithm module saves the relevant information and images and uploads them to the cloud server for evidence storage, and finally, manual review and determination are performed, greatly reducing the workload of review.

[0095] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or steps or stages in other steps.

[0096] Based on the same inventive concept, the embodiments of the present application also provide a warehouse management device for implementing the above-mentioned warehouse management method. The implementation solutions provided by this device to solve problems are similar to the implementation solutions described in the above method. Therefore, the specific limitations in one or more embodiments of the warehouse management device provided below can refer to the limitations on the warehouse management method in the above text, and will not be repeated here.

[0097] In one embodiment, as Figure 6 shown, a warehouse management device 600 is provided, including: a personnel detection module 601, a fire door detection module 602, a trajectory detection module 603, and an information integration module 604, where:

[0098] The personnel detection module 601 is used to obtain the face images of personnel entering and leaving the warehouse, and process the face images through a face recognition model to obtain a face recognition result;

[0099] The fire door detection module 602 is used to obtain the status of the emergency fire door, and the status of the emergency fire door is divided into an open state and a closed state;

[0100] A trajectory detection module 603, configured to obtain monitoring images of a managed object in a warehouse from multiple perspectives, and process the monitoring images of the managed object from multiple perspectives through an object trajectory tracking model to obtain the movement trajectory of the managed object;

[0101] An information integration module 604, configured to obtain the characteristic information of the warehouse, integrate the characteristic information of the warehouse, the face recognition result, the status of the emergency fire door, and the movement trajectory of the managed object to obtain warehouse integration data, and generate a prompt message according to the warehouse integration data.

[0102] In one embodiment, the personnel detection module 601 is further configured to extract the face features of the face image through a face recognition model, and perform a matching comparison between the face features of the face image and the preset authorized face features in the authorized face feature library; if the face features of the face image can match the preset authorized face features in the authorized face feature library, the face recognition result that the personnel is authorized is obtained; if the face features of the face image cannot match the preset authorized face features in the authorized face feature library, the face recognition result that the personnel is not authorized is obtained.

[0103] In one embodiment, the trajectory detection module 603 is further configured to obtain the managed object through an object detection model, and obtain monitoring images of the managed object in the warehouse from multiple perspectives; process the monitoring images of each perspective of the managed object through an object trajectory tracking model to obtain the trajectory position of the managed object in the world coordinate system under each perspective; perform a similarity calculation on the trajectory positions of the managed object in the world coordinate system under each perspective to obtain the movement trajectory of the managed object.

[0104] In one embodiment, the information integration module 604 is further configured to obtain the outbound quantity and inbound quantity of the managed object according to the movement trajectory of the managed object; determine the quantity of the managed object in the warehouse according to the outbound quantity and inbound quantity of the managed object; obtain the characteristic information of the warehouse, and integrate the characteristic information of the warehouse, the face recognition result, the status of the emergency fire door, the movement trajectory of the managed object, and the quantity of the managed object in the warehouse to obtain warehouse integration data.

[0105] In one embodiment, the information integration module 604 is further configured to obtain the stacking layers and the top layer quantity of the managed object, and obtain the stacking quantity of the managed object according to the stacking layers and the top layer quantity of the managed object; determine the outbound quantity and inbound quantity of the managed object according to the movement trajectory and the stacking quantity of the managed object.

[0106] In one embodiment, the apparatus further includes:

[0107] An abnormal behavior detection module is used to obtain the personnel images in the warehouse according to a preset image size; detect the personnel images through a feature detection model, and if it is detected that the personnel in the personnel images carry goods, generate an abnormal behavior warning message.

[0108] Each module in the above warehouse management device can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in the processor of the computer device in hardware form or be independent of it, or be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each of the above modules.

[0109] In one embodiment, a computer device is provided. The computer device can be a terminal, and its internal structure diagram can be as Figure 7 shown. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner. The wireless manner can be achieved through WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a warehouse management method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device, or an external keyboard, a touchpad, or a mouse, etc.

[0110] Those skilled in the art can understand that Figure 7 the structure shown in

[0111] is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.

[0112] Obtain the face images of the personnel entering and leaving the warehouse, and process the face images through a face recognition model to obtain a face recognition result;

[0113] Obtain the status of the emergency fire door. The status of the emergency fire door is divided into an open state and a closed state;

[0114] Obtain monitoring images of the objects to be managed from multiple perspectives within the warehouse, and process the monitoring images of the objects to be managed from multiple perspectives through an object trajectory tracking model to obtain the movement trajectories of the objects to be managed;

[0115] Obtain the characteristic information of the warehouse, integrate the characteristic information of the warehouse, the face recognition result, the status of the emergency fire door, and the movement trajectories of the objects to be managed to obtain warehouse integration data, and generate a prompt message according to the warehouse integration data.

[0116] In one embodiment, when the processor executes the computer program, the following steps are further implemented:

[0117] Extract the face features of the face image through a face recognition model, and perform matching comparison between the face features of the face image and the preset authorized face features in the authorized face feature library;

[0118] If the face features of the face image can match the preset authorized face features in the authorized face feature library, obtain the face recognition result that the person is authorized;

[0119] If the face features of the face image cannot match the preset authorized face features in the authorized face feature library, obtain the face recognition result that the person is not authorized.

[0120] In one embodiment, when the processor executes the computer program, the following steps are further implemented:

[0121] Obtain the objects to be managed through an object detection model, and obtain monitoring images of the objects to be managed from multiple perspectives within the warehouse;

[0122] Process the monitoring images of each perspective of the objects to be managed through an object trajectory tracking model to obtain the trajectory positions of the objects to be managed in the world coordinate system for each perspective;

[0123] Perform similarity calculation on the trajectory positions of the objects to be managed in the world coordinate system for each perspective to obtain the movement trajectories of the objects to be managed.

[0124] In one embodiment, when the processor executes the computer program, the following steps are further implemented:

[0125] Obtain the outbound quantity and inbound quantity of the objects to be managed according to the movement trajectories of the objects to be managed;

[0126] Determine the quantity of the objects to be managed within the warehouse according to the outbound quantity and inbound quantity of the objects to be managed;

[0127] Obtain the characteristic information of the warehouse, and integrate the characteristic information of the warehouse, the face recognition result, the status of the emergency fire door, the movement trajectories of the objects to be managed, and the quantity of the objects to be managed within the warehouse to obtain warehouse integration data.

[0128] In one embodiment, when the processor executes the computer program, the following steps are further implemented:

[0129] Obtain the stacking layer number and top layer number of the object to be managed, and obtain the stacking number of the object to be managed according to the stacking layer number and top layer number of the object to be managed;

[0130] Determine the outbound quantity and inbound quantity of the object to be managed according to the movement trajectory and stacking number of the object to be managed.

[0131] In one embodiment, when the processor executes the computer program, the following steps are further implemented:

[0132] Obtain the personnel image in the warehouse according to the preset image size;

[0133] Detect the personnel image through the feature detection model. If it is detected that the personnel in the personnel image carry goods, an abnormal behavior warning message is generated.

[0134] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, the following steps are implemented:

[0135] Obtain the face image of the personnel entering and leaving the warehouse, and process the face image through the face recognition model to obtain the face recognition result;

[0136] Obtain the status of the emergency fire door, and the status of the emergency fire door is divided into an open state and a closed state;

[0137] Obtain the monitoring images of multiple perspectives of the object to be managed in the warehouse, and process the monitoring images of multiple perspectives of the object to be managed through the object trajectory tracking model to obtain the movement trajectory of the object to be managed;

[0138] Obtain the characteristic information of the warehouse, integrate the characteristic information of the warehouse, the face recognition result, the status of the emergency fire door and the movement trajectory of the object to be managed to obtain the integrated data of the warehouse, and generate a prompt message according to the integrated data of the warehouse.

[0139] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented:

[0140] Extract the face features of the face image through the face recognition model, and match and compare the face features of the face image with the preset authorized face features in the authorized face feature library;

[0141] If the face features of the face image can match the preset authorized face features in the authorized face feature library, the face recognition result that the personnel are authorized is obtained;

[0142] If the facial features of the face image cannot match the preset authorized facial features in the authorized face feature library, a face recognition result of unauthorized personnel is obtained.

[0143] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented:

[0144] Obtain the managed object through the object detection model, and obtain the monitoring images of multiple perspectives of the managed object in the warehouse;

[0145] Process the monitoring images of each perspective of the managed object through the object trajectory tracking model to obtain the trajectory position of the managed object in the world coordinate system for each perspective;

[0146] Calculate the similarity of the trajectory positions of the managed object in the world coordinate system for each perspective to obtain the movement trajectory of the managed object.

[0147] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented:

[0148] Obtain the outbound quantity and inbound quantity of the managed object according to the movement trajectory of the managed object;

[0149] Determine the quantity of the managed object in the warehouse according to the outbound quantity and inbound quantity of the managed object;

[0150] Obtain the feature information of the warehouse, and integrate the feature information of the warehouse, the face recognition result, the status of the emergency fire door, the movement trajectory of the managed object, and the quantity of the managed object in the warehouse to obtain the integrated warehouse data.

[0151] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented:

[0152] Obtain the stacking layers and the top layer quantity of the managed object, and obtain the stacking quantity of the managed object according to the stacking layers and the top layer quantity of the managed object;

[0153] Determine the outbound quantity and inbound quantity of the managed object according to the movement trajectory and the stacking quantity of the managed object.

[0154] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented:

[0155] Obtain the personnel image in the warehouse according to the preset image size;

[0156] Detect the personnel image through the feature detection model. If it is detected that the personnel in the personnel image carry goods, an abnormal behavior warning message is generated.

[0157] In one embodiment, a computer program product is provided, including a computer program which, when executed by a processor, implements the following steps:

[0158] Obtain the face images of the personnel entering and leaving the warehouse, and process the face images through a face recognition model to obtain a face recognition result;

[0159] Obtain the status of the emergency fire door, and the status of the emergency fire door is divided into an open state and a closed state;

[0160] Obtain the surveillance images of multiple perspectives of the objects to be managed in the warehouse, and process the surveillance images of multiple perspectives of the objects to be managed through an object trajectory tracking model to obtain the movement trajectories of the objects to be managed;

[0161] Obtain the characteristic information of the warehouse, integrate the characteristic information of the warehouse, the face recognition result, the status of the emergency fire door, and the movement trajectories of the objects to be managed to obtain the integrated data of the warehouse, and generate a prompt message according to the integrated data of the warehouse.

[0162] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0163] Extract the face features of the face image through the face recognition model, and match and compare the face features of the face image with the preset authorized face features in the authorized face feature library;

[0164] If the face features of the face image can match the preset authorized face features in the authorized face feature library, obtain the face recognition result that the personnel are authorized;

[0165] If the face features of the face image cannot match the preset authorized face features in the authorized face feature library, obtain the face recognition result that the personnel are not authorized.

[0166] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0167] Obtain the objects to be managed through an object detection model, and obtain the surveillance images of multiple perspectives of the objects to be managed in the warehouse;

[0168] Process the surveillance image of each perspective of the object to be managed through an object trajectory tracking model to obtain the trajectory position of the object to be managed in the world coordinate system under each perspective;

[0169] Calculate the similarity of the trajectory positions of the objects to be managed in the world coordinate system under each perspective to obtain the movement trajectories of the objects to be managed.

[0170] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0171] Obtain the outbound quantity and inbound quantity of the managed object based on the movement track of the managed object;

[0172] Determine the quantity of the managed object inside the warehouse according to the outbound quantity and inbound quantity of the managed object;

[0173] Obtain the characteristic information of the warehouse, and integrate the characteristic information of the warehouse, the face recognition result, the status of the emergency fire door, the movement track of the managed object, and the quantity of the managed object inside the warehouse to obtain the integrated warehouse data.

[0174] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented:

[0175] Obtain the stacking layers and the top layer quantity of the managed object, and obtain the stacking quantity of the managed object according to the stacking layers and the top layer quantity of the managed object;

[0176] Determine the outbound quantity and inbound quantity of the managed object according to the movement track and the stacking quantity of the managed object.

[0177] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented:

[0178] Obtain the personnel image inside the warehouse according to the preset image size;

[0179] Detect the personnel image through the feature detection model. If it is detected that the personnel in the personnel image carry goods, an abnormal behavior warning message is generated.

[0180] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0181] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0182] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0183] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A warehouse management method, characterized in that, The method includes: Obtaining a face image of a person entering or leaving the warehouse, and processing the face image through a face recognition model to obtain a face recognition result; Obtaining the status of the emergency fire door, where the status of the emergency fire door is divided into an open state and a closed state; Obtaining monitoring images of multiple perspectives of the objects to be managed in the warehouse, obtaining the target positions and target categories of the objects to be managed through an object detection model, extracting features from the target categories and target positions detected in each perspective through a multi-perspective multi-object tracking model, performing cross-perspective object association, and obtaining the trajectory positions of each object to be managed in the world coordinate system in each perspective; calculating the similarity of the trajectory positions of the objects to be managed in the world coordinate system in each perspective to obtain the movement trajectory of the objects to be managed in the world coordinate system; When the object to be managed is a staff member, identifying the monitoring image through an object detection model, outputting the coordinates of the staff member in the picture, and cropping the staff member area; sending the cropped staff member image into a feature detection model to identify whether the current staff member is carrying goods. If carrying goods, a warning is issued, otherwise no warning is issued; based on the object detection model and the object trajectory tracking model, by judging the entry and exit of the staff member from the warehouse, calculating the number of staff members staying in the warehouse currently and the total number of people entering and leaving the warehouse on the same day; combining the face recognition model to calculate the staying time of the staff member entering the warehouse in the warehouse; in the case of a cold chain warehouse, when the staying time of the staff member in the warehouse exceeds a preset time limit, an alarm is issued to ensure the safety of the staff member; the objects to be managed also include: forklifts, pallets, and cargo boxes; when the object to be managed is a cargo box, extracting video frames from the monitoring video, using an object detection model to identify the stack in the video frame, cropping the stack from the video frame to obtain a stack picture, inputting the stack picture into a cargo box object detection model to obtain the target category and its position information, and obtaining the stack layer number f of the stack and the number n of the top-layer cargo boxes. Obtaining the number of cargo boxes on the stack through the calculation formula N = n * f, and determining the outbound quantity and inbound quantity of the object to be managed according to the movement trajectory of the cargo box and the number of cargo boxes on the stack.

2. The method according to claim 1, characterized in that The processing of the face image through the face recognition model to obtain a face recognition result includes: Extracting the face features of the face image through the face recognition model, and matching and comparing the face features of the face image with the preset authorized face features in the authorized face feature library; If the face features of the face image can match the preset authorized face features in the authorized face feature library, obtaining a face recognition result that the person is authorized; If the face features of the face image cannot match the preset authorized face features in the authorized face feature library, obtaining a face recognition result that the person is not authorized.

3. A warehouse management device, characterized in that, The device includes: A personnel detection module, configured to obtain a face image of a person entering or leaving the warehouse, and process the face image through a face recognition model to obtain a face recognition result; A fire door detection module, configured to obtain the status of the emergency fire door, where the status of the emergency fire door is divided into an open state and a closed state; A trajectory detection module, which is used to obtain monitoring images of a managed object from multiple perspectives in a warehouse, obtain the target position and target category of the managed object through an object detection model, extract features from the target category and target position detected in each perspective through a multi-perspective multi-object tracking model, perform cross-perspective object association, and obtain the trajectory position of each managed object in the world coordinate system under each perspective; calculate the similarity of the trajectory positions of the managed object in the world coordinate system under each perspective to obtain the movement trajectory of the managed object in the world coordinate system. An information integration module, when the managed object is a staff member, is used to identify the monitoring image through an object detection model, output the coordinates of the staff member in the picture, and crop the staff member area; send the cropped personnel image into a feature detection model to identify whether the current staff member is carrying goods. If carrying goods, give an alarm, otherwise do not give an alarm; based on the object detection model and the object trajectory tracking model, by judging the entry and exit of the staff member from the warehouse, calculate the number of staff members staying in the warehouse currently and the total number of people entering and leaving the warehouse on the same day; combine the face recognition model to calculate the staying time of the staff member entering the warehouse in the warehouse; in the case of a cold chain warehouse, when the staying time of the staff member in the warehouse exceeds a preset time limit, give an alarm to ensure the safety of the staff member; the managed objects also include: forklifts, pallets and cargo boxes; when the managed object is a cargo box, extract video frames from the monitoring video, use an object detection model to identify the stack in the video frame, crop the stack from the video frame to obtain a stack picture, input the stack picture into a cargo box object detection model to obtain the target category and its position information, and obtain the number of layers f of the stack and the number of top-layer cargo boxes n of the stack. Calculate the number of cargo boxes on the stack through the calculation formula N = n * f, and determine the outbound quantity and inbound quantity of the managed object according to the movement trajectory of the cargo box and the number of cargo boxes on the stack.

4. The device according to claim 3, characterized in that, The personnel detection module is further used for: extracting the face features of the face image through a face recognition model, and matching and comparing the face features of the face image with the preset authorized face features in the authorized face feature library; if the face features of the face image can match the preset authorized face features in the authorized face feature library, obtaining a face recognition result that the personnel are authorized; if the face features of the face image cannot match the preset authorized face features in the authorized face feature library, obtaining a face recognition result that the personnel are not authorized.

5. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 2.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method described in any one of claims 1 to 2.

7. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method described in any one of claims 1 to 2.

Images