Storage monitoring method and device, computer equipment and storage medium

Abstract

The invention belongs to the technical field of artificial intelligence and big data, and relates to a storage monitoring method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining video streams inputted by a plurality of cameras in a warehouse; distributing a multi-source prefetcher corresponding to the video stream according to the video feature of the video stream; acquiring a video frame processing rate of the target detection model, and performing frame synchronization processing on the video stream by using a multi-source prefetcher according to the video frame processing rate to obtain synchronized frame data; an admission controller is adopted to determine the admission priority of each video frame in the synchronized frame data, and each video frame is input to a target detection model according to the admission priority and the video frame processing rate; and performing target detection on each video frame by using the target detection model to obtain a target detection result. In addition, the invention also relates to a block chain technology, and data such as the video stream and the target detection result can be stored in a block chain. According to the invention, real-time object detection can be realized on a resource-limited edge device.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and big data technology, and in particular to a warehouse monitoring method, device, computer equipment and storage medium. Background Art

[0002] In the field of warehouse management, monitoring whether items have been moved is a vital task. However, traditional monitoring methods mainly rely on manual work, which has many shortcomings. First, manual monitoring is inefficient and prone to errors, because people have limited attention and it is difficult to maintain a high degree of concentration for a long time, which can easily miss some abnormal changes. Secondly, manual monitoring cannot achieve 24-hour uninterrupted monitoring, especially during non-working hours, some abnormal behaviors may be ignored, posing a hidden danger to warehouse safety. In addition, the cost of manual monitoring is relatively high, requiring a large amount of human resources investment, including the recruitment, training and management of monitoring personnel, which is a considerable expense for warehousing companies.

[0003] In order to overcome the limitations of manual monitoring, current technology introduces object detection technology YOLO and deploys YOLO on edge devices such as embedded devices or IoT devices in warehouses. However, since edge devices usually have limited computing resources, including processor speed, memory, and storage space, when YOLO runs on these devices, it is limited by hardware resources and cannot meet real-time or near-real-time performance requirements. Therefore, in the face of the above problems, there is an urgent need for a warehouse monitoring solution that can achieve real-time object detection on resource-constrained edge devices. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a warehouse monitoring method, apparatus, computer equipment and storage medium that can achieve real-time object detection on resource-constrained edge devices.

[0005] In order to solve the above technical problems, the present application embodiment provides a warehouse monitoring method, which adopts the following technical solution:

[0006] Get video streams from multiple cameras in the warehouse;

[0007] Extracting video features from the video stream, and allocating a multi-source prefetcher corresponding to the video stream according to the video features;

[0008] Acquire a video frame processing rate of a target detection model, and use the multi-source pre-fetcher to perform frame synchronization processing on the video stream according to the video frame processing rate to obtain synchronized frame data;

[0009] Inputting the synchronized frame data into an admission controller, and using the admission controller to determine the admission priority of each video frame in the synchronized frame data;

[0010] Inputting each video frame into the object detection model according to the admission priority and the video frame processing rate;

[0011] The target detection model is used to perform target detection on each of the input video frames to obtain a target detection result.

[0012] In order to solve the above technical problems, the embodiment of the present application further provides a storage monitoring device, which adopts the following technical solution:

[0013] An acquisition module, used to acquire video streams input from multiple cameras in the warehouse;

[0014] An allocation module, used for extracting video features from the video stream and allocating a multi-source prefetcher corresponding to the video stream according to the video features;

[0015] A synchronization module, used to obtain a video frame processing rate of a target detection model, and use the multi-source pre-fetcher to perform frame synchronization processing on the video stream according to the video frame processing rate to obtain synchronized frame data;

[0016] A determination module, configured to input the synchronized frame data into an admission controller, and use the admission controller to determine the admission priority of each video frame in the synchronized frame data;

[0017] An input module, configured to input each video frame into the target detection model according to the admission priority and the video frame processing rate;

[0018] The detection module is used to use the target detection model to perform target detection on each input video frame to obtain a target detection result.

[0019] In order to solve the above-mentioned technical problems, an embodiment of the present application also provides a computer device, which adopts the technical solution described below: the computer device includes a memory and a processor, the memory stores computer-readable instructions, and the processor implements the steps of the warehouse monitoring method described in any one of the above items when executing the computer-readable instructions.

[0020] In order to solve the above-mentioned technical problems, an embodiment of the present application also provides a computer-readable storage medium, which adopts the following technical solution: the computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the warehouse monitoring method as described in any one of the above items are implemented.

[0021] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0022] The embodiment of the present application can integrate video information from multiple perspectives and effectively monitor the status of items in the warehouse by acquiring video streams from multiple cameras in the warehouse; by analyzing the video features of the video stream and allocating a suitable multi-source prefetcher to each video stream based on the video features, it is ensured that the video data is optimally allocated in the preprocessing stage, avoiding data accumulation caused by the edge device's untimely data processing, and improving processing efficiency; through the multi-source prefetcher, the video stream is synchronously processed according to the video frame processing rate of the target detection model, which can avoid detection delays or errors caused by frame rate mismatch; by using an admission controller to manage the admission priority of the video frames, and according to the admission priority and the video frame processing rate, the video frames are controlled to be input into the target detection model in sequence and speed, which can ensure that more critical video frames can be processed first in a resource-constrained environment, which helps to improve the real-time response speed and monitoring accuracy; by using the target detection model to efficiently process the synchronously input video frames and output accurate target detection results, comprehensive, real-time and accurate monitoring of items in the warehouse is achieved. Based on the solution of this application, resource-constrained edge devices can achieve real-time object detection without relying on high-performance hardware, which not only improves the efficiency and accuracy of warehouse monitoring, but also reduces the cost and risk of manual monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0025] Figure 2 It is a flowchart of an embodiment of the warehouse monitoring method of the present application;

[0026] Figure 3 It is a structural schematic diagram of an embodiment of a storage monitoring device of the present application;

[0027] Figure 4 It is a structural diagram of an embodiment of a computer device of the present application. DETAILED DESCRIPTION

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0029] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0031] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0032] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0033] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0034] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .

[0035] It should be noted that the warehouse monitoring method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the warehouse monitoring device is generally set in the server / terminal device.

[0036] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0037] Continue to refer Figure 2 , showing a flow chart of an embodiment of the question-answering method according to the present application. The warehouse monitoring method comprises the following steps:

[0038] Step S201, obtaining video streams input from multiple cameras in the warehouse;

[0039] In this embodiment, the electronic device (eg, Figure 1 The server / terminal device shown in the figure can obtain the video streams input from multiple cameras in the warehouse through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 2G / 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0040] Specifically, in warehouse management, in order to achieve comprehensive monitoring, cameras are installed in multiple corners and key locations of the warehouse, and the cameras are used to capture the dynamics of the warehouse in real time and generate relevant video streams. When multiple cameras in the warehouse transmit the generated video streams to the electronic device, the video streams input from the multiple cameras in the warehouse are obtained by the electronic device.

[0041] Step S202, extracting video features from the video stream, and allocating a multi-source prefetcher corresponding to the video stream according to the video features;

[0042] In this embodiment, for the acquired video stream, feature extraction is performed on the video stream to obtain video features of the video stream, wherein the video features may include but are not limited to the clarity, color features, motion features, and video memory features of the video content. According to the extracted video features, a corresponding multi-source prefetcher is assigned to each video stream. For example, for a video stream whose motion feature indicates a high dynamic range, a multi-source prefetcher with higher processing capability and efficiency may be matched.

[0043] It should be noted that the multi-source prefetcher (MSP) is a component used to pre-acquire data from multiple data sources (such as multiple cameras or video streams), and is one of the components of the adaptive frame control (AFC). Among them, the adaptive frame control (AFC) is an architecture or mechanism for managing and optimizing video stream processing, which includes multiple components such as the multi-source prefetcher (MSP) and the admission controller (AC). The multi-source prefetcher can continuously read video frames from different cameras. Through the synchronizer deployed internally, it can coordinate the reading rate of video frames with the processing capacity of the target detection program (that is, the number of frames that can be processed per second FPS). The admission controller (AC) is a component used to control the admission of video frames to the processing queue or target detection program. The admission controller can intelligently control which video frames should be processed and which video frames should be discarded or delayed to ensure that the system's processing capacity and resources are effectively utilized.

[0044] In some optional implementations, the allocating the multi-source prefetcher corresponding to the video stream according to the video feature may include the following steps:

[0045] Classifying and marking the video stream according to the video memory feature in the video feature to obtain a memory tag of the video stream;

[0046] Specifically, according to the video features of the video stream, the video memory features of the video stream are extracted therefrom, and the video memory features may include the storage space size required for the video stream, the storage space size required for each video frame in the video stream, etc. According to the extracted video memory features, the video stream is classified and labeled to obtain a memory label of the video stream, and the memory label may include low memory occupancy, medium memory occupancy, high memory occupancy, etc.

[0047] Obtaining load distribution information of each multi-source prefetcher, assigning a corresponding memory tag to each multi-source prefetcher according to the load distribution information, and obtaining a target memory tag assigned to each multi-source prefetcher;

[0048] Specifically, the load distribution information of each multi-source prefetcher is monitored and recorded, and the load distribution information may include the number of video streams currently processed, the occupied memory resources, the processing speed, etc. According to the load distribution information of each multi-source prefetcher, the available resources of each multi-source prefetcher are evaluated, and one or more memory tags are allocated based on the available resources of each multi-source prefetcher to obtain the target memory tag allocated to each multi-source prefetcher.

[0049] The video stream corresponding to the target memory tag is allocated to the multi-source prefetcher corresponding to the target memory tag.

[0050] Specifically, according to the target memory tag assigned to each multi-source prefetcher, the video stream corresponding to the target memory tag is obtained, and the video stream is matched with the multi-source prefetcher corresponding to the target memory tag.

[0051] The embodiment of the present application classifies and labels the video streams based on video memory features and assigns a memory tag to each video stream, thereby being able to more accurately identify the resource requirements of each video stream; by assigning a target memory tag to each multi-source prefetcher based on the load distribution information of each multi-source prefetcher, load balancing can be achieved, thereby avoiding the situation where some multi-source prefetchers are overloaded while other multi-source prefetchers are idle; by assigning the video stream corresponding to the target memory tag to the corresponding multi-source prefetcher, it is ensured that the video stream can be efficiently assigned to a more suitable multi-source prefetcher for processing, thereby reducing data accumulation, avoiding processing delays of video data in the preprocessing stage, and improving real-time performance.

[0052] Step S203, obtaining a video frame processing rate of the target detection model, and using the multi-source pre-fetcher to perform frame synchronization processing on the video stream according to the video frame processing rate to obtain synchronized frame data;

[0053] In this embodiment, the video frame processing rate of the target detection model is obtained. The target detection model is a computer vision model for accurately identifying and locating the target object from an image or video. The video frame processing rate refers to the number of video frames that the model can process per second. A multi-source pre-fetcher is used to synchronously adjust the reading rate of each video frame of the video stream according to the video frame processing rate of the target detection model, so that the reading rate of each video frame is synchronously matched with the video frame processing rate to obtain synchronized frame data. For example, assuming that the video frame processing rate of the target detection model is 60 frames per second, the reading rate of the multi-source pre-fetcher is set to extract 60 frames from the video stream per second, and the reading time interval between two adjacent frames is set to 1 / 60 seconds.

[0054] In some optional implementations, the multi-source prefetcher is used to perform frame synchronization processing on the video stream according to the video frame processing rate to obtain synchronized frame data, which may include the following steps:

[0055] Inputting the video stream into the multi-source prefetcher, and using the multi-source prefetcher to obtain an input rate of each video frame in the video stream;

[0056] Specifically, the video stream is input into the multi-source prefetcher, and the multi-source prefetcher is used to monitor and calculate the input rate of each video frame in the input video stream, that is, the number of frames input into the multi-source prefetcher per second. The input rate can also be calculated based on the time interval between the input video frames.

[0057] Comparing the input rate with the video frame processing rate to obtain a rate comparison result;

[0058] Specifically, the calculated input rate is compared with the video frame processing rate of the target detection model to determine whether there is a rate mismatch problem, and a rate comparison result is obtained. If the rate comparison result is that the input rate is less than or equal to the video frame processing rate, it indicates that there is no rate mismatch problem; if the rate comparison result is that the input rate is greater than the video frame processing rate, it indicates that there is a rate mismatch problem.

[0059] If the rate comparison result is that the input rate is greater than the video frame processing rate, the reading time interval of the video frame is synchronized with the video frame processing rate to obtain synchronized frame data.

[0060] Specifically, when the rate comparison result is that the input rate is greater than the video frame processing rate, it indicates that there is a rate mismatch problem, so that the number of frames provided by the multi-source prefetcher to the target detection model exceeds the processing capacity of the model. At this time, the reading time interval of the video frame is synchronized with the video frame processing rate of the target detection model to obtain synchronized frame data. For example, if the video frame processing rate of the target detection model can process 30 frames per second, and the input rate reaches 60 frames / second, the reading time interval can be set to twice the time interval between two frames to ensure that only 30 frames are provided to the target detection model per second.

[0061] The embodiment of the present application pre-processes the video stream through a multi-source pre-fetcher, and can obtain the input rate of the video frame in real time, and compare the input rate with the video frame processing rate of the target detection model, ensuring the coordination and matching between the key links (input and processing of the video frame) in the video processing flow. When the input rate is higher than the processing rate, by adjusting the reading time interval of the video frame, the input speed of the video frame can be consistent with the processing speed of the target detection model, thereby avoiding the backlog or loss of the video frame caused by the mismatch of the processing speed. In addition, the frame synchronization process can also achieve more efficient computing resource utilization by ensuring that the input of the video frame matches the processing speed, avoiding the waste of resources caused by the mismatch of the processing speed.

[0062] Step S204, inputting the synchronized frame data into an admission controller, and using the admission controller to determine the admission priority of each video frame in the synchronized frame data;

[0063] In this embodiment, the frame data after the frame synchronization processing by the multi-source prefetcher is input into the admission controller, and the admission controller performs a priority evaluation on each video frame in the synchronized frame data according to a preset priority rule to determine the admission priority of each video frame. The priority rule can be set according to one or more evaluation factors, for example, according to the content of the video frame (such as whether it contains target features), timestamp (such as the video frame with a closer capture time has a higher priority) and the correlation between frames (such as the degree of change between consecutive frames) and other evaluation factors.

[0064] In some optional implementations, the step of using the admission controller to determine the admission priority of each video frame in the synchronized frame data may include the following steps:

[0065] extracting each video frame from the synchronized frame data using the admission controller;

[0066] Calculate the feature hit score of each video frame according to the preset important warehouse area features and suspicious target features, wherein the feature hit score is a numerical value indicating the degree to which the important warehouse area features and suspicious target features are contained in the video frame;

[0067] An admission priority is assigned to each video frame according to the feature hit score.

[0068] Specifically, important warehouse area features refer to specific attributes of the warehouse itself and its surrounding environment, which may include geographical location, architectural features, security facilities, and cargo types. Suspicious target features refer to features that are found in video surveillance to pose security risks or potential threats, which may include abnormal behavior, appearance features, and items carried.

[0069] Specifically, important warehouse area features and suspicious target features that need to be focused on are pre-set. For the synchronized frame data input from the multi-source pre-fetcher, an admission controller is used to extract each video frame from the synchronized frame data, and detect whether each video frame contains preset important warehouse area features and suspicious target features. According to the degree of containing important warehouse area features and suspicious target features in each video frame, the feature hit score of the video frame is calculated. For example, the content of each video frame is identified, and the content of the identified video frame is matched with the preset important warehouse area features and suspicious target features to obtain a feature matching result, which indicates the degree of containing important warehouse area features and suspicious target features in the video frame. According to the feature matching result, the feature hit score of each video frame is calculated, and if the content of the video frame hits the important warehouse area features or the suspicious target features, the score is calculated. The feature hit score indicates the degree of containing important warehouse area features and suspicious target features in the video frame. According to the feature hit score of each frame of video, a corresponding admission priority is assigned to each video frame. For example, the feature hit score of each frame of video is compared with a preset threshold. When the feature hit score is higher than the preset threshold, the video frame corresponding to the feature hit score is assigned an admission priority corresponding to the preset threshold. The admission priority can be divided into low, medium, high, etc.

[0070] The embodiment of the present application calculates the feature hit score of each video frame according to the preset important warehouse area features and suspicious target features, and can quantify the degree of key features contained in the video frame from multiple dimensions, and more comprehensively and accurately evaluate the importance and potential threats of the video frame; by assigning access priority to each video frame according to the feature hit score, it is ensured that high-scoring video frames (i.e., video frames containing important warehouse area features or suspicious target features) can be processed or displayed first. Based on priority allocation, the resources of edge devices can be more effectively utilized, potential security threats can be quickly responded to and processed, and the overall security monitoring efficiency can be improved.

[0071] Step S205, inputting each video frame into the object detection model according to the admission priority and the video frame processing rate;

[0072] Specifically, according to the access priority of each video frame and the video frame processing rate of the target detection model, each video frame is arranged in order from high to low according to the access priority, and according to the video frame processing rate, video frames are selected in turn from the arranged frame data and input into the target detection model, so that the target detection model can process the input video frames.

[0073] In some optional implementations, the step of inputting each video frame into the object detection model according to the admission priority and the video frame processing rate may include the following steps:

[0074] Prioritize each of the video frames according to the admission priority to obtain a queue of video frames to be input;

[0075] Each video frame in the video frame queue is input into the target detection model in sequence according to the video frame processing rate.

[0076] Specifically, each video frame is arranged in descending order of access priority to obtain a video frame queue to be input. For each video frame in the video frame queue, according to the video frame processing rate of the target detection model, video frames are sequentially selected from the video frame queue and input into the target detection model, so that the target detection model processes the input video frames.

[0077] Step S206: Use the target detection model to perform target detection on each input video frame to obtain a target detection result.

[0078] In this embodiment, a pre-trained target detection model is obtained, and the target detection model may be a YOLO (You Only Look Once) model, an SSD (Single Shot MultiBox Detector) model, a Faster R-CNN model, etc. The target detection model is used to perform target detection on each video frame input from the admission controller to obtain a target detection result.

[0079] In some optional implementations, the step of performing target detection on each of the input video frames using the target detection model to obtain a target detection result may include the following steps:

[0080] Using the target detection model to process each of the input video frames to identify the target object in each of the video frames;

[0081] Obtain a predefined storage anomaly detection rule, and perform anomaly detection on the target object according to the storage anomaly detection rule;

[0082] If the target object meets the warehouse anomaly detection rule, a target detection result including anomaly prompt information is generated according to the target object.

[0083] Specifically, the target detection model is used to process each input video frame, such as resizing, color space conversion, normalization, etc., to identify the content contained in each video frame after processing, and obtain the target object in each video frame. A predefined storage anomaly detection rule is obtained, and the storage anomaly detection rule is set based on the monitoring requirements such as the category, quantity, location, and motion state of the objects required for warehouse management. According to the storage anomaly detection rule, anomaly detection is performed on the identified target object, for example, checking whether the category of the target object is consistent with the abnormal category defined in the storage anomaly detection rule, whether the number of target objects exceeds or is lower than the threshold set in the storage anomaly detection rule, whether the location of the target object is within the abnormal area defined in the storage anomaly detection rule, and whether the motion state of the target object is consistent with the abnormal mode defined in the storage anomaly detection rule. When the target object meets the storage anomaly detection rule, the target object is determined to be an abnormal object, and a target detection result containing abnormal prompt information is generated according to the category, location, quantity, motion state, etc. of the target object.

[0084] Optionally, when the target object does not conform to the warehouse anomaly detection rule, the target object is determined to be a normal object, and a target detection result including normal prompt information is generated.

[0085] The embodiment of the present application realizes intelligent monitoring of the storage environment by combining target detection and anomaly detection, which can not only improve the efficiency of storage management, but also reduce the cost of manual monitoring.

[0086] In some optional implementations, after the target detection result is obtained by using the target detection model to perform target detection on each input video frame, the following steps may also be included:

[0087] Extracting abnormal position information of the target object according to the abnormal prompt information in the target detection result;

[0088] The adjustment parameters of the camera are determined according to the abnormal position information, and the camera is controlled to monitor the target object according to the adjustment parameters.

[0089] Specifically, the abnormal prompt information is parsed from the target detection result, and the abnormal position information of the target object (i.e., the abnormal object) is extracted from the abnormal prompt information, and the abnormal position information may include the bounding box coordinates of the target object in the video frame. Determine which cameras the abnormal position information exists within the monitoring range, and determine the camera whose monitoring range contains the abnormal position information as the target camera. Compare the abnormal position information with the field of view of the target camera, and calculate the adjustment parameters (such as direction, angle, zoom, etc.) of the target camera to better monitor the target object. Generate a corresponding camera adjustment instruction based on the calculated adjustment parameters, and control the camera to monitor the target object based on the camera adjustment instruction.

[0090] The embodiment of the present application extracts abnormal position information from the target detection results, compares the abnormal position with the camera's field of view to determine the adjustment parameters, and finally sends adjustment instructions and controls the camera for monitoring, thereby ensuring that the camera can monitor the target object in the abnormal position at a suitable angle and field of view, and continuously detect abnormal situations, thereby improving the intelligence and automation level of warehouse monitoring and providing more accurate and efficient monitoring means for warehouse management.

[0091] The embodiment of the present application adopts the above scheme, specifically by acquiring video streams from multiple cameras in the warehouse, it is possible to integrate video information from multiple perspectives and effectively monitor the status of items in the warehouse; by analyzing the video features of the video stream and allocating a suitable multi-source pre-fetcher to each video stream according to the video features, it is ensured that the video data is optimally allocated in the preprocessing stage, avoiding data accumulation caused by the edge device's untimely data processing, and improving processing efficiency; through the multi-source pre-fetcher, the video stream is synchronously processed according to the video frame processing rate of the target detection model, which can avoid detection delays or errors caused by frame rate mismatch; by using an admission controller to manage the admission priority of the video frames, and according to the admission priority and the video frame processing rate, the video frames are controlled to be input into the target detection model in sequence and speed, which can ensure that more critical video frames can be processed first in a resource-constrained environment, which helps to improve the real-time response speed and monitoring accuracy; by using the target detection model to efficiently process the synchronously input video frames and output accurate target detection results, comprehensive, real-time and accurate monitoring of items in the warehouse is achieved. Based on the solution of this application, resource-constrained edge devices can achieve real-time object detection without relying on high-performance hardware, which not only improves the efficiency and accuracy of warehouse monitoring, but also reduces the cost and risk of manual monitoring.

[0092] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned video streams, target detection models, target detection results and other information as well as other relevant data, the above-mentioned video streams, target detection models, target detection results and other information as well as other relevant data can also be stored in a blockchain node.

[0093] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0094] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0095] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0096] Further references Figure 3 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of a storage monitoring device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0097] like Figure 3As shown, the storage monitoring device 400 described in this embodiment includes: an acquisition module 401, an allocation module 402, a synchronization module 403, a determination module 404, an input module 405 and a detection module 406. Among them:

[0098] An acquisition module 401 is used to acquire video streams input from multiple cameras in the warehouse;

[0099] An allocation module 402, configured to extract video features from the video stream and allocate a multi-source prefetcher corresponding to the video stream according to the video features;

[0100] A synchronization module 403 is used to obtain a video frame processing rate of the target detection model, and use the multi-source pre-fetcher to perform frame synchronization processing on the video stream according to the video frame processing rate to obtain synchronized frame data;

[0101] A determination module 404, configured to input the synchronized frame data into an admission controller, and use the admission controller to determine the admission priority of each video frame in the synchronized frame data;

[0102] An input module 405, configured to input each video frame into the object detection model according to the admission priority and the video frame processing rate;

[0103] The detection module 406 is used to perform target detection on each of the input video frames using the target detection model to obtain a target detection result.

[0104] In this embodiment, in warehouse management, in order to achieve comprehensive monitoring, cameras are installed in multiple corners and key locations of the warehouse, and the cameras are used to capture the dynamics of the warehouse in real time and generate relevant video streams. When multiple cameras in the warehouse transmit the generated video streams to the acquisition module 401, the acquisition module 401 is used to acquire the video streams input from the multiple cameras in the warehouse.

[0105] Specifically, the allocation module 402 is used to extract features from the acquired video stream to obtain video features of the video stream, wherein the video features may include but are not limited to the clarity, color features, motion features, and video memory features of the video content. According to the extracted video features, a corresponding multi-source prefetcher is allocated to each video stream. For example, for a video stream whose motion feature indicates a high dynamic range, a multi-source prefetcher with higher processing capability and efficiency can be matched.

[0106] It should be noted that the multi-source prefetcher (MSP) is a component used to pre-acquire data from multiple data sources (such as multiple cameras or video streams), and is one of the components of the adaptive frame control (AFC). Among them, the adaptive frame control (AFC) is an architecture or mechanism for managing and optimizing video stream processing, which includes multiple components such as the multi-source prefetcher (MSP) and the admission controller (AC). The multi-source prefetcher can continuously read video frames from different cameras. Through the internally deployed synchronizer, it can coordinate the reading rate of video frames with the processing capacity of the target detection program (that is, the number of frames that can be processed per second FPS). The admission controller (AC) is a component used to control the admission of video frames to the processing queue or target detection program. The admission controller can intelligently control which video frames should be processed and which video frames should be discarded or delayed to ensure that the system's processing power and resources are effectively utilized.

[0107] Specifically, the synchronization module 403 is used to obtain the video frame processing rate of the target detection model. The target detection model is a computer vision model used to accurately identify and locate the target object from an image or video. The video frame processing rate refers to the number of video frames that the model can process per second. A multi-source pre-fetcher is used to synchronously adjust the reading rate of each video frame of the video stream according to the video frame processing rate of the target detection model, so that the reading rate of each video frame is synchronously matched with the video frame processing rate to obtain synchronized frame data. For example, assuming that the video frame processing rate of the target detection model is 60 frames per second, the reading rate of the multi-source pre-fetcher is set to extract 60 frames from the video stream per second, and the reading time interval between two adjacent frames is set to 1 / 60 second.

[0108] Specifically, the determination module 404 is used to input the frame data after the frame synchronization processing by the multi-source prefetcher into the admission controller, and the admission controller performs a priority evaluation on each video frame in the synchronized frame data according to a preset priority rule to determine the admission priority of each video frame. The priority rule can be set according to one or more evaluation factors, for example, according to the content of the video frame (such as whether it contains target features), timestamp (such as the video frame with a closer capture time has a higher priority) and the correlation between frames (such as the degree of change between consecutive frames) and other evaluation factors to obtain the priority rule.

[0109] Specifically, the input module 405 is used to arrange each video frame in order from high to low according to the access priority of each video frame and the video frame processing rate of the target detection model, and select video frames from the arranged frame data in turn according to the video frame processing rate and input them into the target detection model so that the target detection model can process the input video frames.

[0110] Specifically, the detection module 406 is used to obtain a pre-trained target detection model, which may be a YOLO (You Only Look Once) model, an SSD (Single Shot MultiBox Detector) model, a Faster R-CNN model, etc. The target detection model is used to perform target detection on each video frame input from the admission controller to obtain a target detection result.

[0111] The warehouse monitoring device 400 of the embodiment of the present application can integrate video information from multiple perspectives and effectively monitor the status of items in the warehouse by acquiring video streams from multiple cameras in the warehouse; by analyzing the video features of the video stream and allocating a suitable multi-source pre-fetcher to each video stream according to the video features, it is ensured that the video data is optimally distributed in the preprocessing stage, avoiding data accumulation caused by the edge device's untimely data processing, and improving processing efficiency; through the multi-source pre-fetcher, the video stream is synchronously processed according to the video frame processing rate of the target detection model, which can avoid detection delays or errors caused by frame rate mismatch; by using an access controller to manage the access priority of the video frames, and according to the access priority and the video frame processing rate, the video frames are controlled to be input into the target detection model in sequence and speed, which can ensure that more critical video frames can be processed first in a resource-constrained environment, which helps to improve the real-time response speed and monitoring accuracy; by using the target detection model to efficiently process the synchronously input video frames and output accurate target detection results, comprehensive, real-time and accurate monitoring of items in the warehouse is achieved. Based on the solution of this application, resource-constrained edge devices can achieve real-time object detection without relying on high-performance hardware, which not only improves the efficiency and accuracy of warehouse monitoring, but also reduces the cost and risk of manual monitoring.

[0112] In some optional implementations of this embodiment, the allocation module 402 may include a classification marking submodule, a label allocation submodule, and a video allocation submodule.

[0113] A classification and marking submodule, used for classifying and marking the video stream according to the video memory feature in the video feature to obtain a memory tag of the video stream;

[0114] A label allocation submodule, used to obtain load distribution information of each multi-source prefetcher, allocate corresponding memory labels to each multi-source prefetcher according to the load distribution information, and obtain target memory labels allocated to each multi-source prefetcher;

[0115] The video allocation submodule is used to allocate the video stream corresponding to the target memory tag to the multi-source prefetcher corresponding to the target memory tag.

[0116] Specifically, the classification and labeling submodule is used to extract the video memory feature of the video stream according to the video feature of the video stream, and the video memory feature may include the storage space size required for the video stream, the storage space size required for each video frame in the video stream, etc. According to the extracted video memory feature, the video stream is classified and labeled to obtain the memory label of the video stream, and the memory label may include low memory occupancy, medium memory occupancy, high memory occupancy, etc.

[0117] Specifically, the label allocation submodule is used to monitor and record the current load distribution information of each multi-source prefetcher, which may include the number of video streams currently processed, the occupied memory resources, the processing speed, etc. According to the load distribution information of each multi-source prefetcher, the available resources of each multi-source prefetcher are evaluated, and one or more memory labels are allocated based on the available resources of each multi-source prefetcher to obtain the target memory label allocated to each multi-source prefetcher.

[0118] Specifically, the video allocation submodule is used to obtain the video stream corresponding to the target memory tag according to the target memory tag allocated by each multi-source prefetcher, and match the video stream with the multi-source prefetcher corresponding to the target memory tag.

[0119] The allocation module 402 of the embodiment of the present application can more accurately identify the resource requirements of each video stream by classifying and marking the video stream based on video memory features and assigning a memory tag to each video stream; load balancing can be achieved by assigning a target memory tag to each multi-source prefetcher according to the load distribution information of each multi-source prefetcher, thereby avoiding the situation where some multi-source prefetchers are overloaded while other multi-source prefetchers are idle; by allocating the video stream corresponding to the target memory tag to the corresponding multi-source prefetcher, it is ensured that the video stream can be efficiently allocated to a more suitable multi-source prefetcher for processing, thereby reducing data accumulation, avoiding processing delays of video data in the preprocessing stage, and improving real-time performance.

[0120] In some optional implementations of this embodiment, the synchronization module 403 may include a rate acquisition submodule, a rate comparison submodule, and a rate synchronization submodule.

[0121] A rate acquisition submodule, used for inputting the video stream into the multi-source prefetcher, and using the multi-source prefetcher to acquire an input rate of each video frame in the video stream;

[0122] A rate comparison submodule, used for comparing the input rate with the video frame processing rate to obtain a rate comparison result;

[0123] The rate synchronization submodule is used to synchronize the reading time interval of the video frame with the video frame processing rate to obtain synchronized frame data if the rate comparison result is that the input rate is greater than the video frame processing rate.

[0124] Specifically, the rate acquisition submodule is used to input the video stream into the multi-source prefetcher, and use the multi-source prefetcher to monitor and calculate the input rate of each video frame in the input video stream, that is, the number of frames input into the multi-source prefetcher per second. The input rate can also be calculated based on the time interval between the input video frames.

[0125] Specifically, the rate comparison submodule is used to compare the calculated input rate with the video frame processing rate of the target detection model to determine whether there is a rate mismatch problem and obtain a rate comparison result. If the rate comparison result is that the input rate is less than or equal to the video frame processing rate, it indicates that there is no rate mismatch problem; if the rate comparison result is that the input rate is greater than the video frame processing rate, it indicates that there is a rate mismatch problem.

[0126] Specifically, the rate synchronization submodule is used to synchronize the reading time interval of the video frame with the video frame processing rate of the target detection model to obtain synchronized frame data when the rate comparison result is that the input rate is greater than the video frame processing rate, indicating that there is a rate mismatch problem, so that the number of frames provided by the multi-source prefetcher to the target detection model exceeds the processing capacity of the model. For example, if the video frame processing rate of the target detection model can process 30 frames per second, and the input rate reaches 60 frames per second, the reading time interval can be set to twice the time interval between the two frames to ensure that only 30 frames are provided to the target detection model per second.

[0127] The synchronization module 403 of the embodiment of the present application pre-processes the video stream through a multi-source pre-fetcher, and can obtain the input rate of the video frame in real time, and compare the input rate with the video frame processing rate of the target detection model, ensuring the coordination and matching between the key links (input and processing of the video frame) in the video processing flow. When the input rate is higher than the processing rate, by adjusting the reading time interval of the video frame, the input speed of the video frame can be consistent with the processing speed of the target detection model, thereby avoiding the backlog or loss of the video frame caused by the mismatch of the processing speed. In addition, the frame synchronization process can also achieve more efficient computing resource utilization by ensuring that the input of the video frame matches the processing speed, avoiding the waste of resources caused by the mismatch of the processing speed.

[0128] In some optional implementations of this embodiment, the determination module 404 may include a data extraction submodule, a score calculation submodule, and a priority allocation submodule.

[0129] a data extraction submodule, configured to extract each video frame from the synchronized frame data using the admission controller;

[0130] A score calculation submodule, used to calculate the feature hit score of each video frame according to preset important warehouse area features and suspicious target features, wherein the feature hit score is a numerical value indicating the degree to which the important warehouse area features and suspicious target features are contained in the video frame;

[0131] The priority allocation submodule is used to allocate an admission priority to each video frame according to the feature hit score.

[0132] Specifically, important warehouse area features refer to specific attributes of the warehouse itself and its surrounding environment, which may include geographical location, architectural features, security facilities, and cargo types. Suspicious target features refer to features that are found in video surveillance to pose security risks or potential threats, which may include abnormal behavior, appearance features, and items carried.

[0133] Specifically, important warehouse area features and suspicious target features that need to be focused on are pre-set. The data extraction submodule is used to extract each video frame from the synchronized frame data input from the multi-source pre-fetcher using an admission controller; the score calculation submodule is used to detect whether each video frame contains preset important warehouse area features and suspicious target features, and calculate the feature hit score of the video frame according to the degree to which each video frame contains important warehouse area features and suspicious target features. For example, the content of each video frame is identified, and the content of the identified video frame is matched with the preset important warehouse area features and suspicious target features to obtain a feature matching result, which indicates the degree to which the video frame contains important warehouse area features and suspicious target features. The feature hit score of each video frame is calculated according to the feature matching result, and the score is calculated if the content of the video frame hits the important warehouse area features or suspicious target features. The feature hit score indicates the degree to which the video frame contains important warehouse area features and suspicious target features. The priority allocation submodule is used to assign a corresponding admission priority to each video frame according to the feature hit score of each frame of video. For example, the feature hit score of each frame of video is compared with a preset threshold. When the feature hit score is higher than the preset threshold, the video frame corresponding to the feature hit score is assigned an admission priority corresponding to the preset threshold. The admission priority can be divided into low, medium, high, etc.

[0134] The determination module 404 of the embodiment of the present application can quantify the degree of key features contained in the video frame from multiple dimensions and more comprehensively and accurately evaluate the importance and potential threats of the video frame by calculating the feature hit score of each video frame according to the preset important warehouse area features and suspicious target features; by assigning access priority to each video frame according to the feature hit score, it is ensured that high-scoring video frames (i.e., video frames containing important warehouse area features or suspicious target features) can be processed or displayed first. Based on priority allocation, the resources of edge devices can be more effectively utilized, potential security threats can be quickly responded to and processed, and the overall security monitoring efficiency can be improved.

[0135] In some optional implementations of this embodiment, the input module 405 may include a video sorting submodule and a model input submodule.

[0136] A video sorting submodule, used to sort the priority of each video frame according to the admission priority to obtain a queue of video frames to be input;

[0137] The model input submodule is used to input each video frame in the video frame queue into the target detection model in sequence according to the video frame processing rate.

[0138] Specifically, the video sorting submodule is used to arrange each video frame in order of access priority from high to low to obtain a video frame queue to be input. The model input submodule is used to select video frames from the video frame queue in sequence according to the video frame processing rate of the target detection model and input them into the target detection model so that the target detection model can process the input video frames.

[0139] In some optional implementations of this embodiment, the detection module 406 may include an object recognition submodule, an anomaly detection submodule, and a result generation submodule.

[0140] An object recognition submodule, configured to process each of the input video frames using the target detection model to recognize a target object in each of the video frames;

[0141] An anomaly detection submodule, used to obtain a predefined storage anomaly detection rule, and perform anomaly detection on the target object according to the storage anomaly detection rule;

[0142] The result generation submodule is used to generate a target detection result including abnormal prompt information according to the target object if the target object meets the warehouse abnormality detection rule.

[0143] Specifically, the object recognition submodule is used to process each input video frame using the target detection model, such as resizing, color space conversion, normalization, etc., to identify the content contained in each video frame after processing, and obtain the target object in each video frame. The anomaly detection submodule is used to obtain a predefined storage anomaly detection rule, which is set based on the monitoring requirements of the category, quantity, location, motion state, etc. of the objects required for warehouse management. According to the storage anomaly detection rule, the identified target object is detected for anomalies, for example, whether the category of the target object is consistent with the anomaly category defined in the storage anomaly detection rule, whether the number of target objects exceeds or is lower than the threshold set in the storage anomaly detection rule, whether the location of the target object is within the anomaly area defined in the storage anomaly detection rule, and whether the motion state of the target object is consistent with the anomaly mode defined in the storage anomaly detection rule. The result generation submodule is used to determine that the target object is an abnormal object when it meets the storage anomaly detection rule, and generate a target detection result containing abnormal prompt information according to the category, location, quantity, motion state, etc. of the target object.

[0144] Optionally, the result generation submodule is further used to determine that the target object is a normal object when the target object does not conform to the warehouse anomaly detection rule, and to generate a target detection result containing normal prompt information.

[0145] The detection module 406 of the embodiment of the present application realizes intelligent monitoring of the storage environment through the combination of target detection and anomaly detection, which can not only improve the efficiency of storage management, but also reduce the cost of manual monitoring.

[0146] In some optional implementations of this embodiment, the warehouse monitoring device 400 may further include an adjustment module. The adjustment module is used to extract abnormal position information of the target object according to the abnormal prompt information in the target detection result; determine the adjustment parameters of the camera according to the abnormal position information, and control the camera to monitor the target object according to the adjustment parameters.

[0147] Specifically, the adjustment module is used to parse the abnormal prompt information from the target detection result, and extract the abnormal position information of the target object (i.e., the abnormal object) from the abnormal prompt information, and the abnormal position information may include the bounding box coordinates of the target object in the video frame. Determine which cameras the abnormal position information exists within the monitoring range, and determine the camera whose monitoring range contains the abnormal position information as the target camera. Compare the abnormal position information with the field of view of the target camera, and calculate the adjustment parameters (such as direction, angle, zoom, etc.) of the target camera to better monitor the target object. Generate corresponding camera adjustment instructions based on the calculated adjustment parameters, and control the camera to monitor the target object based on the camera adjustment instructions.

[0148] The warehouse monitoring device 400 of the embodiment of the present application extracts abnormal position information from the target detection results, compares the abnormal position with the camera field of view to determine the adjustment parameters, and finally sends adjustment instructions and controls the camera for monitoring, thereby ensuring that the camera can monitor the target object in the abnormal position at a suitable angle and field of view, and continuously detect abnormal situations, thereby improving the intelligence and automation level of warehouse monitoring and providing more accurate and efficient monitoring means for warehouse management.

[0149] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0150] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 6 with a memory 61, a processor 62, and a network interface 63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0151] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0152] The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 61 can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions of the warehouse monitoring method, etc. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.

[0153] The processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run the computer-readable instructions stored in the memory 61 or process data, such as computer-readable instructions for running the warehouse monitoring method.

[0154] The network interface 63 may include a wireless network interface or a wired network interface. The network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0155] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the warehouse monitoring method as described above.

[0156] The computer device, computer-readable storage medium and computer-readable instructions provided in the embodiments of the present application can acquire video streams from multiple cameras in the warehouse through processor execution, integrate video information from multiple perspectives, and effectively monitor the status of items in the warehouse; by analyzing the video features of the video stream and allocating a suitable multi-source pre-fetcher to each video stream according to the video features, it is ensured that the video data is optimally allocated in the pre-processing stage, avoiding data accumulation caused by the edge device's untimely data processing, and improving the processing efficiency; through the multi-source pre-fetcher, the video stream is synchronously processed according to the video frame processing rate of the target detection model, which can avoid detection delays or errors caused by frame rate mismatch; by using an admission controller to manage the admission priority of the video frames, and according to the admission priority and the video frame processing rate, the video frames are controlled to be input into the target detection model in sequence and speed, which can ensure that more critical video frames can be processed first in a resource-constrained environment, which helps to improve the real-time response speed and monitoring accuracy; by using the target detection model to efficiently process the synchronously input video frames and output accurate target detection results, comprehensive, real-time and accurate monitoring of items in the warehouse is achieved. Based on the solution of this application, resource-constrained edge devices can achieve real-time object detection without relying on high-performance hardware, which not only improves the efficiency and accuracy of warehouse monitoring, but also reduces the cost and risk of manual monitoring.

[0157] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0158] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to replace some of the technical features therein with equivalents. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

[0159] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

Claims

1. A warehouse monitoring method, characterized in that: The steps include: Get video streams from multiple cameras in the warehouse; Extracting video features from the video stream, and allocating a multi-source prefetcher corresponding to the video stream according to the video features; Acquire a video frame processing rate of a target detection model, and use the multi-source pre-fetcher to perform frame synchronization processing on the video stream according to the video frame processing rate to obtain synchronized frame data; Inputting the synchronized frame data into an admission controller, and using the admission controller to determine the admission priority of each video frame in the synchronized frame data; Inputting each video frame into the object detection model according to the admission priority and the video frame processing rate; The target detection model is used to perform target detection on each input video frame to obtain a target detection result.

2. The warehouse monitoring method according to claim 1, characterized in that: The step of allocating the multi-source prefetcher corresponding to the video stream according to the video feature comprises: Classifying and marking the video stream according to the video memory feature in the video feature to obtain a memory tag of the video stream; Obtaining load distribution information of each multi-source prefetcher, assigning a corresponding memory tag to each multi-source prefetcher according to the load distribution information, and obtaining a target memory tag assigned to each multi-source prefetcher; The video stream corresponding to the target memory tag is allocated to the multi-source prefetcher corresponding to the target memory tag.

3. The warehouse monitoring method according to claim 1, characterized in that: The step of using the multi-source prefetcher to perform frame synchronization processing on the video stream according to the video frame processing rate to obtain synchronized frame data includes: Inputting the video stream into the multi-source prefetcher, and using the multi-source prefetcher to obtain an input rate of each video frame in the video stream; Comparing the input rate with the video frame processing rate to obtain a rate comparison result; If the rate comparison result is that the input rate is greater than the video frame processing rate, the reading time interval of the video frame is synchronized with the video frame processing rate to obtain synchronized frame data.

4. The warehouse monitoring method according to claim 1, characterized in that: The step of using the admission controller to determine the admission priority of each video frame in the synchronized frame data includes: extracting each video frame from the synchronized frame data using the admission controller; According to the preset important warehouse area features and suspicious target features, a feature hit score of each video frame is calculated, wherein the feature hit score is a numerical value indicating the degree to which the important warehouse area features and suspicious target features are contained in the video frame; An admission priority is assigned to each video frame according to the feature hit score.

5. The warehouse monitoring method according to claim 1, characterized in that: The step of inputting each video frame into the target detection model according to the admission priority and the video frame processing rate comprises: Prioritize each of the video frames according to the admission priority to obtain a queue of video frames to be input; Each video frame in the video frame queue is input into the target detection model in sequence according to the video frame processing rate.

6. The warehouse monitoring method according to claim 1, characterized in that: The step of using the target detection model to perform target detection on each of the input video frames to obtain a target detection result includes: Using the target detection model to process each of the input video frames to identify the target object in each of the video frames; Obtain a predefined storage anomaly detection rule, and perform anomaly detection on the target object according to the storage anomaly detection rule; If the target object meets the warehouse anomaly detection rule, a target detection result including anomaly prompt information is generated according to the target object.

7. The warehouse monitoring method according to claim 6, characterized in that: After the step of using the target detection model to perform target detection on each of the input video frames to obtain a target detection result, the method further includes: Extracting abnormal position information of the target object according to the abnormal prompt information in the target detection result; The adjustment parameters of the camera are determined according to the abnormal position information, and the camera is controlled to monitor the target object according to the adjustment parameters.

8. A storage monitoring device, characterized in that: The device comprises: An acquisition module, used to acquire video streams input from multiple cameras in the warehouse; An allocation module, used for extracting video features from the video stream and allocating a multi-source prefetcher corresponding to the video stream according to the video features; A synchronization module, used for acquiring a video frame processing rate of a target detection model, and performing frame synchronization processing on the video stream according to the video frame processing rate using the multi-source prefetcher to obtain synchronized frame data; A determination module, configured to input the synchronized frame data into an admission controller, and use the admission controller to determine the admission priority of each video frame in the synchronized frame data; An input module, configured to input each video frame into the target detection model according to the admission priority and the video frame processing rate; The detection module is used to use the target detection model to perform target detection on each input video frame to obtain a target detection result.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the warehouse monitoring method as described in any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the warehouse monitoring method according to any one of claims 1 to 7 are implemented.

Images