Dynamic parking space state monitoring method based on fisheye vision and SEEM-YOLO cascade architecture

Abstract

The invention provides a dynamic parking space state monitoring method based on fisheye vision and SEEM-YOLO cascade architecture. The method comprises the following steps: shooting an image in a parking space scene by using a fisheye camera; carrying out distortion correction and denoising processing on the collected image; a YOLOv8 model based on an improved CSPDarknet backbone network is used for extracting multi-scale features of the processed image and detecting the multi-scale features in real time, and a storage location bounding box and a state label are output; a region of interest (ROI) is generated based on an SEEM model, pixel-level mask segmentation is realized through multi-mode prompt, and zero sample learning is supported to adapt to a novel storage location layout or an unknown abnormal target; and the segmentation result and the storage location state information are transmitted to the industrial personal computer, and the industrial personal computer writes the storage location information into a database of a warehouse management system so as to realize dynamic storage location state monitoring. According to the invention, the wide-angle visual advantage of the fisheye camera and the precise recognition capability of the deep learning model are fused, and the real-time precise recognition and dynamic management of the state of the parking space are realized, so that the management efficiency of the parking space is improved, and the space utilization rate is maximized.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and intelligent warehouse automation technology, specifically to a dynamic warehouse location status monitoring method based on fisheye vision and SEEM-YOLO cascade architecture, which is applicable to real-time status identification and management of warehouse locations such as storage racks and parking spaces. Background Technology

[0002] Real-time status identification of storage locations (including various forms such as warehouse racks and parking spaces) is a core component of logistics management, intelligent warehousing, and automated parking systems, and is crucial for improving overall operational efficiency. The accuracy of the identification results directly affects the maximization of space utilization, the efficiency of work processes, and the optimization of equipment scheduling, and is a fundamental prerequisite for achieving intelligent and automated management.

[0003] The current mainstream technical solutions and their limitations are as follows: 1. Fixed sensor solutions (such as geomagnetic / infrared): High hardware cost, susceptible to environmental interference, and unable to identify object types and partially obscured scenes.

[0004] 2. Traditional visual methods (color segmentation, background subtraction): rely on manual marking of library boundaries, have a high false detection rate due to changes in lighting, and have poor adaptability to dynamic environments.

[0005] 3. LiDAR solution: expensive equipment, performance degrades in bad weather, and has insufficient detection capability for low-lying objects.

[0006] Therefore, existing warehouse location status identification technologies struggle to simultaneously address the four key requirements of high accuracy, real-time performance, low cost, and adaptability to dynamic environments. High accuracy demands flawless identification results; real-time performance requires rapid response to changes in warehouse location status; low cost necessitates economic feasibility; and dynamic environmental adaptability demands stable operation under various complex conditions. However, existing solutions often compromise on certain aspects, failing to fully meet these demands. This results in inefficient warehouse location management, limited space utilization, and severely hinders the further development of logistics management, intelligent warehousing, and automated parking systems. Summary of the Invention

[0007] This invention aims to address the challenges of existing technologies in simultaneously achieving high precision, real-time performance, low cost, and adaptability to dynamic environments. It provides a dynamic warehouse location status monitoring method based on a fisheye vision and SEEM-YOLO cascade architecture. By integrating the wide-angle vision advantage of fisheye cameras with the precise recognition capabilities of deep learning models, this method overcomes traditional technological bottlenecks, enabling real-time and accurate identification and dynamic management of warehouse location status. This improves warehouse location management efficiency, maximizes space utilization, and provides strong support for the development of logistics management, intelligent warehousing, and automated parking systems.

[0008] The application achieves the above-mentioned purpose through the following technical solutions: A dynamic storage location state monitoring method based on a fisheye vision and SEEM-YOLO cascade architecture, comprising: An image in a storage location scene is captured by using a fisheye camera; The collected image is subjected to distortion correction and denoising processing; A YOLOv8 model based on an improved CSPDarknet backbone network is used to extract multi-scale features from the processed image and perform real-time detection, and output a storage location bounding box and a state label; A region of interest (ROI) is generated based on a SEEM model, pixel-level mask segmentation is realized through multi-modal prompting, zero-shot learning is supported to adapt to new storage location layouts or unknown abnormal targets; The segmentation result and the storage location state information are transmitted to an industrial computer, and the industrial computer writes the storage location information into a database of a warehouse management system to realize dynamic storage location state monitoring.

[0009] According to the dynamic storage location state monitoring method based on the fisheye vision and the SEEM-YOLO cascade architecture, the image in the storage location scene is captured by using the fisheye camera, and the image in the storage location scene is captured by using the fisheye camera. The internal parameter matrix K and the distortion coefficient D of the fisheye camera are obtained by using a checkerboard calibration method, and the formula is:

[0010] Among them, f x , f y is the focal length, c x , c y is the principal point coordinate, k 1, k 2, k 3 is the radial distortion coefficient, p 1, p 2 is the tangential distortion coefficient; The fisheye camera is horizontally fixed to the ceiling by a supporting rod, and the installation position T and the angle R are obtained by camera two-dimensional code external parameter estimation, and the external parameter matrix is represented as:

[0011] A physical origin O is selected in the storage location scene, a three-dimensional wireframe model of the ROI in the image is established in combination with a storage location CAD planning graph and field measurement data, and wireframe vertex coordinates (X, Y, Z) are mapped to image plane coordinates (u, v) through a camera projection model, and the formula is:

[0012] wherein s is a scale factor; The fisheye camera is connected with the industrial computer through a network cable or WIFI, real-time collection of the warehouse site scene image is transmitted to the industrial computer for processing.

[0013] According to the dynamic warehouse site state monitoring method based on the fisheye vision and the SEEM-YOLO cascade architecture, the collected image is subjected to distortion correction and denoising, and the method comprises the following steps: The distortion coefficient obtained by camera calibration D =[ k 1, k 2, p 1, p 2, k 3] are used to correct the radial and tangential distortion of the original image collected by the fisheye camera I raw The mapping relationship between the corrected image coordinates u ′, v ′ and the original coordinates u , v ) is as follows:

[0014]

[0015] wherein, r 2 = u 2 + v 2 is the square of the distance from the pixel point to the principal point; The corrected coordinates u ′, v ′ are mapped back to the original image space through inverse mapping, the bilinear interpolation is used to calculate the pixel value of the corrected image, and the de-distortion image I undistorted is generated. The de-distortion image I undistorted is subjected to the non-local mean denoising algorithm, the similarity weight between the image blocks is calculated, the pixel value is weighted and averaged, and the denoised image I denoised is output, and the formula is as follows:

[0016] wherein the weight w ( i, j, k, l ) is calculated according to the image blocks N i and Nj Gaussian weighted Euclidean distance determination:

[0017] Z(i,j) is the normalization factor, and h is the smoothing parameter. Representing vectors N i and N j The square of the Euclidean distance between them.

[0018] According to the present invention, a dynamic reservoir status monitoring method based on fisheye vision and SEEM-YOLO cascade architecture is provided. During multi-scale image feature extraction, the improved CSPDarknet backbone network utilizes multiple convolutional layers, residual blocks, and CSP structures to process the denoised image. I denoised Perform feature extraction; In convolution operations, for the input feature map F in After convolution kernel K The convolution operation yields the output feature map. F out The calculation formula is as follows:

[0019] in,( x , y ) represents the coordinates on the output feature map, ( i , j () represents the coordinates on the convolution kernel. By using convolution kernels of different sizes and multiple convolution operations, features at different levels of the image are extracted, including low-level edges, texture features and high-level semantic features. According to the present invention, a dynamic storage location status monitoring method based on fisheye vision and SEEM-YOLO cascade architecture is provided. Multi-scale feature extraction is achieved through Feature Pyramid Network (FPN), which upsamples and fuses feature maps at different levels of the backbone network, enabling the model to utilize feature information at different scales. Assume the different level feature maps output by the backbone network are C 2, C 3, C 4, C 5. After upsampling and lateral connection, the feature map used for detection is obtained. P 2, P 3, P 4, P 5. The fusion process is represented as: P i =Upsample( Pi+1 )+Conv( C i ) wherein Upsample represents an up-sampling operation, and Conv represents a convolution operation, used to adjust the channel number and size of the feature map.

[0020] According to the dynamic warehouse position state monitoring method based on the fish-eye vision and the SEEM-YOLO cascade architecture provided by the application, the YOLOv8 model detects the warehouse position target through the preset anchor frame on the extracted multi-scale feature map. For each grid unit on each feature map, the model predicts the offset of the anchor frame (tx, ty, tw, th) and the confidence (confidence) and category probability (class probability) of the target. The predicted offset is converted into the actual boundary box coordinates (x, y, w, h) through the following formula: t x , t y , t w , t h t o b x , b y , b w , b h b x = σ ( t x )+ c x b y = σ ( t y )+ c y b w = p w × e tw b h = p h × e th wherein ( c x , c y ​​​) is the upper left corner coordinate of the grid unit, p w and p h is the width and height of the anchor frame, σ represents the sigmoid function, which is used to map the predicted value to 0-1.

[0021] According to the dynamic warehouse position state monitoring method based on fisheye vision and SEEM-YOLO cascade architecture provided by the application, the state label of the warehouse position is predicted by the model, the class probability is normalized by using the softmax function, and the probability value of each class is obtained, and the calculation formula of the softmax function is:

[0022] Among them, z i is the logical value of the first i class output by the model, S i is the probability value of the first i class; According to the predicted boundary box coordinates and the probability value of the state label, the detection result with a confidence higher than a set threshold is screened out, and the warehouse position boundary box and the corresponding state label are output, at least including idle, occupied or abnormal.

[0023] According to the dynamic warehouse position state monitoring method based on fisheye vision and SEEM-YOLO cascade architecture provided by the application, when the region of interest ROI is generated based on the SEEM model, the SEEM model receives prompt information of multiple modalities such as text, image and boundary box, and maps it to a unified feature space through an encoder: Text prompt encoding: a pre-trained language model is used to encode the text instruction into a text feature vector t∈R dt , and the formula is: t=CLIP text ( T ), T ={“idle”,“occupied”,“abnormal goods”} Image prompt encoding: the visual features v∈R H×W×dv of the input image I are extracted, and multi-scale features are obtained through a convolutional neural network, and the formula is: v=CNN( I ),v i =ResNet i ( I ), i ∈{1,2,3} Boundary box prompt encoding: the warehouse position boundary boxB =( x , y , w , h Transform into spatial features b∈R 4 ×db Spatial information is embedded through location encoding, and the formula is as follows:

[0024] Here, PE indicates that the bounding box B is position-encoded. i It is an index variable used to generate positional codes in different dimensions. d i It is the dimension of the location encoding.

[0025] According to the present invention, a dynamic storage location status monitoring method based on fisheye vision and SEEM-YOLO cascade architecture is provided. The SEEM model fuses multimodal features through a cross-attention mechanism to generate a ROI mask related to the prompt: Using textual features t as the query and visual features v as the key and value, the attention weights are calculated using the following formula:

[0026] Among them, W q W k W v Let M be the projection matrix, and M ∈ R. dt×dv Let A be the fused features, and A be the attention weight matrix. d k It is the dimension of the key vector; The fused features M are upsampled to the original image resolution, and a pixel-level mask S∈[0, 1] is generated using the Sigmoid function. H × W The formula is:

[0027] in, To output the feature map at position ( i , j The pixel value at () Let Conv(M) be the activation function, representing the convolution operation performed on the input feature map M. z It is the input value of the Sigmoid function.

[0028] According to the present invention, a dynamic storage location status monitoring method based on fisheye vision and SEEM-YOLO cascade architecture is provided. For unknown targets, the user inputs descriptive text, and the SEEM model generates corresponding features t through a language encoder. new And match it with the existing feature library, the formula is:

[0029] wherein, is an encoder, is a text feature vector t new between t i , and t is the i-th vector in the existing set of text feature vectors; According to the similarity score Sim, the existing mask is fused to generate a segmentation result for the new target, and the formula is:

[0030] wherein, is a newly generated mask or feature map, is a weight coefficient, is an existing mask or feature map, is the j-th vector in the existing set of text feature vectors, and Sim represents the similarity between vectors; Morphological operation is performed on the generated mask S to remove noise, and the formula is: S final =Morph(S,kernel=3x3) According to the mask S final , the pixel coordinates of the storage location area are extracted to generate an ROI bounding box B ROI , and the formula is: B ROI =BoundingBox(S final >0.5) The ROI bounding box B ROI and the state label are transmitted to the industrial computer to update the WMS database.

[0031] Therefore, compared with the prior art, the dynamic storage location state monitoring method based on the fisheye vision and the SEEM-YOLO cascade architecture has the following beneficial effects: 1. The YOLOv8 detection model based on the improved CSPDarknet backbone network is adopted in the present application, high-precision detection of the storage location boundary is realized, the model can extract multi-scale features including key information such as boundary, texture, shape, etc., and the positioning accuracy of the storage location in a complex scene (such as light change and partial occlusion) is significantly improved.

[0032] 2、By outputting accurate bin boundary boxes and state labels (free / occupied / abnormal), the application can more accurately distinguish between goods and abnormal targets, thereby greatly reducing the probability of false positives and false negatives. In addition, in combination with the ROI (Region of Interest) region generated by the SEEM model, pixel-level mask segmentation is realized through text prompts and detection box guidance, further improving the segmentation accuracy in complex scenes and ensuring the accuracy of the identification results.

[0033] 3、The application is not only limited to identifying the status of conventional bins, but also has the ability to predict abnormal targets through advanced algorithms and feature extraction techniques. It can analyze the characteristics of objects within the bin and identify potential abnormalities such as stacking beyond limits, misplacement, etc., providing timely warning information for warehouse management personnel, making warehouse management more intelligent, and helping to discover and solve problems in advance, avoiding potential safety hazards and operational risks.

[0034] 4、The application can easily adapt to different dynamic environments, whether it is changes in lighting conditions, partial occlusion of objects, or adjustments to bin layouts. Through automatic adjustment and optimization algorithms, it maintains efficient bin management capabilities, ensuring the stability and reliability of the system in various actual scenarios, providing a strong guarantee for the continuity and efficiency of warehouse management.

[0035] 5、The implementation of the application only requires the installation of a single fisheye camera and a network cable, which can cover multiple bins for real-time monitoring. This simple deployment method not only reduces hardware costs but also simplifies the construction process. If the bin size or number needs to be changed in the future, as long as it is within the camera coverage range, only program settings are required to complete the adjustment without additional construction work, making bin management more convenient and efficient, reducing the operational and time costs of enterprises.

[0036] The application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0037] Fig. 1 is a flowchart of an embodiment of a dynamic bin status monitoring method based on fisheye vision and SEEM-YOLO cascade architecture.

[0038] Fig. 2 is a flowchart of an embodiment of a dynamic bin status monitoring method based on fisheye vision and SEEM-YOLO cascade architecture.

[0039] Fig. 3 is a camera deployment flowchart of an embodiment of a dynamic bin status monitoring method based on fisheye vision and SEEM-YOLO cascade architecture.

[0040] Fig. 4 is a schematic diagram of an embodiment of a dynamic storage space state monitoring system based on fisheye vision and a SEEM-YOLO cascade architecture. DETAILED DESCRIPTION

[0041] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0042] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearance of the phrase in various places in the specification indicates that the described feature, structure, or characteristic can be included in at least one embodiment of the application. It is explicitly and implicitly understood that the embodiments described herein can be combined.

[0043] Referring to Figs. 1 to 3 , the present embodiment provides a dynamic storage space state monitoring method based on fisheye vision and a SEEM-YOLO cascade architecture, comprising: Step S1, using a fisheye camera to shoot an image in a storage space scene; Step S2, performing distortion correction and denoising processing on the collected image; Step S3, using a YOLOv8 model based on an improved CSPDarknet backbone network to extract multi-scale features from the processed image and perform real-time detection, outputting a storage space bounding box and a state label; Step S4, generating a region of interest (ROI) based on a SEEM model, realizing pixel-level mask segmentation through multi-modal prompting, and supporting zero-shot learning to adapt to new storage space layouts or unknown abnormal targets; Step S5, transmitting the segmentation result and the storage space state information to an industrial computer, and writing the storage space information into a database of a warehouse management system by the industrial computer to realize dynamic storage space state monitoring.

[0044] In the above step S1, the fisheye camera is used to shoot an image in a storage space scene, comprising: The chessboard calibration method is used to obtain the intrinsic matrix K and the distortion coefficient D of the fisheye camera, and the formula is:

[0045] wherein, f x , fy focal length, c x , c y principal point coordinates, k 1, k 2, k 3 radial distortion coefficients, p 1, p 2 tangential distortion coefficients; The fisheye camera is fixed horizontally on the ceiling by a support rod, and the installation position and angle are obtained by camera two-dimensional code external parameter estimation T and angle R The external parameter matrix is represented as:

[0046] In the warehouse site scene, a physical origin O is selected, and a three-dimensional wireframe model of the ROI in the image is established in combination with the warehouse site CAD planning map and the field measurement data. The wireframe vertex coordinates (X, Y, Z) are mapped to the image plane coordinates (u, v) through the camera projection model, and the formula is:

[0047] Wherein, s is a scale factor; The fisheye camera is connected with the industrial computer through a network cable or WIFI, and real-time warehouse site image is collected and transmitted to the industrial computer for processing.

[0048] In the above step S2, the collected image is subjected to distortion correction and denoising, including: The distortion coefficients obtained by camera calibration D [ k 1, k 2, p 1, p 2, k 3] are used to correct the radial and tangential distortion of the original image I raw collected by the fisheye camera, and the mapping relationship between the corrected image coordinates u ′, v ′ and the original coordinates u , v is:

[0049]

[0050] Wherein, r 2 u 2 + v 2 ​The square of the distance from the pixel point to the principal point; The corrected coordinates are mapped back to the original image space by inverse mapping, and the corrected pixel values are calculated by bilinear interpolation to generate a de-distorted image u , v ′) are mapped back to the original image space by inverse mapping, and the corrected pixel values are calculated by bilinear interpolation to generate a de-distorted image I undistorted ; The de-distorted image I undistorted is denoised by using a non-local mean denoising algorithm, which calculates the similarity weight between image blocks and performs weighted average on the pixel values to output the denoised image I denoised , the formula is:

[0051] Among them, the weight w ( i, j, k, l ) is determined by the Gaussian weighted Euclidean distance of the image blocks N i and N j :

[0052] Z(i,j) is a normalization factor, h is a smoothing parameter, represents the Euclidean distance between the vectors N i and N j .

[0053] When performing image multi-scale feature extraction, the improved CSPDarknet backbone network extracts features from the denoised image I denoised through multiple convolution layers, residual blocks and CSP structure; In the convolution operation, for the input feature map F in , the output feature map K F out is obtained after convolution operation of the convolution kernel , and the calculation formula is:

[0054] Among them, ( x , y ) represents the coordinates on the output feature map, ( i , j ) represents the coordinates on the convolution kernel, and different sizes of convolution kernel and multiple convolution operations are used to extract different levels of features of the image, including low-level edge, texture features and high-level semantic features; Multi-scale feature extraction is achieved through structures such as feature pyramid network (FPN) or path aggregation network (PAN). Taking FPN as an example, it up-samples and fuses feature maps of different levels of the backbone network, so that the model can utilize feature information of different scales; Suppose the feature maps output by the backbone network are C 2, C 3, C 4, C 5, after up-sampling and horizontal connection, the feature maps for detection are obtained P 2, P 3, P 4, P 5, the fusion process is represented as: P i =Upsample( P i+1 )+Conv( C i ) Where Upsample represents the up-sampling operation, and Conv represents the convolution operation, which is used to adjust the channel number and size of the feature map.

[0055] On the extracted multi-scale feature maps, the YOLOv8 model detects the target in the warehouse position through preset anchor boxes. For each grid cell on each feature map, the model predicts the anchor box offset ( t x , t y , t w , t h ) and the confidence t o and class probability of the target. The predicted offset is converted into the actual bounding box coordinates ( b x , b y , b w , b h ) through the following formula: b x = σ ( t x )+ c x b y = σ ( ty + c y b w = p w × e tw b h = p h × e th where, c x , c y is the top-left coordinate of the grid cell, p w and p h are the width and height of the anchor box, σ represents the sigmoid function, which is used to map the predicted value to 0-1.

[0056] The state label of the storage location is predicted by the model, and the softmax function is used to normalize the probability of the class (free, occupied, abnormal) to obtain the probability value of each class. The formula for calculating the softmax function is:

[0057] where, z i is the logical value of the i th class output by the model, S i is the probability value of the i th class; According to the predicted bounding box coordinates and the probability value of the state label, the detection results with a confidence higher than the set threshold are selected, and the bounding box of the storage location and the corresponding state label are output, including at least free, occupied or abnormal.

[0058] When generating the region of interest (ROI) based on the SEEM model, the SEEM model receives multiple modal prompt information such as text, image, and bounding box, and maps it to a unified feature space through an encoder: Text prompt encoding: use a pre-trained language model to encode the text instruction (such as "split the stacked over-limit goods") into a text feature vector t∈R dt , the formula is: t=CLIP text ( T ), T={“free”,“occupied”,“abnormal goods”} Image prompt encoding: encode the input image I Extract visual features v∈R H×W×dv , obtain multi-scale features through a convolutional neural network, and the formula is: v=CNN( I ),v i =ResNet i ( I ), i ∈{1,2,3} Boundary box prompt encoding: convert the warehouse location boundary box output by YOLOv8 B =( x , y , w , h ) into spatial features b∈R 4 ×db , embed spatial information through position encoding, and the formula is:

[0059] where PE represents position encoding of the boundary box B, i is an index variable for generating position encodings of different dimensions, d i is the dimension of the position encoding.

[0060] The SEEM model fuses multi-modal features through cross-attention mechanism to generate ROI masks related to the prompt: Text features t are used as queries, and visual features v are used as keys and values to calculate attention weights, and the formula is:

[0061] where W q ,W k ,W v are projection matrices, M∈R dt×dv is the fused feature, A is the attention weight matrix, d k is the dimension of the key vector; Upsample the fused feature M to the original image resolution, and generate a pixel-level mask S∈[0, H × W , and the formula is:

[0062] where, is the pixel value of the output feature map at position i , j ​For the activation function, Conv(M) represents the convolution operation on the input feature map M, z is the input value of the Sigmoid function.

[0063] For unknown targets (such as new goods), the user inputs a description text (such as "irregularly stacked objects"), and the SEEM model generates corresponding features t new by a language encoder, and matches them with existing feature libraries, with the formula:

[0064] wherein, is the encoder, is the similarity between the text feature vector t new and t i , and is the i-th vector in the existing set of text feature vectors; According to the similarity score Sim, the existing masks are fused to generate a segmentation result for the new target, with the formula:

[0065] wherein, is the newly generated mask or feature map, is the weight coefficient, is the existing mask or feature map, is the j-th vector in the existing set of text feature vectors, and Sim represents the similarity between the vectors; Morphological operations are performed on the generated mask S to remove noise, with the formula: S final = Morph(S, kernel = 3 x 3) According to the mask S final , the pixel coordinates of the storage location area are extracted to generate the ROI bounding box B ROI , with the formula: B ROI = BoundingBox(S final > 0.5) The ROI bounding box B ROI and the state label are transmitted to the industrial computer to update the WMS database.

[0066] In this embodiment, the YOLOv8 model is optimized in the following ways: TensorRT model quantization technology is used to convert the YOLOv8 model into TensorRT format, achieving efficient inference and optimization of the model, improving inference speed and reducing memory occupancy.

[0067] Specifically, first, the trained YOLOv8 detection model and the SEEM segmentation model are preprocessed to extract the model weight file and network structure information; using the model conversion tool provided by TensorRT (such as trtexec or programming interface), the weight file and network structure of the YOLOv8 and SEEM model are converted from the original framework (such as PyTorch, ONNX) to the.engine format supported by the TensorRT engine, which is deeply optimized for edge devices (such as NVIDIA Jetson series).

[0068] During the conversion process, according to the computing resources and real-time requirements of the edge device, the quantization precision parameters of TensorRT are configured, including FP32 (full precision), FP16 (half precision), or INT8 (8-bit integer) quantization; using dynamic range quantization or symmetric quantization strategy, the model calculation complexity and memory occupation are reduced as much as possible under the premise of ensuring the model inference accuracy.

[0069] Then, based on the converted model file, the optimized inference engine is generated using the TensorRT engine builder. During the engine building process, optimization techniques such as layer fusion and kernel automatic tuning are automatically applied to maximize the computing efficiency of the edge device; for the multi-scale feature extraction characteristics of the YOLOv8 model, the dynamic shape input support of TensorRT is configured to ensure that the engine can process input images of different resolutions.

[0070] The memory management mechanism of TensorRT is integrated in the inference engine to reduce memory occupation during model inference by reusing memory buffers and reducing memory fragmentation; the dynamic batch processing function is enabled to dynamically adjust the batch size of input images according to the real-time load of the edge device to balance the inference delay and throughput.

[0071] Finally, the optimized TensorRT engine is deployed to the edge device (such as NVIDIA Jetson AGXXavier) and integrated into the library position state monitoring system through the edge computing framework (such as ROS 2, GStreamer).

[0072] During real-time inference, the efficient execution capability of the TensorRT engine is utilized in combination with real-time image data collected by the fisheye camera to realize rapid recognition and state updating of the library position state, meeting the real-time requirements of logistics management, intelligent warehousing, and other scenarios.

[0073] As shown in Fig. 4 The embodiment also provides a dynamic library position state monitoring system based on a fisheye vision and SEEM-YOLO cascade architecture, which comprises: Fish-eye camera: configured at the ceiling position of the scene of the to-be-monitored storage location, fixedly installed horizontally through a support rod, ensuring that the field of view after distortion correction can cover all target storage location areas without obstruction; connected with a switch through a wired network (network cable), establishing a high-speed data transmission channel with an industrial computer, realizing real-time image data acquisition and transmission; Industrial computer: deployed in a machine room or control console, serving as the core control unit of the storage location identification system; integrated with a wired network (network cable) or wireless network (WIFI) interface, maintaining network intercommunication with the fish-eye camera, responsible for receiving real-time image data collected by the fish-eye camera and writing inference results into the WMS (warehouse management system) database; running the storage location area segmentation module and the storage location inference module to realize image data processing and real-time identification of storage location status; Storage location area segmentation module: embedded in the industrial computer in the form of a software module, realizing pixel-level segmentation of each storage location in the scene based on the SEEM (Segment Everything Everywhere All at Once) model; receiving image data transmitted by the fish-eye camera, generating ROI (Region of Interest) area masks containing the boundaries of each storage location, and feeding back the segmentation results to the industrial computer; supporting multi-modal prompt input (such as the text instruction "segment stacked over-limit goods"), realizing adaptation of new storage location layout or unknown abnormal targets under zero-shot learning; Storage location inference module: embedded in the industrial computer in the form of a software module, realizing real-time detection of storage location status based on the YOLOv8 (You Only Look Once: Unified, Real-Time Object Detection) model; receiving the ROI area mask output by the storage location area segmentation module, combining the improved CSPDarknet backbone network to extract multi-scale features (boundary, texture, shape), and inferring the state label (idle / occupied / abnormal) of each storage location; optimizing inference efficiency and reducing memory occupation through TensorRT model quantization technology, meeting the real-time processing needs of edge devices; Specifically, the fish-eye camera, the industrial computer, the storage location area segmentation module, and the storage location inference module establish a data interaction link through a wired / wireless network, forming a closed-loop storage location status monitoring process; the industrial computer coordinates the operation of each module, realizing fully automated processing from image acquisition, storage location segmentation, state inference to result storage, supporting dynamic storage location layout adjustment and abnormal state early warning.

[0074] In actual application, the dynamic storage location status monitoring method based on the fish-eye vision and SEEM-YOLO cascade architecture provided by the embodiment further includes: Camera process initialization phase The asynchronous camera acquisition process is started, the parameter configuration of the fisheye camera is completed, including resolution, frame rate and exposure time, a real-time data transmission channel with the industrial computer is established, and the multi-thread scheduling framework is initialized to allocate independent thread resources for subsequent image preprocessing, model inference and data synchronization tasks; System resource initialization phase Load the pre-trained bin segmentation model (SEEM architecture) and goods state detection model (YOLOv8 architecture), implement model quantization and hardware acceleration deployment through the TensorRT engine; Configure the system log module and define different levels of log recording rules; Initialize the database connection pool and establish data interaction interfaces with the WMS system; Real-time image processing phase Image acquisition: Read the latest frame image data from the asynchronous camera process buffer, support dynamic resolution adjustment to adapt to network bandwidth fluctuations; Preprocessing: Perform radial distortion correction on fisheye images, use bilateral filtering algorithm to remove noise, and enhance contrast through histogram equalization; Bin segmentation: Input the preprocessed image into the SEEM model to generate a pixel-level bin area mask, and optimize the segmentation boundary through morphological operations; State inference: Based on the YOLOv8 model, detect the target in each bin area and output the goods state label (idle / occupied / abnormal) and confidence score; Visualization and data synchronization phase State visualization: According to the inference result, superimpose semantic annotations on the original image, use color coding (green / red / yellow) to distinguish different bin states, and generate a monitoring picture with state indication; Data persistence: Write bin state information (timestamp, position coordinates, state label) into a time series database, trigger the WMS system interface to update the static web page display; Exception feedback: When detecting that the goods are stacked out of limits or the position is offset, generate an alarm event and push it to the management terminal through the MQTT protocol.

[0075] Further, the SEEM model supports multi-modal prompt input, which can dynamically adjust the segmentation target through text instructions (such as "only segment dangerous goods bins"); Further, the YOLOv8 model uses a dynamic anchor box matching strategy to adapt to the detection needs of goods of different sizes; Further, the database synchronization mechanism includes a conflict detection module that automatically triggers an optimistic lock retry mechanism when multiple threads write concurrently.

[0076] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, any combination of the technical features is considered to be within the scope of the present disclosure.

[0077] The above embodiments are merely preferred embodiments of the present application, and cannot be used to limit the protection scope of the present application, and any non-essential changes and replacements made by those skilled in the art based on the present application shall fall within the protection scope of the present application.

Claims

1. A dynamic warehouse location state monitoring method based on a fish-eye vision and SEEM-YOLO cascade architecture, characterized in that, The method comprises the following steps: taking an image in a warehouse location scene by using a fisheye camera; performing distortion correction and denoising on the collected image; extracting multi-scale features from the processed image by using a YOLOv8 model based on an improved CSPDarknet backbone network and performing real-time detection, and outputting a warehouse location bounding box and a state label; generating a region of interest (ROI) based on a SEEM model, implementing pixel-level mask segmentation through multi-modal prompts, supporting zero-shot learning to adapt to new warehouse location layouts or unknown abnormal targets; transmitting the segmentation result and the warehouse location state information to an industrial computer, and writing the warehouse location information into a database of a warehouse management system by the industrial computer to realize dynamic warehouse location state monitoring.

2. The method of claim 1, wherein, The method of taking an image in a warehouse location scene by using a fisheye camera comprises the following steps: acquiring an intrinsic matrix K and a distortion coefficient D of the fisheye camera by using a checkerboard calibration method, and the formula is: wherein f x , f y is the focal length, c x , c y is the principal point coordinate, k 1, k 2, k 3 is a radial distortion coefficient, p 1, p 2 is a tangential distortion coefficient; The fisheye camera is fixed horizontally on the ceiling through a support rod, and an installation position is obtained through a camera two-dimensional code external parameter estimation T and angle R The external parameter matrix is represented as: selecting a physical origin O in the warehouse location scene, combining a warehouse location CAD planning graph and field measurement data, and establishing a three-dimensional wireframe model of the ROI in the image, and mapping the wireframe vertex coordinates (X, Y, Z) to the image plane coordinates (u, v) through a camera projection model, and the formula is: where s is a scale factor; connecting the fisheye camera with the industrial computer through a network cable or WIFI, collecting the warehouse location scene image in real time, and transmitting the image to the industrial computer for processing.

3. The method of claim 2, wherein, The method of performing distortion correction and denoising on the collected image comprises the following steps: Distortion coefficients obtained by camera calibration D [ k 1, k 2, p 1, p 2, k 3],to the raw image collected by the fisheye camera I raw radial and tangential distortion correction, the mapping relationship between the corrected image coordinates u ′, v ′) and the original coordinates u , v ) is: wherein, r 2 = u 2 + v 2 is the square of the distance of the pixel point to the principal point; The corrected coordinates are obtained through inverse mapping. u ′, v The image is mapped back to the original image space, and the corrected pixel values ​​are calculated using bilinear interpolation to generate a distortion-free image. I undistorted ; To the deformed image I undistorted The non-local mean denoising algorithm is adopted, pixel values are weighted and averaged through calculating similarity weights between image blocks, and a denoised image is output I denoised The formula is: wherein the weights w ( i, j, k, l ) are determined by the Gaussian-weighted Euclidean distance of the image blocks N i and N j ​ Z(i,j) is a normalization factor, h is a smoothing parameter, denotes the vector N i and N j the squared Euclidean distance between 4. The method of claim 1, wherein: In the image multi-scale feature extraction, the improved CSPDarknet backbone network extracts features from the denoised image through multiple convolutional layers, residual blocks and CSP structure I denoised extracts features In the convolution operation, for the input feature map F in , the output feature map F out is obtained through the convolution operation of the convolution kernel K , and the calculation formula is: wherein, x , y ) represents the coordinates on the output feature map, and ( i , j ) represents the coordinates on the convolution kernel. Through convolution kernels of different sizes and multi-layer convolution operations, different levels of features of the image are extracted, including low-level edge, texture features and high-level semantic features.

5. The method of claim 4, wherein: Multi-scale feature extraction is realized by a feature pyramid network (FPN), which up-samples and fuses feature maps of different levels of the backbone network, so that the model can utilize feature information of different scales. Assume the different level feature maps output by the backbone network are C 2, C 3, C 4, C 5, after upsampling and horizontal connection, the feature maps for detection are obtained P 2, P 3, P 4, P 5, the fusion process is represented as: P i = Upsample( P i+1 )+Conv( C i ) where Upsample represents an up-sampling operation, and Conv represents a convolution operation, which is used to adjust the channel number and size of the feature map.

6. The method of claim 5, wherein: On the extracted multi-scale feature map, the YOLOv8 model detects the target of the storage location through a preset anchor box library. For each grid unit on each feature map, the model predicts the offset of the anchor box t x , t y , t w , t h ) and the confidence t o and class probability of the target. The predicted offset is converted into the actual boundary box coordinates b x b y b w b h by the following formula:​​​ b x = σ ( t x )+ c x b y = σ ( t y )+ c y b w = p w × e tw b h = p h × e th where, c x , c y is the top-left corner coordinate of the grid cell, p w and p h is the width and height of the anchor box, σ denotes the sigmoid function, which is used to map the predicted value to between 0 - 1.

7. The method of claim 6, wherein: The model simultaneously predicts the state label of the warehouse location, normalizes the class probability by using a softmax function, and obtains the probability value of each class, and the calculation formula of the softmax function is: wherein, z i is a logical value of the i th class of the model output, S i is a probability value of the i th class. According to the predicted bounding box coordinates and the probability value of the state label, the detection result with a confidence higher than a set threshold is screened out, and the warehouse location bounding box and the corresponding state label are output, at least including idle, occupied or abnormal.

8. The method of any one of claims 1 to 7, wherein: When generating the region of interest (ROI) based on the SEEM model, the SEEM model receives prompt information of multiple modalities such as text, image and bounding box, and maps it to a unified feature space through an encoder: Text prompt encoding: encode the text instruction into a text feature vector t ∈ R using a pre-trained language model dt , where the formula is: t = CLIP text ( T ), T ={"idle", "occupied", "abnormal goods"} Image prompt encoding: on the input image I Extract visual features v e R H×W×dv Multi-scale features are obtained through a convolutional neural network, and the formula is: v = CNN I ), v i = ResNet i ( I ), i ∈ {1,2,3} Bounding box hint encoding: Encoding the library location bounding boxes output by YOLOv8. B =( x , y , w , h Transform into spatial features b∈R 4×db Spatial information is embedded through location encoding, and the formula is as follows: wherein PE denotes a positional encoding of the bounding box B, i is an index variable for generating positional encodings of different dimensions, d i is the dimension of the positional encoding.

9. The method of claim 8, wherein: The SEEM model fuses multi-modal features through a cross-attention mechanism to generate an ROI mask related to the prompt: The text feature t is taken as a query, and the visual feature v is taken as a key and a value, an attention weight is calculated, and the formula is: where W q ,W k ,W v is a projection matrix, M e R dt×dv is the fused feature, A is the attention weight matrix, d k is the dimension of the key vector; The fusion feature M is up-sampled to the original image resolution, and a pixel-level mask S ∈ [0, H × W , and the formula is: wherein, is a pixel value of the output feature map at a position ( i , j ), is an activation function, and Conv(M) denotes a convolution operation on an input feature map M, z is an input value of a Sigmoid function.

10. The method of claim 9, wherein: For unknown targets, the user inputs a description text, and the SEEM model generates corresponding features t through a language encoder new and matches them with the existing feature library, formula: wherein, is an encoder, is a text feature vector t new is a similarity between t i and t is the i-th vector in a set of existing text feature vectors; According to the similarity score Sim, the existing mask is fused to generate a segmentation result for the new target, and the formula is: wherein, is a newly generated mask or feature map, is a weight coefficient, is an existing mask or feature map, is the jth vector in the existing set of text feature vectors, and Sim denotes the similarity between vectors. The generated mask S is subjected to morphological operation to remove noise, and the formula is: S final = Morph(S, kernel=3x3) According to the mask S final The pixel coordinates of the storage location area are extracted to generate an ROI bounding box B ROI The formula is: B ROI = BoundingBox(S final > 0.5) Boundary box of ROI B ROI The status tags are transmitted to the industrial control computer to update the WMS database.

Images