Wrong-pick-and-place warning method and system for drone parts warehouse

Abstract

The present invention provides a method and system for warning of wrong taking or wrong placing in a drone parts warehouse, which relates to the field of intelligent warehousing technology, including acquiring real-time operation image data through an image acquisition device, collecting parts position data using a distributed sensor network, and generating a parts operation data stream through standardized processing; a deep learning recognition model identifies the operation behavior type and parts feature information, and calculates the real-time spatial position coordinates of the parts in combination with a three-dimensional spatial positioning algorithm; an early warning analysis model determines the operation intention, matches the parts information in real time, compares the three-dimensional virtual model of the warehouse, and pushes warning information and visually marks it when an erroneous operation is detected.

Description

Technical Field

[0001] The present invention relates to intelligent warehousing technology, and in particular to a method and system for early warning of wrong taking and wrong placing of parts in a warehouse for unmanned aerial vehicles. Background Art

[0002] Traditional warehouse management methods for drone parts mainly rely on manual operations and simple barcode systems, which are difficult to meet the needs of efficient and accurate management of modern drone parts warehouses. The existing warehouse management system has the following problems: First, the existing system lacks real-time and accurate operation monitoring capabilities. Traditional methods cannot capture the specific operation behaviors of operators in a timely manner, resulting in problems such as wrong placement being difficult to detect and correct in a timely manner, affecting warehouse management efficiency and parts integrity.

[0003] Secondly, it is difficult to accurately locate and identify accessories with existing technologies. Most systems rely only on simple barcode or RFID technology, which cannot accurately obtain the three-dimensional spatial location information and detailed features of accessories, increasing the difficulty of inventory counting and finding accessories.

[0004] Finally, the existing early warning mechanism is relatively backward and single. Most systems can only detect problems after the fact through inventory counting and other methods. They lack real-time early warning capabilities, cannot remind operators as soon as errors occur, and cannot intuitively display the specific location and related information of erroneous operations. Summary of the invention

[0005] The embodiments of the present invention provide a method and system for early warning of wrong taking and wrong placing of parts in a warehouse for unmanned aerial vehicle parts, which can solve the problems in the prior art.

[0006] According to a first aspect of an embodiment of the present invention, a wrong-taking and wrong-placing warning method applicable to a drone parts warehouse is provided, comprising: acquiring real-time operation image data of an operator through an image acquisition device deployed in the warehouse, acquiring the position data of the parts through a distributed sensor network, standardizing the collected real-time operation image data and position data, and generating a parts operation data stream; inputting the parts operation data stream into a deep learning recognition model, identifying the operation behavior type of the operator and the feature information of the parts, calculating the real-time spatial position coordinates of the parts by using a three-dimensional spatial positioning algorithm combined with a multi-sensor data fusion technology, and inputting the identified operation behavior type, the feature information of the parts, and the real-time spatial position coordinates of the parts into a warning analysis model; the warning analysis model determines the operation intention of the operator based on the identified operation behavior type, matches the feature information of the parts with a preset parts information database in real time, and maps and compares the real-time spatial position coordinates of the parts with a three-dimensional virtual model of the warehouse in real time; when it is detected that the operator's part picking behavior or placement behavior does not conform to the preset cargo location rules in the three-dimensional virtual model of the warehouse, a warning message is pushed to the operator, and the specific location and part information of the wrong operation are marked in a three-dimensional visualization interface.

[0007] In an optional embodiment, real-time operation image data of operators are acquired through an image acquisition device deployed in a warehouse, location data of accessories are acquired through a distributed sensor network, and the acquired real-time operation image data and location data are standardized. The steps of generating an accessory operation data stream include: arranging high-definition camera arrays on the top and sides of the shelf to acquire real-time operation image data of operators, triggering a high frame rate acquisition mode and performing target tracking when an operator is detected entering the operation area; arranging UWB positioning base stations at the four corners of the shelf to acquire the location data of accessories using a time difference positioning algorithm; performing distortion correction, illumination compensation and image enhancement processing on the acquired real-time operation image data, and performing Kalman filtering on the location data; segmenting the processed real-time operation image data and location data using a sliding window method, performing time alignment on the segmented data based on a network time protocol, and generating an accessory operation data stream containing timestamps, image data and location data after wavelet denoising and standardization.

[0008] In an optional embodiment, the accessory operation data stream is input into a deep learning recognition model to identify the operator's operation behavior type and accessory feature information, and the steps of using a three-dimensional spatial positioning algorithm combined with multi-sensor data fusion technology to calculate the real-time spatial position coordinates of the accessory include: inputting the accessory operation data stream into a deep learning recognition model, the deep learning recognition model includes a two-stream network structure, the RGB image branch of the two-stream network structure uses a three-dimensional convolutional residual network to extract image features, and the skeleton sequence branch uses a spatiotemporal graph convolutional network to extract human skeleton features, the attention weight matrix is ​​calculated through the spatiotemporal attention module, and the features of the RGB image branch and the skeleton sequence branch are dynamically fused to obtain spatiotemporal fusion features, and the spatiotemporal fusion features are input into a temporal modeling network The long-range dependency is extracted to obtain the operation behavior type; the deep learning recognition model also includes a multi-scale feature pyramid network, which fuses the convolutional features with the upsampled high-level features at different scale levels to obtain multi-scale fusion features, applies the attention mechanism to the multi-scale fusion features to generate an attention map, and extracts the accessory instance-level feature vector through the instance alignment layer to obtain the accessory feature information; the image data in the accessory operation data stream is processed through the deep learning recognition model to obtain the visual positioning result, and the ultra-wideband ranging value and the radio frequency signal strength index are obtained through the distributed sensor network; the three-dimensional spatial positioning algorithm is combined with the multi-sensor data fusion technology, and the visual positioning, ultra-wideband ranging and radio frequency signal strength are fused through the extended Kalman filter to obtain the real-time spatial position coordinates of the accessory.

[0009] In an optional embodiment, the attention weight matrix is ​​calculated by the spatiotemporal attention module and the features of the RGB image branch and the skeleton sequence branch are dynamically fused to obtain the spatiotemporal fusion features. The spatiotemporal fusion features are input into the temporal modeling network to extract the long-range dependencies to obtain the operation behavior type. The steps include: rearranging the image features extracted from the RGB image branch in the time dimension to obtain a time feature map, processing the time feature map through the time self-attention calculation unit to obtain the time attention weight matrix, and the time self-attention calculation unit includes a time query weight matrix and a time key value weight matrix; rearranging the human skeleton features extracted from the skeleton sequence branch in the spatial dimension to obtain a spatial feature map, processing the spatial feature map through the spatial self-attention calculation unit to obtain the spatial attention weight matrix, and the spatial self-attention calculation unit includes a spatial query weight matrix and a spatial key value weight matrix; mapping the image features and the human skeleton features to the same dimensional space through the feature mapping layer to obtain RGB mapping features and skeleton mapping features, and using The multi-layer perceptron processes the RGB mapping features and the skeleton mapping features respectively to obtain the RGB feature scores and the skeleton feature scores, and generates adaptive fusion weights based on the RGB feature scores and the skeleton feature scores; the temporal attention weight matrix is ​​fused with the image features to obtain the temporal enhanced image features, the spatial attention weight matrix is ​​fused with the human skeleton features to obtain the spatial enhanced skeleton features, and the temporal enhanced image features and the spatial enhanced skeleton features are dynamically fused using the adaptive fusion weights to obtain the spatiotemporal fusion features; the spatiotemporal fusion features are input into the temporal modeling network, and the temporal modeling network includes a multi-scale feature extraction module, a feature aggregation module and a residual connection module, the multi-scale feature extraction module uses one-dimensional convolutions with different convolution kernel sizes to extract multi-scale temporal features in parallel, the feature aggregation module dynamically fuses the multi-scale temporal features through an attention mechanism, and the residual connection module performs residual connection on the fused features and the input features to extract long-range dependencies to obtain the type of work behavior.

[0010] In an optional embodiment, the deep learning recognition model further includes a multi-scale feature pyramid network, and the step of fusing convolution features with upsampled high-level features at different scale levels to obtain multi-scale fusion features includes: the multi-scale feature pyramid network includes five feature levels, and the spatial resolution of each feature level is reduced by two times, wherein the spatial resolution of the first feature level is the highest, and the spatial resolution of the fifth feature level is the lowest, and each feature level is subjected to feature extraction through a convolution layer with a step size to obtain basic features, and the convolution layer with a step size includes multiple convolution kernels, and the step size of each convolution kernel is 2, which is used to downsample and extract features of the input features to obtain each The invention relates to a method for merging the basic features of a multi-scale fusion image, wherein the method comprises: performing upsampling on the high-level features through a deconvolution operation to obtain upsampling features, wherein the spatial resolution of the upsampling features is the same as the spatial resolution of the adjacent low-level features; generating low-level feature adaptive weights for the low-level features using a 1×1 convolution layer, and generating upsampling feature adaptive weights for the upsampling features using a 1×1 convolution layer; multiplying the low-level feature adaptive weights with the low-level features to obtain weighted low-level features, multiplying the upsampling feature adaptive weights with the upsampling features to obtain weighted upsampling features, and performing weighted summation on the weighted low-level features and the weighted upsampling features to obtain multi-scale fusion features.

[0011] In an optional implementation, a three-dimensional spatial positioning algorithm is combined with multi-sensor data fusion technology to fuse visual positioning, ultra-wideband ranging and radio frequency signal strength through an extended Kalman filter to obtain the real-time spatial position coordinates of the accessory, including: establishing a state vector containing the three-dimensional position and velocity of the accessory, the state vector including a position component and a velocity component, constructing a state transfer equation based on a uniform motion model, the state transfer equation describing the evolution relationship between the position and velocity of the accessory through a state transfer matrix; constructing a multi-sensor observation equation and acquiring multi-sensor data, the multi-sensor observation equation including a visual positioning observation model, an ultra-wideband ranging observation model and a radio frequency signal strength observation model, wherein the visual positioning observation model directly acquires the three-dimensional coordinates of the accessory, and the ultra-wideband ranging observation model acquires the three-dimensional coordinates of the accessory. The broadband ranging observation model calculates the ranging value based on the accessory position and the base station position, and the radio frequency signal strength observation model establishes a mapping relationship between signal strength and distance based on the path loss index. The multi-sensor data includes visual positioning data, ultra-wideband ranging data and radio frequency signal strength data; an adaptive factor is introduced to dynamically weight the multi-sensor data, and the adaptive factor calculates the confidence of each sensor data based on the observation residual, and uses the confidence to update the observation noise covariance matrix; an extended Kalman filter framework is used to recursively estimate the state vector, and a priori state estimation is calculated based on the state transfer equation in the prediction stage. In the update stage, the multi-sensor observation equation and the adaptive factor are used to correct the prior state estimation to obtain the real-time spatial position coordinates of the accessory.

[0012] In an optional implementation, the early warning analysis model determines the operator's operating intention based on the identified operating behavior type, matches the accessory feature information with a preset accessory information database in real time, and maps and compares the real-time spatial position coordinates of the accessories with the three-dimensional virtual model of the warehouse in real time. When it is detected that the operator's accessory picking behavior or placement behavior is inconsistent with the preset cargo location rules in the three-dimensional virtual model of the warehouse, a warning message is pushed to the operator, and the specific location of the erroneous operation and the accessory information are marked in the three-dimensional visualization interface. The steps include: the early warning analysis model includes an operation intention recognition module, a feature matching module, a spatial constraint verification module and an early warning decision module; the operation intention recognition module is used to input the operation behavior feature vector containing action type, duration, and acceleration into a bidirectional long short-term memory network model, the bidirectional long short-term memory network model includes a hidden layer structure for capturing the front-end dependency relationship of the action sequence and a time attention mechanism for calculating the importance weight of the time step, and outputs the operator through a softmax classifier. The probability distribution of the picking intention and the placing intention; the feature matching module is used to match the accessory feature information with the preset accessory information database in real time, map the size, weight and material features of the accessory into a binary feature fingerprint, adopt the local sensitive hashing algorithm based on the feature weight vector for feature matching, realize the rapid retrieval of the binary feature fingerprint through the block index structure, dynamically adjust the similarity threshold based on the historical matching accuracy, and perform real-time feature matching in the accessory information index tree; the spatial constraint verification module is used to construct the spatial index structure of the three-dimensional virtual model of the warehouse, divide the warehouse space into grid units for storing the cargo location rule information, and map and compare the real-time spatial position coordinates of the accessory with the three-dimensional virtual model of the warehouse in real time; the early warning decision module is used to make a legality judgment based on the probability distribution of the operator's operation intention and the real-time spatial position of the accessory. When it is detected that the picking or placing behavior of the accessory is inconsistent with the cargo location rule, the early warning information is pushed to the operator, and the specific location and accessory information of the wrong operation are marked in the three-dimensional visualization interface.

[0013] According to a second aspect of an embodiment of the present invention, a wrong-taking and wrong-placing warning system for a drone parts warehouse is provided, comprising: a first unit, which is used to obtain real-time operation image data of an operator through an image acquisition device deployed in the warehouse, collect the location data of the accessories through a distributed sensor network, and standardize the collected real-time operation image data and location data to generate an accessories operation data stream; a second unit, which is used to input the accessories operation data stream into a deep learning recognition model, identify the operation behavior type of the operator and the feature information of the accessories, calculate the real-time spatial position coordinates of the accessories by using a three-dimensional spatial positioning algorithm combined with a multi-sensor data fusion technology, and input the identified operation behavior type, the feature information of the accessories and the real-time spatial position coordinates of the accessories into a warning analysis model; a third unit, which is used for the warning analysis model to judge the operation intention of the operator based on the identified operation behavior type, match the feature information of the accessories with a preset accessories information database in real time, and map and compare the real-time spatial position coordinates of the accessories with a three-dimensional virtual model of the warehouse in real time. When it is detected that the operator's accessory taking behavior or placement behavior does not conform to the preset cargo location rules in the three-dimensional virtual model of the warehouse, a warning message is pushed to the operator, and the specific location and accessory information of the wrong operation are marked in the three-dimensional visualization interface.

[0014] According to a third aspect of an embodiment of the present invention, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0015] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0016] The present invention deploys image acquisition devices and distributed sensor networks to obtain the operator's operation images and accessory location data in real time. Combined with the deep learning recognition model, it can accurately identify the operation behavior type and accessory feature information, greatly improve the recognition accuracy of incorrect operations, and effectively reduce human judgment errors.

[0017] The present invention uses a three-dimensional spatial positioning algorithm and multi-sensor data fusion technology to achieve accurate calculation of the real-time spatial position coordinates of accessories. By mapping and comparing these data with the three-dimensional virtual model of the warehouse in real time, it is possible to quickly and accurately determine whether the accessories are placed in the correct position, significantly improving the spatial accuracy and efficiency of warehouse management.

[0018] Based on the early warning analysis model, the system can intelligently judge the operator's operating intention and match the accessory information with the preset database in real time. When an erroneous operation is detected, the system will immediately push the early warning information to the operator and clearly mark the error location and accessory information in the three-dimensional visualization interface. This intelligent early warning and visual display method greatly improves the efficiency of correcting erroneous operations and reduces the labor cost and error rate of warehouse management. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flow chart of a method for early warning of wrong taking and wrong placing of parts in a warehouse for drone parts according to an embodiment of the present invention; Figure 2 The present invention is a schematic diagram of the structure of a wrong-taken or wrong-placed warning system applicable to a drone parts warehouse according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0022] Figure 1 FIG. 1 is a flow chart of a method for early warning of wrong taking or wrong placing of parts in a warehouse of a drone according to an embodiment of the present invention. Figure 1As shown, the method includes: S1. acquiring real-time operation image data of operators through an image acquisition device deployed in a warehouse, acquiring the position data of accessories through a distributed sensor network, standardizing the acquired real-time operation image data and position data, and generating an accessory operation data stream; S2. inputting the accessory operation data stream into a deep learning recognition model, identifying the operation behavior type of operators and the feature information of accessories, calculating the real-time spatial position coordinates of accessories by using a three-dimensional spatial positioning algorithm combined with multi-sensor data fusion technology, and inputting the identified operation behavior type, accessory feature information and the real-time spatial position coordinates of accessories into an early warning analysis model; S3. The early warning analysis model determines the operation intention of operators based on the identified operation behavior type, matches the accessory feature information with a preset accessory information database in real time, and maps and compares the real-time spatial position coordinates of accessories with a three-dimensional virtual model of the warehouse in real time. When it is detected that the operator's accessory picking behavior or placement behavior is inconsistent with the preset cargo location rules in the three-dimensional virtual model of the warehouse, a warning message is pushed to the operator, and the specific location and accessory information of the wrong operation are marked in the three-dimensional visualization interface.

[0023] For example, first, image acquisition devices such as high-definition cameras and depth cameras are deployed in the warehouse to obtain real-time operation image data of operators. At the same time, a distributed sensor network is set up in the warehouse, including ultra-wideband positioning sensors, RFID readers, etc., to collect real-time location data of accessories. The collected image data and location data are standardized, including image denoising, scale normalization, etc., to generate a standardized accessory operation data stream.

[0024] Next, the standardized accessory operation data stream is input into the pre-trained deep learning recognition model. The model uses a convolutional neural network structure, which contains multiple convolutional layers, pooling layers, and fully connected layers. The model can identify the type of operator's operation behavior, such as picking up and placing, and identify the characteristic information of accessories, such as type and model. For example, it is recognized that the operator is taking a propeller from the shelf in area A.

[0025] Then, the real-time spatial position coordinates of the accessory are calculated by using a three-dimensional spatial positioning algorithm combined with multi-sensor data fusion technology. Specifically, the approximate position of the accessory is first obtained using an ultra-wideband positioning sensor, and then the depth information of the depth camera is combined for precise positioning, and finally the tag information of the RFID reader is integrated to obtain the precise three-dimensional coordinates of the accessory. For example, the coordinates of the propeller are calculated to be (x=2.5m, y=1.8m, z=1.2m).

[0026] The identified operation behavior type, accessory feature information, and calculated real-time spatial position coordinates are input into the early warning analysis model. The model is built based on the rule engine and knowledge graph, and can judge the operator's operating intention. For example, it is judged that the operator intends to move the propeller from area A to area B. At the same time, the model matches the identified accessory feature information with the preset accessory information database in real time to verify the accuracy of the accessory information. In addition, the model also performs real-time mapping and comparison of the real-time spatial position coordinates of the accessories with the pre-established three-dimensional virtual model of the warehouse.

[0027] When the early warning analysis model detects that the operator's behavior of picking up or placing accessories does not conform to the preset cargo location rules in the warehouse's three-dimensional virtual model, it is judged as an incorrect operation. For example, it is detected that the operator mistakenly placed the propeller in area A in area C. At this time, the system pushes an early warning message to the operator's mobile terminal, prompting "The propeller model XX is placed in the wrong position, please move it to the correct cargo location." At the same time, on the three-dimensional visualization interface of the warehouse monitoring center, the specific location of the incorrect operation is marked in red, and the relevant accessory information is displayed.

[0028] Finally, the system records the erroneous operation and stores the relevant data in the database for subsequent analysis and optimization of warehouse management.

[0029] The present invention can accurately capture the operating behavior and accessory position of operators through real-time image recognition and multi-sensor positioning, greatly improving the accuracy and real-time performance of wrong-taking and wrong-placing detection; it adopts deep learning and multimodal data fusion technology, can adapt to complex and changeable warehouse environments, and has strong robustness and generalization capabilities; combined with three-dimensional visualization and mobile terminal push, it realizes fast and accurate error positioning and timely and effective early warning prompts, significantly reduces human operating errors, and improves warehouse management efficiency.

[0030] In an optional embodiment, the real-time operation image data of the operator is obtained by an image acquisition device deployed in the warehouse, the location data of the accessories is collected through a distributed sensor network, and the collected real-time operation image data and location data are standardized. The steps of generating the accessory operation data stream include: arranging high-definition camera arrays on the top and sides of the shelf to obtain the real-time operation image data of the operator, triggering the high frame rate acquisition mode and performing target tracking when the operator is detected entering the operation area; arranging UWB positioning base stations at the four corners of the shelf to obtain the location data of the accessories using a time difference positioning algorithm; performing distortion correction, illumination compensation and image enhancement processing on the collected real-time operation image data, and performing Kalman filtering on the location data; using a sliding window method to segment the processed real-time operation image data and location data, and performing time alignment on the segmented data based on the network time protocol, and generating an accessory operation data stream containing timestamps, image data and location data after wavelet denoising and standardization.

[0031] Exemplarily, this embodiment provides a method for acquiring real-time operation image data of operators through an image acquisition device deployed in a warehouse, collecting location data of accessories through a distributed sensor network, standardizing the collected real-time operation image data and location data, and generating an accessories operation data stream.

[0032] First, high-definition camera arrays are placed on the top and sides of the shelves in the warehouse. The high-definition camera array consists of multiple high-resolution cameras, each with a resolution of no less than 1920x1080 pixels and a frame rate of up to 30fps. The camera array is arranged in a matrix to ensure full coverage of the entire operating area. At the same time, infrared sensors are integrated into the camera array to detect the entry of operators.

[0033] When the infrared sensor detects that a worker has entered the work area, the high frame rate acquisition mode is triggered. In this mode, the frame rate of the camera array is increased to 60fps or higher to capture the worker's rapid movements. At the same time, the target tracking algorithm is started. The algorithm is based on a convolutional neural network and can accurately identify and track the position and movement of the worker. The output of the target tracking algorithm includes the coordinates of the worker, posture information, and action classification results.

[0034] UWB (ultra-wideband) positioning base stations are arranged at the four corners of the shelves, and the positioning accuracy of each base station is better than 10cm. The UWB base station uses a time difference positioning algorithm to obtain the location data of the accessories. Specifically, a small UWB tag is installed on each accessory, and the tag sends a signal regularly. After multiple UWB base stations receive the signal, they calculate the three-dimensional coordinates of the accessory based on the time difference of signal arrival. The time difference positioning algorithm takes into account the influence of multipath effect and non-line-of-sight propagation, and improves the positioning accuracy through optimization algorithm.

[0035] The collected real-time operation image data is post-processed. First, distortion correction is performed, and a polynomial model is used to eliminate lens distortion to ensure the geometric accuracy of the image. Then, illumination compensation is performed, and an adaptive histogram equalization method is used to improve image contrast and detail visibility. Finally, image enhancement is performed, and a non-local mean filtering algorithm is applied to remove noise, and a sharpening algorithm is used to enhance edge details.

[0036] The Kalman filter algorithm is applied to the accessory position data obtained by UWB positioning. The algorithm combines the motion model and measurement model of the accessory to smooth the original position data, effectively suppress random errors, and improve positioning accuracy and stability. The filtered position data includes the three-dimensional coordinates of the accessory and its uncertainty estimation.

[0037] The processed real-time operation image data and position data are segmented using the sliding window method. The window size is set to 1 second and the step size is 0.1 second to ensure the continuity and overlap of the data. For each time window, the image sequence and corresponding position data within the window are extracted.

[0038] The segmented data is time-aligned based on the Network Time Protocol (NTP). The NTP server provides a unified time base for all devices, accurate to the millisecond level. During the alignment process, the timestamps of the image data and location data are adjusted to the same time axis to ensure data synchronization.

[0039] The data after time series alignment is subjected to wavelet denoising. The db4 wavelet basis function is used to perform three-layer wavelet decomposition on the image data and position data respectively. The wavelet coefficients are processed by the soft threshold method to remove high-frequency noise, and then wavelet reconstruction is performed to obtain the denoised data.

[0040] Finally, the denoised data is normalized. For image data, the pixel values ​​are normalized to the range of 0-1; for position data, the coordinate values ​​are normalized to the predefined warehouse coordinate system. The standardized data is combined into a parts operation data stream, and each record contains a timestamp, standardized image data, and standardized position data.

[0041] The present invention improves the accuracy and reliability of data collection. Through the combination of high-definition camera array and UWB positioning system, high-precision tracking of operators and accessories is achieved. The post-processing algorithm further improves the data quality, provides a reliable basis for subsequent analysis, and enhances the temporal and spatial consistency of the data. The sliding window method and NTP time synchronization mechanism are used to ensure the consistency of image data and position data in time and space, which is conducive to the accurate restoration of the operation process; and realizes the standardization and unified expression of data. Through a series of processing steps, data from different sources and in different forms are converted into a unified format of accessory operation data stream, which is convenient for subsequent storage, analysis and application, and improves the versatility and scalability of the system.

[0042] In an optional implementation, the accessory operation data stream is input into a deep learning recognition model to identify the operator's operation behavior type and accessory feature information, and the steps of using a three-dimensional spatial positioning algorithm combined with multi-sensor data fusion technology to calculate the real-time spatial position coordinates of the accessory include: inputting the accessory operation data stream into a deep learning recognition model, the deep learning recognition model includes a two-stream network structure, the RGB image branch of the two-stream network structure uses a three-dimensional convolutional residual network to extract image features, and the skeleton sequence branch uses a spatiotemporal graph convolutional network to extract human skeleton features, the attention weight matrix is ​​calculated through the spatiotemporal attention module, and the features of the RGB image branch and the skeleton sequence branch are dynamically fused to obtain spatiotemporal fusion features, and the spatiotemporal fusion features are input into a temporal modeling network The long-range dependency is extracted to obtain the operation behavior type; the deep learning recognition model also includes a multi-scale feature pyramid network, which fuses the convolutional features with the upsampled high-level features at different scale levels to obtain multi-scale fusion features, applies the attention mechanism to the multi-scale fusion features to generate an attention map, and extracts the accessory instance-level feature vector through the instance alignment layer to obtain the accessory feature information; the image data in the accessory operation data stream is processed through the deep learning recognition model to obtain the visual positioning result, and the ultra-wideband ranging value and the radio frequency signal strength index are obtained through the distributed sensor network; the three-dimensional spatial positioning algorithm is combined with the multi-sensor data fusion technology, and the visual positioning, ultra-wideband ranging and radio frequency signal strength are fused through the extended Kalman filter to obtain the real-time spatial position coordinates of the accessory.

[0043] Exemplarily, the accessory operation data stream is input into the deep learning recognition model. The deep learning recognition model includes a two-stream network structure, in which the RGB image branch uses a three-dimensional convolutional residual network and the skeleton sequence branch uses a spatiotemporal graph convolutional network. Specifically, the RGB image branch uses a 3D ResNet-50 network, which includes 5 convolution blocks, each of which includes multiple 3D convolutional layers, batch normalization layers, and ReLU activation functions. The skeleton sequence branch uses an ST-GCN network (spatial-temporal graph convolutional neural network), which includes 9 ST-GCN blocks, each of which includes a graph convolution layer and a temporal convolution layer.

[0044] The two branches of the two-stream network extract image features and human skeleton features respectively. The RGB image branch outputs a 1024-dimensional feature vector, and the skeleton sequence branch outputs a 256-dimensional feature vector. Then the attention weight matrix is ​​calculated through the spatiotemporal attention module to dynamically fuse the features of the two branches. Specifically, the two feature vectors are first concatenated and a 1280-dimensional fusion feature is obtained through a fully connected layer. Then the self-attention mechanism is used to calculate the attention weight and weight the fusion feature. Finally, a 1280-dimensional spatiotemporal fusion feature is obtained.

[0045] The spatiotemporal fusion features are input into the time series modeling network to extract long-range dependencies. The network adopts a bidirectional LSTM (Long Short-Term Memory Network) structure, which contains 2 layers and 128 hidden units in each layer. The LSTM network can capture long-term time series information and output a 256-dimensional time series feature vector. Finally, the probability distribution of the operation behavior type is obtained through the fully connected layer and the Softmax classifier. For example, the operation behavior can be divided into 10 categories, including picking up accessories, installing accessories, and checking accessories.

[0046] The deep learning recognition model also includes a multi-scale feature pyramid network FPN. FPN includes a bottom-up backbone network and a top-down feature fusion network. The backbone network adopts the ResNet-50 structure and contains 5 convolution stages. The convolution features are fused with the upsampled high-level features at different scale levels to obtain feature maps of multiple scales, with sizes of 28×28, 56×56, 112×112, and 224×224 respectively.

[0047] The channel attention and spatial attention mechanisms are applied to the multi-scale fusion features to generate an attention map. The channel attention adopts the SE Block structure, and the spatial attention adopts the non-local module. The attention mechanism can highlight important features and suppress irrelevant features. Then the accessory instance-level features are extracted through the ROI Align layer. A 7×7 feature map is extracted for each candidate region, and a 2048-dimensional feature vector is obtained through the fully connected layer as the feature information of the accessory. Accessory features can contain information such as category, color, and shape.

[0048] Next, the image data in the accessory operation data stream is processed through a deep learning recognition model to obtain the visual positioning result. Specifically, the YOLOv5 target detection network is used to detect the accessories in the image and obtain the 2D bounding box coordinates of the accessories. Then, the PnP algorithm is used in combination with the camera intrinsic parameters to convert the 2D coordinates into 3D space coordinates to obtain the visual positioning result.

[0049] At the same time, the UWB distance measurement value and RF signal strength index are obtained through the distributed sensor network. The distributed sensor network includes multiple UWB positioning base stations and RF beacons. The UWB base station measures the distance from the accessory to the base station through the TOF ranging principle. The RF beacon estimates the relative distance between the accessory and the beacon through the RF signal strength index.

[0050] Finally, the three-dimensional spatial positioning algorithm is combined with multi-sensor data fusion technology to fuse visual positioning, ultra-wideband ranging and radio frequency signal strength through extended Kalman filtering. The extended Kalman filter includes two stages: prediction and update. The prediction stage predicts the current state based on the state and motion model at the previous moment. The update stage corrects the prediction results based on the observed values. The real-time three-dimensional spatial position coordinates of the accessories are obtained through iterative optimization.

[0051] The present invention combines a deep learning recognition model with a dual-stream network and spatiotemporal attention, which can accurately identify the types of operating behaviors in complex scenarios and improve the accuracy of behavior recognition. It uses a multi-scale feature pyramid network and an attention mechanism to extract accessory features, which can obtain rich accessory semantic information, which is beneficial to subsequent accessory tracking and management. It combines visual positioning, ultra-wideband ranging and radio frequency signal strength for multi-sensor fusion, which can achieve high-precision real-time positioning of accessories and improve the accuracy and robustness of spatial positioning.

[0052] In an optional embodiment, the attention weight matrix is ​​calculated by the spatiotemporal attention module and the features of the RGB image branch and the skeleton sequence branch are dynamically fused to obtain the spatiotemporal fusion features. The spatiotemporal fusion features are input into the temporal modeling network to extract the long-range dependencies and obtain the operation behavior type. The steps include: rearranging the image features extracted from the RGB image branch in the time dimension to obtain a time feature map, processing the time feature map through the time self-attention calculation unit to obtain the time attention weight matrix, and the time self-attention calculation unit includes a time query weight matrix and a time key value weight matrix; rearranging the human skeleton features extracted from the skeleton sequence branch in the spatial dimension to obtain a spatial feature map, processing the spatial feature map through the spatial self-attention calculation unit to obtain the spatial attention weight matrix, and the spatial self-attention calculation unit includes a spatial query weight matrix and a spatial key value weight matrix; mapping the image features and the human skeleton features to the same dimensional space through the feature mapping layer to obtain RGB mapping features and skeleton mapping features, and using The multi-layer perceptron processes the RGB mapping features and the skeleton mapping features respectively to obtain the RGB feature scores and the skeleton feature scores, and generates adaptive fusion weights based on the RGB feature scores and the skeleton feature scores; the temporal attention weight matrix is ​​fused with the image features to obtain the temporal enhanced image features, the spatial attention weight matrix is ​​fused with the human skeleton features to obtain the spatial enhanced skeleton features, and the temporal enhanced image features and the spatial enhanced skeleton features are dynamically fused using the adaptive fusion weights to obtain the spatiotemporal fusion features; the spatiotemporal fusion features are input into the temporal modeling network, and the temporal modeling network includes a multi-scale feature extraction module, a feature aggregation module and a residual connection module, the multi-scale feature extraction module uses one-dimensional convolutions with different convolution kernel sizes to extract multi-scale temporal features in parallel, the feature aggregation module dynamically fuses the multi-scale temporal features through an attention mechanism, and the residual connection module performs residual connection on the fused features and the input features to extract long-range dependencies to obtain the type of work behavior.

[0053] Exemplarily, this embodiment provides a method for calculating the attention weight matrix through the spatiotemporal attention module and dynamically fusing the features of the RGB image branch and the skeleton sequence branch to obtain the spatiotemporal fusion features, and inputting the spatiotemporal fusion features into the temporal modeling network to extract the long-range dependencies to obtain the type of work behavior. Specifically, the following steps are included: First, the image features extracted by the RGB image branch are rearranged in the time dimension to obtain a time feature map. Specifically, assuming that the input RGB image sequence contains 16 frames of images, each frame size is 224x224x3, and a 16x7x7x512 feature map is obtained after passing through the CNN feature extraction network. The feature map is rearranged according to the time dimension to obtain a 7x7x8192 time feature map.

[0054] Then, the temporal feature map is processed by the temporal self-attention calculation unit to obtain the temporal attention weight matrix. The temporal self-attention calculation unit includes the temporal query weight matrix and the temporal key value weight matrix, where the size of the temporal query weight matrix and the temporal key value weight matrix are both 8192x512. Through matrix multiplication operation, a 16x16 temporal attention weight matrix is ​​obtained.

[0055] At the same time, the human skeleton features extracted by the skeleton sequence branch are rearranged in spatial dimensions to obtain a spatial feature map. Assuming that the input skeleton sequence contains 16 frames, each frame contains 17 key points, and each key point is represented by a 3D coordinate. After passing through the feature extraction network, a 16x17x64 feature map is obtained. The feature map is rearranged according to the spatial dimension to obtain a 17x1024 spatial feature map.

[0056] Next, the spatial feature map is processed using the spatial self-attention calculation unit to obtain the spatial attention weight matrix. The spatial self-attention calculation unit includes a spatial query weight matrix and a spatial key value weight matrix, where the size of the spatial query weight matrix and the spatial key value weight matrix are both 1024x128. Through matrix multiplication, a 17x17 spatial attention weight matrix is ​​obtained.

[0057] Next, the image features and human skeleton features are mapped to the same dimensional space through the feature mapping layer. Specifically, a 1x1 convolutional layer is used to map the 16x7x7x512 image features to 16x256 RGB mapping features, and a fully connected layer is used to map the 16x17x64 human skeleton features to 16x256 skeleton mapping features.

[0058] Then, the RGB mapping features and skeleton mapping features are processed by multi-layer perceptron to obtain RGB feature scores and skeleton feature scores. The multi-layer perceptron contains two hidden layers, and the number of neurons in the hidden layers is 128 and 64 respectively. The last layer outputs a scalar as the feature score. Based on the RGB feature score and the skeleton feature score, the adaptive fusion weight is generated by the softmax function.

[0059] Next, the temporal attention weight matrix is ​​fused with the image features to obtain the temporal enhanced image features. Specifically, the 16x16 temporal attention weight matrix is ​​matrix multiplied with the 16x7x7x512 image features to obtain the 16x7x7x512 temporal enhanced image features. Similarly, the 17x17 spatial attention weight matrix is ​​matrix multiplied with the 16x17x64 human skeleton features to obtain the 16x17x64 spatial enhanced skeleton features.

[0060] Then, the temporal enhanced image features and spatial enhanced skeleton features are dynamically fused using adaptive fusion weights to obtain the temporal and spatial fusion features. Assuming that the RGB feature score is 0.6 and the skeleton feature score is 0.4, the temporal and spatial fusion features can be expressed as: 0.6×temporal enhanced image features + 0.4×spatial enhanced skeleton features.

[0061] Finally, the spatiotemporal fusion features are input into the time series modeling network to extract long-range dependencies and obtain the types of work behaviors. The time series modeling network includes a multi-scale feature extraction module, a feature aggregation module, and a residual connection module. The multi-scale feature extraction module uses one-dimensional convolutions with kernel sizes of 3, 5, and 7 to extract multi-scale time series features in parallel. The feature aggregation module dynamically fuses multi-scale time series features through the attention mechanism. Specifically, a 1x1 convolution layer is used to calculate the attention weight of each scale feature, and then a weighted sum is performed. The residual connection module performs a residual connection between the fused features and the input features, and finally obtains the probability distribution of each work behavior type through a fully connected layer and a softmax function.

[0062] The present invention calculates the time and space attention weight matrices respectively through the spatiotemporal attention module, which can effectively capture the temporal information of the RGB image sequence and the spatial structure information of the skeleton sequence, and improve the feature representation capability; the adaptive fusion weights are used to dynamically fuse the RGB features and the skeleton features, and the importance of the two modal features can be adaptively adjusted according to different samples, thereby improving the generalization ability of the model; the temporal modeling network using multi-scale feature extraction and attention aggregation can effectively extract and fuse information of different time scales, enhance the modeling capability of long-range dependencies, and improve the accuracy of job behavior recognition.

[0063] In an optional embodiment, the deep learning recognition model further includes a multi-scale feature pyramid network, and the step of fusing convolution features with upsampled high-level features at different scale levels to obtain multi-scale fusion features includes: the multi-scale feature pyramid network includes five feature levels, and the spatial resolution of each feature level is reduced by two times, wherein the spatial resolution of the first feature level is the highest, and the spatial resolution of the fifth feature level is the lowest, and each feature level is subjected to feature extraction through a convolution layer with a step length to obtain a basic feature, and the convolution layer with a step length includes multiple convolution kernels, and the step length of each convolution kernel is 2, which is used to downsample and extract features from the input features to obtain each The invention relates to a method for merging the basic features of a multi-scale fusion image, wherein the method comprises: performing upsampling on the high-level features through a deconvolution operation to obtain upsampling features, wherein the spatial resolution of the upsampling features is the same as the spatial resolution of the adjacent low-level features; generating low-level feature adaptive weights for the low-level features using a 1×1 convolution layer, and generating upsampling feature adaptive weights for the upsampling features using a 1×1 convolution layer; multiplying the low-level feature adaptive weights with the low-level features to obtain weighted low-level features, multiplying the upsampling feature adaptive weights with the upsampling features to obtain weighted upsampling features, and performing weighted summation on the weighted low-level features and the weighted upsampling features to obtain multi-scale fusion features.

[0064] Exemplarily, this embodiment provides a deep learning recognition model, which includes a multi-scale feature pyramid network for fusing convolutional features with upsampled high-level features at different scale levels to obtain multi-scale fusion features. The specific implementation process is as follows: First, a multi-scale feature pyramid network is constructed, including five feature levels. The spatial resolution of each feature level is reduced by two times, where the spatial resolution of the first feature level is the highest and the spatial resolution of the fifth feature level is the lowest. For example, for an image with an input size of 224x224, the spatial resolutions of the five feature levels are 56x56, 28x28, 14x14, 7x7, and 4x4, respectively.

[0065] Next, feature extraction is performed on each feature level through a stride convolution layer to obtain basic features. The stride convolution layer includes multiple convolution kernels, each with a stride of 2, which are used to downsample the input features and extract features. For example, for the first feature level, 64 3x3 convolution kernels with a stride of 2 can be used to downsample the 56x56 input feature map to a 28x28 feature map and extract features at the same time.

[0066] Then, the obtained basic features are fused across layers. First, the high-level features are upsampled through deconvolution operations to obtain upsampled features. The spatial resolution of the upsampled features is the same as the spatial resolution of the adjacent low-level features. For example, the 7x7 fourth-layer features are upsampled to 14x14 feature maps through a 2x2 deconvolution kernel with a step size of 2, which is the same as the spatial resolution of the third-layer features.

[0067] Next, a 1x1 convolution layer is used to generate low-level feature adaptive weights for low-level features, and a 1x1 convolution layer is used to generate up-sampled feature adaptive weights for up-sampled features. The role of the 1x1 convolution layer is to learn the weight relationship between different channels and adjust the importance of features. For example, for a low-level feature with 256 channels, 256 1x1 convolution kernels can be used to generate corresponding 256 weight values.

[0068] Subsequently, the low-level feature adaptive weight is multiplied by the low-level feature to obtain the weighted low-level feature; the upsampled feature adaptive weight is multiplied by the upsampled feature to obtain the weighted upsampled feature. This step realizes the adaptive weighting of features, highlights important features, and suppresses unimportant features.

[0069] Finally, the weighted low-level features and weighted up-sampled features are weighted summed to obtain multi-scale fusion features. The weighted summation can be achieved by element-by-element addition, and the obtained fusion features contain both low-level detail information and high-level semantic information.

[0070] In order to further improve the effect of feature fusion, a nonlinear activation function, such as the ReLU function, can be added after the weighted summation to increase the nonlinear expression ability of the model. In addition, a 1x1 convolution layer can be added after the fusion feature to adjust the number of channels of the fusion feature to match the subsequent network structure.

[0071] In practical applications, the number of feature levels and the number of convolution kernels per layer can be adjusted according to the needs of specific tasks. For example, for complex object detection tasks, the number of feature levels can be increased to 6 or 7 to obtain feature information of more scales. At the same time, the number of convolution kernels per layer can be appropriately adjusted according to the limitations of computing resources to strike a balance between model performance and computing efficiency.

[0072] The present invention uses a feature pyramid network and cross-layer feature fusion, so that the model can effectively combine feature information of different scales, retaining the detailed information of low-level features and including the semantic information of high-level features, thereby improving the model's ability to recognize objects of different sizes; using a 1x1 convolutional layer to generate adaptive weights, dynamic weighting of features is achieved, and the importance of different features can be automatically adjusted according to the characteristics of the input data, thereby improving the adaptability and robustness of the model; using a convolutional layer with a step size for feature extraction and downsampling reduces the amount of calculation and memory usage. At the same time, using a 1x1 convolutional layer for feature fusion and channel adjustment further optimizes the computational efficiency, making the model more suitable for deployment and application on resource-constrained devices.

[0073] In an optional implementation, a three-dimensional spatial positioning algorithm is combined with multi-sensor data fusion technology to fuse visual positioning, ultra-wideband ranging and radio frequency signal strength through an extended Kalman filter to obtain the real-time spatial position coordinates of the accessory, including: establishing a state vector containing the three-dimensional position and velocity of the accessory, the state vector including a position component and a velocity component, constructing a state transfer equation based on a uniform motion model, the state transfer equation describing the evolution relationship between the position and velocity of the accessory through a state transfer matrix; constructing a multi-sensor observation equation and acquiring multi-sensor data, the multi-sensor observation equation including a visual positioning observation model, an ultra-wideband ranging observation model and a radio frequency signal strength observation model, wherein the visual positioning observation model directly acquires the three-dimensional coordinates of the accessory, and the ultra-wideband ranging observation model acquires the three-dimensional coordinates of the accessory. The broadband ranging observation model calculates the ranging value based on the accessory position and the base station position, and the radio frequency signal strength observation model establishes a mapping relationship between signal strength and distance based on the path loss index. The multi-sensor data includes visual positioning data, ultra-wideband ranging data and radio frequency signal strength data; an adaptive factor is introduced to dynamically weight the multi-sensor data, and the adaptive factor calculates the confidence of each sensor data based on the observation residual, and uses the confidence to update the observation noise covariance matrix; an extended Kalman filter framework is used to recursively estimate the state vector, and a priori state estimation is calculated based on the state transfer equation in the prediction stage. In the update stage, the multi-sensor observation equation and the adaptive factor are used to correct the prior state estimation to obtain the real-time spatial position coordinates of the accessory.

[0074] Exemplarily, this embodiment provides a method for obtaining the real-time spatial position coordinates of an accessory by combining a three-dimensional spatial positioning algorithm with a multi-sensor data fusion technology, and fusing visual positioning, ultra-wideband ranging, and radio frequency signal strength through an extended Kalman filter. The specific steps are as follows: First, a state vector containing the three-dimensional position and velocity of the accessory is established. The state vector includes a position component (x, y, z) and a velocity component (vx, vy, vz), a total of 6 elements. A state transfer equation is constructed based on a uniform motion model, and the equation describes the evolution relationship between the position and velocity of the accessory through a state transfer matrix. Specifically, assuming that the accessory moves at a uniform speed in a short period of time, the position at the next moment is equal to the current position plus the product of the velocity and the time interval, and the velocity remains unchanged. For example, if the current position of the accessory is (1m, 2m, 3m), the velocity is (0.1m / s, 0.2m / s, 0.3m / s), and the time interval is 0.1s, the predicted position at the next moment is (1.01m, 2.02m, 3.03m).

[0075] Next, construct a multi-sensor observation equation and obtain multi-sensor data. The multi-sensor observation equation includes a visual positioning observation model, an ultra-wideband ranging observation model, and an RF signal strength observation model. The visual positioning observation model directly obtains the three-dimensional coordinates of the accessory. For example, the spatial coordinates of the accessory (1.5m, 2.5m, 3.5m) can be obtained through stereo vision or a depth camera. The ultra-wideband ranging observation model calculates the ranging value based on the location of the accessory and the location of the base station. Assuming that there are three base stations with coordinates of (0m, 0m, 0m), (10m, 0m, 0m), and (0m, 10m, 0m), the three ranging values ​​can be calculated to be 4.56m, 8.91m, and 9.01m, respectively. The RF signal strength observation model establishes a mapping relationship between signal strength and distance based on the path loss index. For example, when the signal strength is -60dBm, the estimated distance is about 5m. The multi-sensor data includes visual positioning data, ultra-wideband ranging data, and RF signal strength data.

[0076] Then, an adaptive factor is introduced to dynamically weight the multi-sensor data. The adaptive factor calculates the confidence of each sensor data based on the observation residual, and uses the confidence to update the observation noise covariance matrix. Specifically, the difference between the observed value and the predicted value of each sensor is calculated. The smaller the difference, the higher the confidence. For example, if the observation residuals of visual positioning, ultra-wideband ranging, and RF signal strength are 0.1m, 0.3m, and 0.5m, respectively, their confidences can be calculated to be 0.6, 0.3, and 0.1, respectively. According to these confidences, the observation noise covariance matrix is ​​adjusted so that sensor data with high confidence has a greater weight.

[0077] Finally, the extended Kalman filter framework is used to recursively estimate the state vector. In the prediction phase, the prior state estimate is calculated based on the state transition equation. For example, using the state estimate value of the previous moment (1m, 2m, 3m, 0.1m / s, 0.2m / s, 0.3m / s), the prior estimate of the current moment (1.01m, 2.02m, 3.03m, 0.1m / s, 0.2m / s, 0.3m / s) can be obtained. In the update phase, the prior state estimate is corrected using the multi-sensor observation equation and the adaptive factor. Specifically, the observation value of each sensor is compared with the predicted value of the observation equation, the Kalman gain is calculated, and then the prior estimate is corrected according to the Kalman gain. Assume that the corrected state estimate is (1.02m, 2.03m, 3.04m, 0.11m / s, 0.21m / s, 0.31m / s), which is the final real-time spatial position coordinates and speed of the accessory.

[0078] The present invention improves the accuracy and reliability of spatial positioning of accessories by fusing multiple sensor data. Visual positioning provides direct three-dimensional coordinate information, ultra-wideband ranging has high-precision distance measurement capabilities, and radio frequency signal strength is suitable for rough positioning at longer distances. The advantages of these three sensors complement each other and can maintain good positioning effects under different environments and distance conditions; the introduction of adaptive factors realizes dynamic weighting of multi-sensor data, and can automatically adjust its weight in the fusion process according to the real-time reliability of each sensor data. This method can effectively cope with the influence of factors such as sensor performance fluctuations and environmental interference, and improve the robustness and adaptability of the system; the extended Kalman filter framework is used to realize the joint estimation of the position and speed of the accessory, which can not only obtain the current position, but also predict the movement trend in the short term. This method takes into account the movement characteristics of the accessory, and can maintain a good tracking effect when the sensor data is temporarily missing or unreliable, thereby improving the continuity and stability of the system.

[0079] In an optional implementation, the early warning analysis model determines the operator's operating intention based on the identified operation behavior type, matches the accessory feature information with the preset accessory information database in real time, and maps and compares the real-time spatial position coordinates of the accessory with the three-dimensional virtual model of the warehouse in real time. When it is detected that the operator's accessory picking behavior or placement behavior is inconsistent with the preset cargo location rules in the three-dimensional virtual model of the warehouse, a warning message is pushed to the operator, and the specific location of the erroneous operation and the accessory information are marked in the three-dimensional visualization interface. The steps include: the early warning analysis model includes an operation intention recognition module, a feature matching module, a spatial constraint verification module and an early warning decision module; the operation intention recognition module is used to input the operation behavior feature vector containing action type, duration, and acceleration into a bidirectional long short-term memory network model, the bidirectional long short-term memory network model includes a hidden layer structure for capturing the front-end dependency relationship of the action sequence and a time attention mechanism for calculating the importance weight of the time step, and outputs the operator through a softmax classifier. The probability distribution of the picking intention and the placing intention; the feature matching module is used to match the accessory feature information with the preset accessory information database in real time, map the size, weight and material features of the accessory into a binary feature fingerprint, adopt the local sensitive hashing algorithm based on the feature weight vector for feature matching, realize the rapid retrieval of the binary feature fingerprint through the block index structure, dynamically adjust the similarity threshold based on the historical matching accuracy, and perform real-time feature matching in the accessory information index tree; the spatial constraint verification module is used to construct the spatial index structure of the three-dimensional virtual model of the warehouse, divide the warehouse space into grid units for storing the cargo location rule information, and map and compare the real-time spatial position coordinates of the accessory with the three-dimensional virtual model of the warehouse in real time; the early warning decision module is used to make a legality judgment based on the probability distribution of the operator's operation intention and the real-time spatial position of the accessory. When it is detected that the picking or placing behavior of the accessory is inconsistent with the cargo location rule, the early warning information is pushed to the operator, and the specific location and accessory information of the wrong operation are marked in the three-dimensional visualization interface.

[0080] Exemplarily, the early warning analysis model first receives the operation behavior information, including characteristics such as action type, duration, and acceleration. These characteristics are organized into a behavior feature vector and passed as input to the operation intention recognition module. The operation intention recognition module adopts a bidirectional long short-term memory network model, which contains multiple hidden units and can capture the front-end dependency of the action sequence. For example, the picking action usually starts with reaching out and grasping, while the placing action will have actions such as lifting and releasing. The model also introduces a temporal attention mechanism to weight the importance of different time steps. Through the softmax classifier, the probability distribution of the operator's picking intention and placing intention is finally output.

[0081] At the same time, the feature matching module receives the feature information of the accessories, including size, weight, material, etc. These features are mapped into binary feature fingerprints, for example, the size of 20cm×30cm×10cm is mapped to 100110, the weight of 2kg is mapped to 010, the material of plastic is mapped to 01, etc. The local sensitive hashing algorithm is used to match the features, which introduces feature weight vectors to assign different importance to different features. Fast retrieval is achieved through the block index structure, for example, the feature space is divided into 1000 subspaces, each subspace stores similar feature fingerprints. The system will dynamically adjust the similarity threshold according to the historical matching accuracy, such as the initial threshold is 0.8, and it is reduced to 0.75 when the accuracy decreases. Real-time matching is performed in the accessory information index tree to quickly locate the most similar accessory information.

[0082] The spatial constraint verification module constructs the spatial index structure of the warehouse's three-dimensional virtual model. The warehouse space is divided into grid units, each unit is 1m×1m×1m in size, and the cargo location rule information of the location is stored. For example, the 1st to 3rd floors of area A are used to store small items weighing less than 5kg, and the 4th to 6th floors of area B are used to store medium items weighing 5-20kg. The module receives the real-time spatial location coordinates of the accessories and performs real-time mapping and comparison with the virtual model. If the accessory is detected at the coordinates (5,3,2), the cargo location rules of the corresponding grid unit are searched for matching.

[0083] The early warning decision module comprehensively considers the probability distribution of the operation intention and the spatial position of the accessories to make a legal judgment. For example, if the probability of the detected intention to take is 0.9, and the current position of the accessories is (5,3,2), corresponding to the 2nd floor of area A, which meets the small item storage rules, it is determined to be a legal operation. If the probability of the detected intention to place is 0.85, but the position of the accessories is (10,8,5), corresponding to the 5th floor of area B, which does not meet the small item rules, an early warning is triggered. The system pushes early warning information to the operator, such as "Warning: Small accessories are placed on the 5th floor of area B by mistake." At the same time, in the 3D visualization interface, the specific location of the wrong operation (10,8,5) and related accessory information are highlighted in red.

[0084] The present invention realizes accurate recognition of operation intentions through deep learning models, and improves the recognition accuracy of complex action sequences by combining the temporal attention mechanism, effectively reducing the misjudgment rate; adopts binary feature fingerprints and local sensitive hashing algorithms to realize rapid matching of massive accessory information, greatly improving the system response speed and meeting the needs of real-time monitoring; based on three-dimensional virtual models and spatial index structures, it realizes accurate spatial constraint inspection in complex warehouse environments, and combines comprehensive decision-making with multi-dimensional information to significantly improve the accuracy and timeliness of early warning.

[0085] Figure 2FIG. 1 is a schematic diagram of a structure of a wrong-taken or wrong-placed warning system applicable to a drone parts warehouse according to an embodiment of the present invention. Figure 2 As shown, the system includes: a first unit, which is used to obtain the real-time operation image data of the operator through the image acquisition device deployed in the warehouse, collect the location data of the accessories through the distributed sensor network, standardize the collected real-time operation image data and location data, and generate the accessory operation data stream; a second unit, which is used to input the accessory operation data stream into the deep learning recognition model, identify the operation behavior type of the operator and the feature information of the accessories, calculate the real-time spatial position coordinates of the accessories by using the three-dimensional spatial positioning algorithm combined with the multi-sensor data fusion technology, and input the identified operation behavior type, the feature information of the accessories and the real-time spatial position coordinates of the accessories into the early warning analysis model; a third unit, which is used for the early warning analysis model to judge the operation intention of the operator based on the identified operation behavior type, match the feature information of the accessories with the preset accessory information database in real time, and map and compare the real-time spatial position coordinates of the accessories with the three-dimensional virtual model of the warehouse in real time. When it is detected that the operator's accessory picking behavior or placement behavior is inconsistent with the preset cargo location rules in the three-dimensional virtual model of the warehouse, a warning information is pushed to the operator, and the specific location and accessory information of the wrong operation are marked in the three-dimensional visualization interface.

[0086] According to a third aspect of an embodiment of the present invention, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0087] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0088] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for early warning of wrong taking and wrong placing of UAV parts in a warehouse, characterized in that: include: The real-time operation image data of the operators is obtained through the image acquisition device deployed in the warehouse, and the location data of the accessories is collected through the distributed sensor network. The collected real-time operation image data and location data are standardized to generate the accessory operation data stream; Input the accessory operation data stream into the deep learning recognition model to identify the operator's operation behavior type and accessory feature information, use the three-dimensional spatial positioning algorithm combined with multi-sensor data fusion technology to calculate the real-time spatial position coordinates of the accessories, and input the identified operation behavior type, accessory feature information and real-time spatial position coordinates of the accessories into the early warning analysis model; The early warning analysis model determines the operator's operating intention based on the identified operation behavior type, matches the accessory feature information with the preset accessory information database in real time, and maps and compares the real-time spatial position coordinates of the accessories with the three-dimensional virtual model of the warehouse in real time. When it is detected that the operator's accessory picking or placing behavior does not conform to the preset cargo location rules in the three-dimensional virtual model of the warehouse, the early warning information is pushed to the operator, and the specific location and accessory information of the wrong operation are marked in the three-dimensional visualization interface; The steps of calculating the attention weight matrix through the spatiotemporal attention module and dynamically fusing the features of the RGB image branch and the skeleton sequence branch to obtain the spatiotemporal fusion features, and inputting the spatiotemporal fusion features into the temporal modeling network to extract the long-range dependency relationship and obtain the operation behavior type include: The image features extracted by the RGB image branch are rearranged in time dimension to obtain a time feature map, and the time feature map is processed by a time self-attention calculation unit to obtain a time attention weight matrix, and the time self-attention calculation unit includes a time query weight matrix and a time key value weight matrix; the human skeleton features extracted by the skeleton sequence branch are rearranged in space dimension to obtain a spatial feature map, and the spatial feature map is processed by a spatial self-attention calculation unit to obtain a spatial attention weight matrix, and the spatial self-attention calculation unit includes a spatial query weight matrix and a spatial key value weight matrix; The image features and the human skeleton features are mapped to the same dimensional space through a feature mapping layer to obtain RGB mapping features and skeleton mapping features, and the RGB mapping features and the skeleton mapping features are processed by a multi-layer perceptron to obtain RGB feature scores and skeleton feature scores, and adaptive fusion weights are generated based on the RGB feature scores and the skeleton feature scores; The temporal attention weight matrix is ​​fused with the image features to obtain a temporal enhanced image feature, the spatial attention weight matrix is ​​fused with the human skeleton features to obtain a spatial enhanced skeleton feature, and the temporal enhanced image feature and the spatial enhanced skeleton feature are dynamically fused using the adaptive fusion weight to obtain a spatiotemporal fusion feature; The spatiotemporal fusion features are input into a time series modeling network, wherein the time series modeling network includes a multi-scale feature extraction module, a feature aggregation module and a residual connection module. The multi-scale feature extraction module extracts multi-scale time series features in parallel using one-dimensional convolutions of different convolution kernel sizes. The feature aggregation module dynamically fuses the multi-scale time series features through an attention mechanism. The residual connection module performs a residual connection between the fused features and the input features to extract long-range dependencies and obtain the operation behavior type. The early warning analysis model determines the operator's operating intention based on the identified operation behavior type, matches the accessory feature information with the preset accessory information database in real time, and maps and compares the real-time spatial position coordinates of the accessory with the three-dimensional virtual model of the warehouse in real time. When it is detected that the operator's accessory picking behavior or placement behavior is inconsistent with the preset cargo location rules in the three-dimensional virtual model of the warehouse, the early warning information is pushed to the operator, and the specific location and accessory information of the wrong operation are marked in the three-dimensional visualization interface. The steps include: The early warning analysis model includes an operation intention recognition module, a feature matching module, a space constraint verification module and an early warning decision module; The operation intention recognition module is used to input the operation behavior feature vector including action type, duration, and acceleration into a bidirectional long short-term memory network model, wherein the bidirectional long short-term memory network model includes a hidden layer structure for capturing the dependencies between action sequences and a time attention mechanism for calculating the importance weights of time steps, and outputs the probability distribution of the operator's intention to take and the intention to place through a softmax classifier; The feature matching module is used to match the accessory feature information with a preset accessory information database in real time, map the size, weight, and material features of the accessory into a binary feature fingerprint, use a local sensitive hashing algorithm based on a feature weight vector for feature matching, implement fast retrieval of binary feature fingerprints through a block index structure, dynamically adjust the similarity threshold based on historical matching accuracy, and perform real-time feature matching in the accessory information index tree; The spatial constraint checking module is used to construct a spatial index structure of a three-dimensional virtual model of the warehouse, divide the warehouse space into grid units for storing cargo location rule information, and perform real-time mapping and comparison between the real-time spatial position coordinates of the accessories and the three-dimensional virtual model of the warehouse; The warning decision module is used to make a legality judgment based on the probability distribution of the operator's operating intention and the real-time spatial position of the accessories. When it is detected that the accessory picking or placement behavior is inconsistent with the cargo space rules, a warning message is pushed to the operator, and the specific location and accessory information of the erroneous operation are marked in the three-dimensional visualization interface.

2. The method according to claim 1, characterized in that The steps of acquiring real-time operation image data of operators through image acquisition devices deployed in the warehouse, acquiring the location data of accessories through a distributed sensor network, and standardizing the acquired real-time operation image data and location data to generate the accessory operation data stream include: High-definition camera arrays are placed on the top and sides of the shelves to obtain real-time image data of operators. When operators are detected entering the working area, high-frame rate acquisition mode is triggered and target tracking is performed. UWB positioning base stations are placed at the four corners of the shelves to use time difference positioning algorithms to obtain the location data of accessories. The collected real-time operation image data is subjected to distortion correction, illumination compensation and image enhancement processing, and the position data is subjected to Kalman filtering. The processed real-time operation image data and position data are segmented using the sliding window method, and the segmented data are time-series aligned based on the network time protocol. After wavelet denoising and standardization, an accessory operation data stream containing timestamp, image data and position data is generated.

3. The method according to claim 1, characterized in that The steps of inputting the accessory operation data stream into the deep learning recognition model, identifying the operator's operation behavior type and accessory feature information, and using the three-dimensional spatial positioning algorithm combined with multi-sensor data fusion technology to calculate the real-time spatial position coordinates of the accessory include: Input the accessory operation data stream into a deep learning recognition model, the deep learning recognition model includes a two-stream network structure, the RGB image branch of the two-stream network structure uses a three-dimensional convolutional residual network to extract image features, and the skeleton sequence branch uses a spatiotemporal graph convolutional network to extract human skeleton features, calculate the attention weight matrix through the spatiotemporal attention module, and dynamically fuse the features of the RGB image branch and the skeleton sequence branch to obtain spatiotemporal fusion features, and input the spatiotemporal fusion features into the temporal modeling network to extract long-range dependencies to obtain the operation behavior type; The deep learning recognition model further includes a multi-scale feature pyramid network, which fuses convolutional features with upsampled high-level features at different scale levels to obtain multi-scale fused features, applies an attention mechanism to the multi-scale fused features to generate an attention map, and extracts accessory instance-level feature vectors through an instance alignment layer to obtain accessory feature information; The image data in the accessory operation data stream is processed through a deep learning recognition model to obtain the visual positioning result, and the ultra-wideband ranging value and radio frequency signal strength index are obtained through a distributed sensor network. The three-dimensional spatial positioning algorithm is combined with multi-sensor data fusion technology, and the visual positioning, ultra-wideband ranging and radio frequency signal strength are fused through the extended Kalman filter to obtain the real-time spatial position coordinates of the accessory.

4. The method according to claim 3, characterized in that The deep learning recognition model further includes a multi-scale feature pyramid network, and the steps of fusing convolutional features with upsampled high-level features at different scale levels to obtain multi-scale fused features include: The multi-scale feature pyramid network includes five feature levels, and the spatial resolution of each feature level is reduced by two times, wherein the spatial resolution of the first feature level is the highest, and the spatial resolution of the fifth feature level is the lowest. Each feature level is subjected to feature extraction through a convolutional layer with a step length to obtain basic features, wherein the convolutional layer with a step length includes a plurality of convolution kernels, and the step length of each convolution kernel is 2, which is used to downsample and extract features from input features to obtain basic features corresponding to each level; Performing cross-layer feature fusion on the basic features, including: upsampling the high-level features through a deconvolution operation to obtain upsampled features, wherein the spatial resolution of the upsampled features is the same as the spatial resolution of the adjacent low-level features; A 1×1 convolutional layer is used to generate low-level feature adaptive weights for low-level features, and a 1×1 convolutional layer is used to generate up-sampled feature adaptive weights for up-sampled features; the low-level feature adaptive weights are multiplied by the low-level features to obtain weighted low-level features, the up-sampled feature adaptive weights are multiplied by the up-sampled features to obtain weighted up-sampled features, and the weighted low-level features and the weighted up-sampled features are weightedly summed to obtain multi-scale fusion features.

5. The method according to claim 3, characterized in that: The steps of using the three-dimensional spatial positioning algorithm combined with multi-sensor data fusion technology to fuse visual positioning, ultra-wideband ranging and radio frequency signal strength through extended Kalman filtering to obtain the real-time spatial position coordinates of the accessories include: Establishing a state vector including the three-dimensional position and velocity of the accessory, the state vector includes a position component and a velocity component, and constructing a state transfer equation based on a uniform motion model, the state transfer equation describing the evolution relationship between the position and velocity of the accessory through a state transfer matrix; Constructing a multi-sensor observation equation and acquiring multi-sensor data, wherein the multi-sensor observation equation includes a visual positioning observation model, an ultra-wideband ranging observation model, and a radio frequency signal strength observation model, wherein the visual positioning observation model directly acquires the three-dimensional coordinates of the accessory, the ultra-wideband ranging observation model calculates the ranging value based on the accessory position and the base station position, and the radio frequency signal strength observation model establishes a mapping relationship between signal strength and distance based on a path loss index, and the multi-sensor data includes visual positioning data, ultra-wideband ranging data, and radio frequency signal strength data; An adaptive factor is introduced to dynamically weight the multi-sensor data, wherein the adaptive factor calculates the confidence of each sensor data based on the observation residual, and uses the confidence to update the observation noise covariance matrix; The state vector is recursively estimated using an extended Kalman filter framework, a priori state estimation is calculated based on the state transfer equation in the prediction phase, and the priori state estimation is corrected using the multi-sensor observation equation and the adaptive factor in the update phase to obtain the real-time spatial position coordinates of the accessory.

6. A misplacement warning system applicable to a drone parts warehouse, used to implement the method described in any one of claims 1 to 5, characterized in that: include: The first unit is used to obtain real-time operation image data of operators through image acquisition devices deployed in the warehouse, collect the location data of accessories through a distributed sensor network, standardize the collected real-time operation image data and location data, and generate an accessories operation data stream; The second unit is used to input the accessory operation data stream into the deep learning recognition model, identify the operator's operation behavior type and accessory feature information, use the three-dimensional spatial positioning algorithm combined with multi-sensor data fusion technology to calculate the real-time spatial position coordinates of the accessory, and input the identified operation behavior type, accessory feature information and the real-time spatial position coordinates of the accessory into the early warning analysis model; The third unit is used for the early warning analysis model to judge the operating intention of the operator based on the identified operating behavior type, to match the accessory feature information with the preset accessory information database in real time, and to map and compare the real-time spatial position coordinates of the accessory with the three-dimensional virtual model of the warehouse in real time. When it is detected that the operator's accessory picking or placing behavior is inconsistent with the preset cargo location rules in the three-dimensional virtual model of the warehouse, a warning message is pushed to the operator, and the specific location and accessory information of the erroneous operation are marked in the three-dimensional visualization interface.

7. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 5.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images