Method, device and medium for identifying and evaluating warehouse logistics operation actions

Abstract

The application provides a warehouse logistics operation action recognition and evaluation method, device, equipment and medium, and relates to the field of image processing. The method comprises the following steps: acquiring a video stream of a warehouse area, performing skeleton feature extraction on the video stream to obtain skeleton data, and performing frame extraction on the video stream to obtain a frame sequence; determining joint data according to the skeleton data, performing spatio-temporal feature extraction on the joint data to obtain global spatio-temporal features; performing convolution operations on the frame sequence through a common convolution layer and a dilated convolution layer respectively to obtain first local features and second local features, and performing feature fusion on the first local features and the second local features to obtain local fusion features; determining the operation action of a target operator according to the global spatio-temporal features and the local fusion features, and evaluating the operation action of the target operator.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to a warehouse logistics operation action recognition and evaluation method and device, computer equipment and readable storage medium. BACKGROUND

[0002] With the rapid development of e-commerce and the increasing diversification of consumer demand, the warehouse logistics industry is facing great challenges and opportunities. Technological progress, such as automation, artificial intelligence, the Internet of Things, and big data analysis, provides new possibilities for the optimization of warehouse logistics. These technologies not only improve the efficiency and accuracy of warehouse operations, but also enhance the ability to monitor and manage inventory in real time.

[0003] In related technologies, action recognition of operation personnel in warehouse logistics is an important technical means to improve efficiency, ensure safety, and optimize management. However, the current action recognition accuracy of operation personnel in the warehouse is low. SUMMARY

[0004] Therefore, the present application provides a warehouse logistics operation action recognition and evaluation method and device, computer equipment and readable storage medium, which solves the problem of low action recognition accuracy of operation personnel in the warehouse.

[0005] In a first aspect, the present application provides a warehouse logistics operation action recognition and evaluation method, comprising:

[0006] obtaining a video stream of a warehouse area, extracting skeleton features from the video stream to obtain skeleton data, and extracting frames from the video stream to obtain a frame sequence;

[0007] determining joint data from the skeleton data, extracting global spatio-temporal features from the joint data;

[0008] performing convolution operations on the frame sequence through ordinary convolution layers and dilated convolution layers to obtain first local features and second local features, and performing feature fusion on the first local features and the second local features to obtain local fusion features;

[0009] determining the operation action of the target operation personnel according to the global spatio-temporal features and the local fusion features, and evaluating the operation action of the target operation personnel.

[0010] In a second aspect, the present application provides a warehouse logistics operation action recognition and evaluation device, comprising:

[0011] a data preprocessing module for obtaining a video stream of a warehouse area, extracting skeleton features from the video stream to obtain skeleton data, and extracting frames from the video stream to obtain a frame sequence;

[0012] a global feature extraction module configured to determine joint data according to the skeleton data, and perform spatio-temporal feature extraction on the joint data to obtain global spatio-temporal features;

[0013] a local feature extraction module configured to perform convolution operations on the frame sequence through a normal convolution layer and a dilated convolution layer respectively to obtain first local features and second local features, and perform feature fusion on the first local features and the second local features to obtain local fusion features;

[0014] a work action evaluation module configured to determine work actions of a target work personnel according to the global spatio-temporal features and the local fusion features, and evaluate the work actions of the target work personnel.

[0015] In a third aspect, an embodiment of the present application provides a computer device, which comprises a processor and a memory. The memory stores programs or instructions executable on the processor. When the programs or instructions are executed by the processor, the steps of the method according to the first aspect are implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a readable storage medium, which stores programs or instructions. When the programs or instructions are executed by a processor, the steps of the method according to the first aspect are implemented.

[0017] The warehouse logistics work action recognition and evaluation method, device, computer device and readable storage medium provided by the present application, by acquiring a video stream of a warehouse area, performing feature extraction on the video stream to obtain skeleton data, and frame extraction on the video stream to obtain a frame sequence, determining joint data according to the skeleton data and performing spatio-temporal feature extraction to obtain global spatio-temporal features, performing convolution operations on the frame sequence respectively to obtain first local features and second local features and performing feature fusion to obtain local fusion features, and determining work actions according to the global spatio-temporal features and the local fusion features and evaluating work actions of a target work personnel. The present application, by acquiring a video stream of a warehouse area and performing skeleton feature extraction and frame extraction, ensures the accuracy and processing efficiency of feature extraction, determines joint data and extracts global spatio-temporal features, captures long-term dynamics of actions, extracts local features through a normal convolution layer and a dilated convolution layer and performs fusion, enhances the perception ability of local motion details, determines work actions in combination with global features and local features and performs evaluation, and realizes accurate recognition and effective evaluation of work behaviors.

[0018] The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, the content of the specification can be implemented, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0020] Figure 1 A flowchart of a warehouse logistics operation action recognition and evaluation method according to an embodiment of the application is shown;

[0021] Figure 2 A structural block diagram of a warehouse logistics operation action recognition and evaluation device according to an embodiment of the application is shown;

[0022] Figure 3 A structural block diagram of a computer device according to an embodiment of the application is shown. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the application will be clearly described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some, but not all of the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art belong to the scope of protection of the application.

[0024] The terms "first", "second", and the like in the specification and claims of the application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second", etc. are usually a category and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / ", generally represents a "or" relationship between the front and rear associated objects.

[0025] The warehouse logistics operation action recognition and evaluation method, device, computer device and readable storage medium provided by the embodiments of the application will be described in detail below with reference to the drawings and specific embodiments and their application scenarios. In the case of no conflict, the embodiments described below and the features in the embodiments can be combined with each other.

[0026] The embodiments of the application provide a warehouse logistics operation action recognition and evaluation method, as shown in Figure 1 The method comprises:

[0027] Step 101, obtaining a video stream of a warehouse area, performing skeleton feature extraction on the video stream to obtain skeleton data, and performing frame extraction on the video stream to obtain a frame sequence;

[0028] In step 102, joint data is determined according to the skeleton data, and global spatio-temporal features are extracted from the joint data.

[0029] In step 103, a first local feature and a second local feature are obtained by performing convolution operations on the frame sequence through a normal convolution layer and a dilated convolution layer, respectively, and the first local feature and the second local feature are fused to obtain a local fusion feature.

[0030] In step 104, the operation action of the target operator is determined according to the global spatio-temporal features and the local fusion features, and the operation action of the target operator is evaluated.

[0031] In an implementation, a camera installed in the warehouse is used to obtain a video stream, and warehouse logistics operation action recognition and evaluation are realized based on machine vision. Machine vision is a technology that simulates human visual functions by using optical equipment and image processing technology. It captures images through a camera or other optical sensors, and then uses computer algorithms to process and analyze these images to achieve specific goals such as object recognition, detection, measurement, and positioning.

[0032] In an implementation, skeleton feature extraction is performed on the video stream to obtain skeleton data. Skeleton data is a simplified and abstract representation of the body posture of an operator. It describes the posture and action of the human body through the position and connection relationship of key points, rather than directly using pixel information in the video. This abstract representation has the following advantages: (1) reduces data volume. Skeleton data only contains coordinate information of key points, including points and edges. Compared with the original video stream, the data volume is greatly reduced, which is convenient for processing and analysis; (2) improves robustness: skeleton data has strong robustness to changes in lighting, background, and clothing, and can more stably describe human actions; (3) facilitates modeling: the structured form of skeleton data facilitates mathematical modeling and feature extraction, and is suitable for action recognition and analysis. Standardization processing is performed on the skeleton data to ensure that the coordinates of each joint are on the same scale. Standardization processing can eliminate the scale differences between different videos and improve the generalization ability of the model. Through standardization, the model can better process skeleton data from different data sources and reduce recognition errors caused by scale differences.

[0033] In an implementation, joint data is determined based on skeleton data, and the joint data includes joint topology and joint motion data, and global spatio-temporal features are further extracted, so that the action pattern of the operator can be understood as a whole. The extraction of joint data not only includes the topology structure, but also covers the motion information, providing rich details for comprehensive analysis of actions. The extraction of global spatio-temporal features helps to capture long-term dynamics of actions and enhances the understanding ability of complex actions.

[0034] In an implementation, frame sequences are obtained by frame decimation on a video stream. The video stream is divided into n non-overlapping segments, each segment containing a plurality of frames, and a plurality of adjacent frames are randomly sampled from each segment to form a frame sequence I * ={I1,I2,…,I n}. The video data has a time dimension. By dividing the segments and sampling the frames, the computational load can be effectively reduced while retaining key spatio-temporal information, reducing the demand for computing resources while retaining key information in the video, improving the efficiency and accuracy of the model.

[0035] In an implementation, the frame sequences are respectively subjected to convolution operations by a normal convolution layer and a dilated convolution layer to extract first local features and second local features, and the first local features and the second local features are fused to obtain local fusion features that can reflect local motion details and more extensive spatio-temporal relationships. This multi-scale feature fusion strategy not only enriches the feature representation, but also improves the model's perception of local motion, making the action recognition more accurate.

[0036] Further, the work action of the target work personnel is determined according to the global spatio-temporal features and the local fusion features, and the work action of the target work personnel is evaluated.

[0037] The embodiments of the present application provide a warehouse logistics work action recognition and evaluation method based on machine vision. By obtaining a video stream of a warehouse area and performing skeleton feature extraction and frame decimation operations, the accuracy and processing efficiency of feature extraction are ensured, joint data is determined and global spatio-temporal features are extracted, long-term dynamics of actions are captured, local features are extracted by a normal convolution layer and a dilated convolution layer and fused, the perception of local motion details is enhanced, the work action is determined in combination with the global features and the local features, and the work action is evaluated, thereby realizing accurate recognition and effective evaluation of work behavior.

[0038] In an embodiment of the present application, spatio-temporal feature extraction is performed on the joint data to obtain global spatio-temporal features, including:

[0039] A spatio-temporal topology graph is constructed according to the joint data;

[0040] A topology feature is extracted from the spatio-temporal topology graph by adaptive graph convolution, and a time sequence feature is extracted from the spatio-temporal topology graph by time domain dilated convolution;

[0041] The topology feature and the time sequence feature are fused to obtain a time sequence topology feature, a global relationship feature is extracted from the time sequence topology feature by a spatial transformer, and a global spatio-temporal feature is extracted from the global relationship feature by a temporal transformer.

[0042] In this embodiment, the spatio-temporal data of the skeleton joints are converted into a graph structure, i.e., a spatio-temporal topology graph, which explicitly represents the spatial connections between joints, e.g., elbow and shoulder, and the temporal continuity of the same joint in consecutive frames. Exemplarily, in the "running" action, the spatio-temporal graph can represent the swing trajectory of the leg joints (temporal edges) and the linkage relationship of the torso joints (spatial edges), providing a structured input for subsequent feature extraction.

[0043] The topology features are extracted from the spatio-temporal topology graph by adaptive graph convolutional network (AGCN), which can capture potential joint relationships beyond the natural anatomical connections, such as action-related but not directly connected joints. Exemplarily, in the "waving hand" action, the adaptive graph convolution can identify the dynamic association between the hand and the shoulder, even if they are not directly connected, enhancing the representation ability of the local topology features.

[0044] The temporal features are extracted from the spatio-temporal topology graph by temporal dilation convolutional network (TDCN), which uses multi-scale dilation convolution to expand the temporal receptive field and capture long-distance temporal dependencies, such as the persistence and periodicity of the action. Exemplarily, in the "walking" action, the temporal dilation convolution can model the continuous swing pattern of the legs from stepping to landing, avoiding the loss of long-term temporal information by short-time convolution.

[0045] The topology features and the temporal features are fused to obtain the temporal topology features. The global relationship features are extracted from the temporal topology features by a spatial transformer, which acts on the temporal topology features within a single frame and dynamically models the global dependency relationships between all joints through a self-attention mechanism, breaking the limitations of traditional graph structures on natural anatomical connections. For example, in the "carrying heavy objects" action, even if the hand and the waist are not directly connected, the spatial Transformer can still capture the collaborative changes between them due to mechanical balance, identify the correlation strength of non-adjacent joints (such as calculating the attention weight of the hand and the waist), and thus extract global relationship features reflecting the whole-body coordination.

[0046] The global relationship features are extracted by a time transformer to obtain global spatio-temporal features. The time transformer (also referred to as a time Transformer) analyzes the cross-frame dynamic evolution of the same joint along the time dimension based on the global relationship features output by the spatial Transformer, and captures long-term time sequence patterns of the action, such as the starting, continuing, and ending stages of the action. For example, in the “continuous bending” scenario, the time Transformer compares the angle change trends of the spine joint in different frames, identifies posture deterioration caused by fatigue accumulation, such as the bending amplitude increasing frame by frame, and finally generates global spatio-temporal features that fuse the spatio-temporal continuity, thereby providing complete time sequence basis for action risk assessment.

[0047] In the embodiments of the present application, the spatio-temporal association between joints, such as trunk linkage and time trajectory, is explicitly modeled by constructing a spatio-temporal topology graph, the action-related relationship outside the potential anatomical connection is dynamically captured by using an adaptive graph convolution, such as the dynamic association of the hand shoulder when waving the hand, and the periodic long-range action pattern, such as the walking gait cycle, is accurately modeled by using a time domain dilated convolution to break through the short-time dependence limitation, thereby solving the problem of incomplete modeling or incorrect recognition in the recognition process caused by the inability to completely model short-time actions in the related art. Finally, through hierarchical feature fusion of the spatio-temporal Transformer, progressive abstraction from local topology to global spatio-temporal relationship is realized, which significantly enhances the modeling capability of the model for complex motion patterns such as multi-joint coordination of running, and solves the limitations of traditional methods in terms of spatial sparse connection and temporal long-range dependence.

[0048] In an embodiment of the present application, a convolution operation is performed on the frame sequence by a normal convolution layer and a dilated convolution layer to obtain first local features and second local features, including:

[0049] A convolution operation is performed on the frame sequence by a normal convolution layer to obtain first motion features, and a convolution operation is performed on the first motion features to obtain first local features;

[0050] A convolution operation is performed on the frame sequence by a dilated convolution layer to obtain second motion features, and a max-pooling operation is performed on the second motion features to obtain second local features.

[0051] In this embodiment, a normal convolution layer, for example, a 3D convolution layer of 3x3x3, is used to perform a convolution operation on the frame sequence to extract first motion features F1, and a normal convolution layer is used again to perform a convolution operation on the first motion features F1 to obtain refined first local features F1', that is, detailed features.

[0052] By using a 3D convolution layer of 3x3x3 to perform two convolution operations on the frame sequence, first motion features F1 and first local features F1' are obtained, which can effectively extract detailed information of local motion. This operation mode can capture subtle motion changes between adjacent frames, thereby providing rich local motion features for subsequent action recognition. This fine local motion feature extraction is crucial for understanding the dynamic changes of actions, especially when dealing with complex actions, which can significantly improve the model's perception of local motion.

[0053] Using a dilated convolution layer, such as a 3x3x3 convolution layer with a dilation rate of 2, to perform convolution operations on the frame sequence, motion features with a larger receptive field, i.e., second motion features F2, are obtained. The second motion features F2 are subjected to a max-pooling operation in the spatial dimension to obtain down-sampled second local features F2', i.e., detailed features.

[0054] By using a 3x3x3 dilated convolution layer with a dilation rate of 2 to perform convolution operations on the frame sequence, second motion features F2 are obtained, and further through a max-pooling operation, second local features F2' are obtained. The use of dilated convolution layers effectively expands the receptive field of the convolution kernel, enabling the model to capture motion information over a larger range. This operation not only preserves the details of local motion but also captures more extensive spatio-temporal relationships, thereby providing a more comprehensive representation of motion features for action recognition. This expanded receptive field feature extraction method helps the model better understand the overall dynamics of actions, especially when dealing with long temporal actions, which can significantly improve recognition accuracy.

[0055] In an embodiment of the present application, the first local features and the second local features are fused to obtain local fusion features, including:

[0056] The first local features and the second local features are fused to obtain first fusion features;

[0057] The first fusion features and appearance features obtained by feature extraction of the frame sequence through a ResNet-50 backbone network are fused to obtain second fusion features;

[0058] The second fusion features are subjected to feature extraction to obtain spatial features, and the second fusion features are modeled through a 3D convolution layer to obtain weight features;

[0059] The second fusion features are subjected to an average pooling operation in the time dimension to obtain spatial descriptors, the spatial descriptors are modeled through a 2D convolution layer to obtain spatial information, and the spatial information is activated through a Sigmoid function to obtain spatial weight features;

[0060] The average pooling operation is performed on the spatial dimension of the second fusion feature to obtain a time descriptor, the time descriptor is modeled by a 1D convolution layer to obtain time information, and the time information is activated by a Sigmoid function to obtain a time weight feature;

[0061] The spatial feature, the weight feature, the spatial weight feature, and the time weight feature are spliced to obtain a local fusion feature.

[0062] In this embodiment, the first local feature F1' and the second local feature F2' are fused to obtain a first fusion feature. This fusion method combines motion features of different scales, enabling the model to capture both local details and global dynamics. In addition, the appearance feature is extracted from the frame sequence by the ResNet-50 backbone network, and the extracted appearance feature is further fused with the first fusion feature to obtain a second fusion feature. This multi-scale feature fusion strategy not only enriches the feature representation, but also improves the model's perception of motion information of different scales, effectively improving the model's robustness and accuracy in processing complex actions, especially when facing actions of different speeds and scales, significantly improving the recognition performance.

[0063] In an implementation manner, the ResNet-50 is a deep residual convolutional neural network. By introducing a residual block (Residual Block) and a skip connection (Skip Connection), the gradient vanishing problem in the training of a deep network is solved, and high-level semantic features of an image can be efficiently extracted. In this process, the ResNet-50 extracts features from the video frame sequence, and captures global appearance information (such as color, texture, shape) and contextual semantic information (such as human pose, tool shape) of objects in the image through its 50-layer network structure (including 49 convolutional layers and 1 fully connected layer).

[0064] The second fusion feature is extracted to obtain a spatial feature, and the second fusion feature is modeled by a 3D convolution layer to obtain a weight feature. In an implementation manner, the "D" in the 3D convolution layer refers to the depth, that is, the size of the convolution kernel (or filter) in the time dimension or the depth dimension. The spatial feature is obtained by extracting the second fusion feature, and the weight feature is obtained by modeling these features by a 3D convolution layer. This operation can effectively capture the spatial information of each frame in the video, and through the modeling of the 3D convolution layer, the expression ability of these spatial features is further enhanced. This enhanced spatial feature extraction method helps the model better understand the static information of each frame in the video, thereby providing more rich spatial context for subsequent action recognition.

[0065] In machine learning and deep learning, modeling refers to learning and expressing the characteristics of data through mathematical methods and algorithms, so as to extract useful information therefrom. For example, modeling the characteristics through a convolutional layer can extract spatial or temporal characteristics in the data. Spatial descriptor: the spatial descriptor is a concise representation of the spatial information of each frame in the video. By performing an average pooling operation in the time dimension on the second fused feature, the spatial descriptor of each frame can be obtained, which contains the information of all spatial positions in the frame. Temporal descriptor: the temporal descriptor is a concise representation of the temporal information of each frame in the video. By performing an average pooling operation in the spatial dimension on the second fused feature, the temporal descriptor of each frame can be obtained, which contains the dynamic change information of the frame in the time sequence.

[0066] In an implementation, the spatial descriptor is obtained by performing an average pooling operation in the time dimension on the second fused feature, and then the spatial descriptor is modeled through a 2D convolutional layer and activated through a Sigmoid function to obtain a spatial weight feature. In this way, the importance of different spatial regions in each frame can be dynamically adjusted, so that the model can pay more attention to the regions that are more meaningful for action recognition. This dynamic adjustment mechanism of spatial weight improves the perception ability of the model to key spatial regions, thereby enhancing the accuracy of action recognition.

[0067] In an implementation, the temporal descriptor is obtained by performing an average pooling operation in the spatial dimension on the second fused feature, and then the temporal descriptor is modeled through a 1D convolutional layer and activated through a Sigmoid function to obtain a temporal weight feature. In this way, the importance of different time points can be dynamically adjusted, so that the model can pay more attention to the time periods that are more meaningful for action recognition. This dynamic adjustment mechanism of temporal weight improves the perception ability of the model to key time points, thereby enhancing the accuracy of action recognition.

[0068] Further, the spatial feature, the weight feature, the spatial weight feature and the temporal weight feature are spliced to obtain more rich local fused features. This multi-dimensional feature fusion method integrates the information of the spatial and temporal dimensions, so that the model can more comprehensively understand the action features in the video. This fusion method not only enriches the feature representation, but also improves the perception ability of the model to different dimensional information, thereby significantly improving the performance of action recognition.

[0069] In an embodiment of the present application, the evaluation of the operation action of the target operator includes:

[0070] The similarity between the operation action and the standard action is calculated to obtain a risk score;

[0071] If the risk score is less than a first threshold, the target operator is determined to be a standard operator.

[0072] If the risk score is greater than or equal to the first threshold value, the target operator is determined to be a high-risk operation.

[0073] In this embodiment, the risk score is obtained by similarity calculation between the operation action and the standard action, which can evaluate whether the operation action of the operator conforms to the standard in real time, ensure that potential high-risk behaviors are found in the operation process in real time, and thus timely measures are taken to reduce the risk of accidents. Real-time risk assessment improves the safety of the operation site and ensures that the operator can work in a safe environment.

[0074] In an implementation manner, the similarity calculation is performed by quantifying the similarity between the two actions, for example: (1) first extract the key features from the operation action and the standard action, such as the angle, speed, and acceleration of the action, and evaluate the similarity by calculating the difference between these features. Common methods include Euclidean distance, cosine similarity, etc.; (2) use a deep learning model such as a convolutional neural network or a recurrent neural network to learn the feature representation of the action, and calculate the similarity through the feature vector output by the model. This method can automatically learn the complex features of the action and improve the accuracy of similarity calculation.

[0075] In an embodiment of the present application, the method further comprises: in the case where the target operator is determined to be a high-risk operation, performing a target operation, the target operation comprising at least one of the following: issuing an alarm, suspending the operation equipment, recording high-risk operation information, providing safety prompts and operation guidance.

[0076] In this embodiment, when a high-risk operation behavior is determined, in addition to issuing an alarm or suspending the operation equipment, the following operations can also be taken: (1) automatically record detailed information of the high-risk operation, including time, location, operator identity, and specific action, etc., and generate a safety report. The safety report can be used for subsequent safety analysis and accident investigation; (2) provide immediate safety prompts and operation guidance to the operator, such as voice guidance, and report the risk to the management to help the operator correct non-standard actions and avoid further risks.

[0077] As a specific implementation of the above-mentioned identification and evaluation method of warehouse logistics operation actions, an embodiment of the present application provides an identification and evaluation device for warehouse logistics operation actions. As shown in Figure 2 The identification and evaluation device for warehouse logistics operation actions 200 comprises a data preprocessing module 201, a global feature extraction module 202, a local feature extraction module 203, and an operation action evaluation module 204.

[0078] The data preprocessing module 201 is configured to obtain a video stream of a warehouse area, perform skeleton feature extraction on the video stream to obtain skeleton data, and perform frame extraction on the video stream to obtain a frame sequence.

[0079] The global feature extraction module 202 is configured to determine joint data according to the skeleton data, and perform spatio-temporal feature extraction on the joint data to obtain global spatio-temporal features.

[0080] The local feature extraction module 203 is configured to perform convolution operations on the frame sequence through a normal convolution layer and a dilated convolution layer respectively to obtain first local features and second local features, and perform feature fusion on the first local features and the second local features to obtain local fusion features.

[0081] The work action evaluation module 204 is configured to determine work actions of a target work personnel according to the global spatio-temporal features and the local fusion features, and evaluate the work actions of the target work personnel.

[0082] Further, the joint data includes joint topology and joint motion data.

[0083] Further, the global feature extraction module 202 is specifically configured to:

[0084] construct a spatio-temporal topology graph according to the joint data;

[0085] extract topology features from the spatio-temporal topology graph through adaptive graph convolution, and extract time sequence features from the spatio-temporal topology graph through time domain dilated convolution;

[0086] fuse the topology features and the time sequence features to obtain time sequence topology features, extract global relationship features from the time sequence topology features through a spatial transformer, and extract global spatio-temporal features from the global relationship features through a temporal transformer.

[0087] Further, the local feature extraction module 203 is specifically configured to:

[0088] perform a convolution operation on the frame sequence through a normal convolution layer to obtain first motion features, and perform a convolution operation on the first motion features again to obtain first local features;

[0089] perform a convolution operation on the frame sequence through a dilated convolution layer to obtain second motion features, and perform a maximum pooling operation on the second motion features to obtain second local features.

[0090] Further, the local feature extraction module 203 is specifically configured to:

[0091] fuse the first local features and the second local features to obtain first fusion features;

[0092] extract appearance features from the frame sequence through a ResNet-50 backbone network, and fuse the first fusion features and the appearance features to obtain second fusion features.

[0093] perform feature extraction on the second fusion feature to obtain a spatial feature, and model the second fusion feature through a 3D convolution layer to obtain a weight feature;

[0094] perform an average pooling operation in a time dimension on the second fusion feature to obtain a spatial descriptor, model the spatial descriptor through a 2D convolution layer to obtain spatial information, and activate the spatial information through a Sigmoid function to obtain a spatial weight feature;

[0095] perform an average pooling operation in a spatial dimension on the second fusion feature to obtain a time descriptor, model the time descriptor through a 1D convolution layer to obtain time information, and activate the time information through a Sigmoid function to obtain a time weight feature;

[0096] concatenate the spatial feature, the weight feature, the spatial weight feature and the time weight feature to obtain a local fusion feature.

[0097] Further, the operation action evaluation module 204 is specifically configured to:

[0098] perform similarity calculation on the operation action and the standard action to obtain a risk score;

[0099] if the risk score is less than a first threshold value, determine that the target operation personnel is a standard operation;

[0100] if the risk score is greater than or equal to the first threshold value, determine that the target operation personnel is a high-risk operation.

[0101] Further, the device further includes:

[0102] an execution module, configured to: in a case where the target operation personnel is determined to be a high-risk operation, execute a target operation, the target operation including at least one of the following: issuing an alarm, suspending an operation device, recording high-risk operation information, providing a safety prompt and operation guidance.

[0103] The warehouse logistics operation action recognition and evaluation device 200 in the embodiment of the application can be a computer device, or a component in a computer device, such as an integrated circuit or a chip. The warehouse logistics operation action recognition and evaluation device 200 provided in the embodiment of the application can implement each process of the warehouse logistics operation action recognition and evaluation method embodiment, and thus repeated descriptions are omitted here. Figure 1

[0104] The embodiment of the application further provides a computer device, such as a server or a terminal. Figure 3 ​As shown, the computer device 300 includes a processor 301 and a memory 302, and the memory 302 stores a program or instruction executable on the processor 301, which, when executed by the processor 301, implements each step of the above-mentioned warehouse logistics operation action identification and evaluation method embodiment and achieves the same technical effects. To avoid repetition, details are not described here.

[0105] The memory 302 can be used to store software programs and various data. The memory 302 can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. In addition, the memory 302 can include a volatile memory or a non-volatile memory, or the memory 302 can include both volatile and non-volatile memories. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 302 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0106] The processor 301 can include one or more processing units; optionally, the processor 301 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 301.

[0107] The embodiment of the present application further provides a readable storage medium, and the readable storage medium stores programs or instructions, the programs or instructions are executed by a processor to realize each process of the warehouse logistics operation action recognition and evaluation method embodiment, and the same technical effects can be achieved. To avoid repetition, details are not described herein.

[0108] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, method, article, or apparatus that includes a list of elements not only includes those elements, but also includes other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element. In addition, it should be pointed out that the scope of the methods and apparatus in the embodiments of the present application is not limited to the order of performing the functions as shown or discussed, but can also include performing the functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method can be performed in an order different from that described, and various steps can also be added, omitted, or combined. In addition, the features described with reference to certain examples can be combined in other examples.

[0109] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above specific embodiments, and the above specific embodiments are only illustrative and not restrictive. Those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope of protection of the claims, and all forms belong to the protection of the present application.

Claims

1. A method of identifying and evaluating a warehouse logistics operation action, characterized by, The method comprises the following steps: acquiring a video stream of a storage area, performing skeleton feature extraction on the video stream to obtain skeleton data, and performing frame extraction on the video stream to obtain a frame sequence; determining joint data according to the skeleton data, performing spatio-temporal feature extraction on the joint data to obtain global spatio-temporal features; performing convolution operations on the frame sequence through a normal convolution layer and a dilated convolution layer respectively to obtain first local features and second local features, and performing feature fusion on the first local features and the second local features to obtain local fusion features; determining the work action of a target work personnel according to the global spatio-temporal features and the local fusion features, and evaluating the work action of the target work personnel; wherein the feature fusion of the first local features and the second local features to obtain local fusion features comprises: fusing the first local features and the second local features to obtain first fusion features; performing feature extraction on the frame sequence through a ResNet-50 backbone network to obtain appearance features, fusing the first fusion features and the appearance features to obtain second fusion features; performing feature extraction on the second fusion features to obtain spatial features, modeling the second fusion features through a 3D convolution layer to obtain weight features; performing average pooling operation on the second fusion features in the time dimension to obtain a spatial descriptor, modeling the spatial descriptor through a 2D convolution layer to obtain spatial information, and activating the spatial information through a Sigmoid function to obtain a spatial weight feature; performing average pooling operation on the second fusion features in the spatial dimension to obtain a time descriptor, modeling the time descriptor through a 1D convolution layer to obtain time information, and activating the time information through a Sigmoid function to obtain a time weight feature; splicing the spatial features, the weight features, the spatial weight features, and the time weight features to obtain the local fusion features.

2. The method of claim 1, wherein: the joint data comprises joint topology and joint motion data.

3. The method of claim 1, wherein, the spatio-temporal feature extraction on the joint data to obtain global spatio-temporal features comprises: constructing a spatio-temporal topology graph according to the joint data; performing feature extraction on the spatio-temporal topology graph through adaptive graph convolution to obtain topology features, and performing feature extraction on the spatio-temporal topology graph through time domain dilated convolution to obtain time sequence features; fusing the topology features and the time sequence features to obtain time sequence topology features, performing feature extraction on the time sequence topology features through a spatial transformer to obtain global relationship features, and performing feature extraction on the global relationship features through a time transformer to obtain global spatio-temporal features.

4. The method of claim 1, wherein, the convolution operations on the frame sequence through a normal convolution layer and a dilated convolution layer to obtain first local features and second local features comprise: performing convolution operations on the frame sequence through a normal convolution layer to obtain first motion features, and performing convolution operations again on the first motion features to obtain first local features; A second motion feature is obtained by performing a convolution operation on the frame sequence through a dilated convolution layer, and a second local feature is obtained by performing a max-pooling operation on the second motion feature.

5. The method of claim 1, wherein, The evaluation of the work action of the target work personnel includes: Similarity calculation is performed between the work action and a standard action to obtain a risk score; If the risk score is less than a first threshold, the target work personnel is determined to be a standard work; If the risk score is greater than or equal to the first threshold, the target work personnel is determined to be a high-risk work.

6. The method of claim 5, wherein, The method further includes: In a case where the target work personnel is determined to be a high-risk work, a target operation is performed, the target operation including at least one of the following: issuing an alarm, suspending a work device, recording high-risk work information, providing a safety prompt, and providing operation guidance.

7. An apparatus for identifying and evaluating a warehouse logistics operation action, characterized by comprising: a warehouse logistics operation action identifying unit 1; a warehouse logistics operation action evaluating unit 2; and a warehouse logistics operation action identifying and evaluating unit 3. It includes: A data preprocessing module is configured to obtain a video stream of a warehouse area, extract skeleton features from the video stream to obtain skeleton data, and frame the video stream to obtain a frame sequence; A global feature extraction module is configured to determine joint data according to the skeleton data, and extract global spatio-temporal features from the joint data; A local feature extraction module is configured to perform convolution operations on the frame sequence through an ordinary convolution layer and a dilated convolution layer to obtain first local features and second local features, and fuse the first local features and the second local features to obtain local fusion features; A work action evaluation module is configured to determine a work action of a target work personnel according to the global spatio-temporal features and the local fusion features, and evaluate the work action of the target work personnel; The local feature extraction module is specifically configured to: fuse the first local features and the second local features to obtain first fusion features; extract appearance features from the frame sequence through a ResNet-50 backbone network, fuse the first fusion features and the appearance features to obtain second fusion features; extract spatial features from the second fusion features, and model the second fusion features through a 3D convolution layer to obtain weight features; perform average pooling operations on the second fusion features in a spatial dimension to obtain spatial descriptors, model the spatial descriptors through a 2D convolution layer to obtain spatial information, and activate the spatial information through a Sigmoid function to obtain spatial weight features; perform average pooling operations on the second fusion features in a spatial dimension to obtain temporal descriptors, model the temporal descriptors through a 1D convolution layer to obtain temporal information, and activate the temporal information through a Sigmoid function to obtain temporal weight features; concatenate the spatial features, the weight features, the spatial weight features, and the temporal weight features to obtain the local fusion features.

8. A computer device, comprising: It includes a processor and a memory, the memory stores programs or instructions running on the processor, and the programs or instructions are executed by the processor to realize the steps of the warehouse logistics work action identification and evaluation method according to any one of claims 1 to 6.

9. A readable storage medium, on which a program or instructions are stored, characterized in that, The program or the instruction realizes the steps of the identification evaluation method of the warehouse logistics operation action according to any one of claims 1 to 6 when executed by the processor.

Images