Warehouse logistics operation action recognition and evaluation method and device, equipment and medium

Abstract

The invention provides a warehouse logistics operation action recognition and evaluation method, device and equipment and a medium, and relates to the field of image processing. The method comprises the following steps: acquiring a video stream of a storage area, performing skeleton feature extraction on the video stream to obtain skeleton data, and performing frame extraction on the video stream to obtain a frame sequence; determining joint data according to the skeleton data, and performing spatial-temporal feature extraction on the joint data to obtain global spatial-temporal features; respectively performing convolution operation on the frame sequence through a common convolution layer and an expansion convolution layer to obtain a first local feature and a second local feature, and performing feature fusion on the first local feature and the second local feature to obtain a local fusion feature; and according to the global spatial-temporal features and the local fusion features, determining operation actions of a target operator, and evaluating the operation actions of the target operator.

Description

Technical Field

[0001] This application relates to the field of image processing, and particularly to a method, device, computer device, and readable storage medium for identifying and evaluating the operation actions in warehousing and logistics operations. Background Art

[0002] With the booming development of e-commerce and the increasing diversification of consumer demands, the warehousing and logistics industry is facing huge challenges and opportunities. The progress of technologies, such as automation, artificial intelligence, Internet of Things, and big data analysis, provides new possibilities for the optimization of warehousing and logistics. These technologies not only improve the efficiency and accuracy of warehousing operations but also enhance the real-time monitoring and management capabilities of inventory.

[0003] In related technologies, the action recognition of operators in warehousing and logistics is an important technical means to improve efficiency, ensure safety, and optimize management. However, currently, the accuracy of action recognition for operators in the warehouse is relatively low. Summary of the Invention

[0004] In view of this, this application provides a method, device, computer device, and readable storage medium for identifying and evaluating the operation actions in warehousing and logistics, which solves the problem of relatively low action recognition for operators in the warehouse.

[0005] In a first aspect, an embodiment of this application provides a method for identifying and evaluating the operation actions in warehousing and logistics, including:

[0006] Obtain the video stream of the warehousing area, extract the skeleton features from the video stream to obtain skeleton data, and extract frames from the video stream to obtain a frame sequence;

[0007] Determine joint data according to the skeleton data, and extract spatio-temporal features from the joint data to obtain global spatio-temporal features;

[0008] Perform convolution operations on the frame sequence through a common convolution layer and a dilated convolution layer respectively to obtain a first local feature and a second local feature, and fuse the first local feature and the second local feature to obtain a local fusion feature;

[0009] Determine the operation actions of the target operator according to the global spatio-temporal features and the local fusion features, and evaluate the operation actions of the target operator.

[0010] In a second aspect, an embodiment of this application provides a device for identifying and evaluating the operation actions in warehousing and logistics, including:

[0011] A data preprocessing module, configured to obtain the video stream of the warehousing area, extract the skeleton features from the video stream to obtain skeleton data, and extract frames from the video stream to obtain a frame sequence;

[0012] A global feature extraction module, configured to determine joint data according to the skeleton data, and perform spatio-temporal feature extraction on the joint data to obtain global spatio-temporal features;

[0013] A local feature extraction module, configured to perform convolution operations on the frame sequence through a normal convolution layer and a dilated convolution layer respectively to obtain a first local feature and a second local feature, and fuse the first local feature and the second local feature to obtain a local fusion feature;

[0014] A job action evaluation module, configured to determine the job actions of a target operator according to the global spatio-temporal features and the local fusion features, and evaluate the job actions of the target operator.

[0015] In a third aspect, an embodiment of the present application provides a computer device, which includes a processor and a memory. The memory stores a program or instruction that can run on the processor. When the program or instruction is executed by the processor, the steps of the method in the first aspect are implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method in the first aspect are implemented.

[0017] The recognition and evaluation method, device, computer device, and readable storage medium for warehousing and logistics operation actions proposed in the present application obtain a video stream of a warehousing area, perform feature extraction on the video stream to obtain skeleton data, and extract frames from the video stream to obtain a frame sequence; determine joint data according to the skeleton data and perform spatio-temporal feature extraction to obtain global spatio-temporal features; perform convolution operations on the frame sequence respectively to obtain a first local feature and a second local feature and perform feature fusion to obtain a local fusion feature; determine operation actions according to the global spatio-temporal features and the local fusion features, and evaluate the operation actions of the target operator. In the present application, by obtaining the video stream of the warehousing area and performing skeleton feature extraction and frame extraction operations, the accuracy and processing efficiency of feature extraction are ensured, joint data is determined and global spatio-temporal features are extracted to capture the long-term dynamics of actions, local features are extracted and fused through a normal convolution layer and a dilated convolution layer to enhance the perception ability of local motion details, and operation actions are determined and evaluated by combining global features and local features, realizing accurate recognition and effective evaluation of operation behaviors.

[0018] The above description is only an overview of the technical solution of the present application. In order to be able to understand the technical means of the present application more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features, and advantages of the present application more obvious and understandable, the specific embodiments of the present application are specifically listed below. Description of the Drawings

[0019] The accompanying drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0020] Figure 1 A schematic flow diagram of a method for identifying and evaluating warehousing and logistics operation actions according to an embodiment of the present application is shown;

[0021] Figure 2 A structural block diagram of an apparatus for identifying and evaluating warehousing and logistics operation actions according to an embodiment of the present application is shown;

[0022] Figure 3 A structural block diagram of a computer device according to an embodiment of the present application is shown. Detailed implementation manners

[0023] Next, the technical solutions in the embodiments of the present application will be clearly described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application belong to the scope of protection of the present application.

[0024] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order different from those illustrated or described herein, and the objects distinguished by "first", "second", etc. generally belong to the same category, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / " generally indicates an "or" relationship between the associated objects before and after.

[0025] Next, in conjunction with the accompanying drawings, a method, apparatus, computer device, and readable storage medium for identifying and evaluating warehousing and logistics operation actions provided by the embodiments of the present application will be described in detail through specific embodiments and their application scenarios. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0026] The embodiments of the present application provide a method for identifying and evaluating warehousing and logistics operation actions, as Figure 1 shown, the method includes:

[0027] Step 101, obtaining a video stream of a warehousing area, extracting skeleton features from the video stream to obtain skeleton data, and extracting frames from the video stream to obtain a frame sequence;

[0028] Step 102: Determine joint data based on the skeleton data, and extract spatio-temporal features from the joint data to obtain global spatio-temporal features;

[0029] Step 103: Perform convolution operations on the frame sequence through a normal convolutional layer and a dilated convolutional layer respectively to obtain a first local feature and a second local feature, and fuse the first local feature and the second local feature to obtain a local fusion feature;

[0030] Step 104: Determine the operation actions of the target operator based on the global spatio-temporal features and the local fusion features, and evaluate the operation actions of the target operator.

[0031] In one implementation, a video stream is obtained through a camera installed in the warehouse to implement the recognition and evaluation of warehousing and logistics operation actions based on machine vision. Machine vision is a technology that uses optical devices and image processing techniques to simulate the visual functions of humans. It captures images through a camera or other optical sensors, and then uses computer algorithms to process and analyze these images to achieve specific goals, such as object recognition, detection, measurement, and positioning.

[0032] In one implementation, skeleton features are extracted from the video stream to obtain skeleton data. The skeleton data is a simplified and abstract representation of the body posture of the operator. It describes the human posture and actions through the positions and connection relationships of key points, rather than directly using the pixel information in the video. This abstract representation has the following advantages: (1) Reducing the amount of data. The skeleton data only contains the coordinate information of key points. The coordinate information includes points and edges. The points are the joints. Compared with the original video stream, the amount of data is greatly reduced, which is convenient for processing and analysis; (2) Improving robustness: The skeleton data has strong robustness to changes in lighting, background, and clothing, and can more stably describe human actions; (3) Facilitating modeling: The structured form of the skeleton data is convenient for mathematical modeling and feature extraction, and is suitable for action recognition and analysis. Standardize the skeleton data to ensure that the coordinates of each joint are on the same scale. The standardization process can eliminate the scale differences between different videos and improve the generalization ability of the model. Through standardization, the model can better process the skeleton data from different data sources and reduce the recognition errors caused by scale differences.

[0033] In one implementation, joint data is determined based on the skeleton data. The joint data includes joint topology and joint motion data, and further extracts global spatio-temporal features, so as to be able to understand the action pattern of the operator as a whole. The extraction of joint data not only includes the topological structure, but also covers the motion information, providing rich details for the comprehensive analysis of actions. The extraction of global spatio-temporal features helps to capture the long-term dynamics of actions and enhances the ability to understand complex actions.

[0034] In one implementation, frames are extracted from the video stream to obtain a frame sequence. The video stream is divided into n non-overlapping segments, each segment containing several frames. Multiple adjacent frames are randomly sampled from each segment to form the frame sequence I * ={I1, I2, …, I n}. Since video data has a time dimension, by dividing segments and sampling frames, the computational load can be effectively reduced while retaining key spatio-temporal information, reducing the demand for computing resources while retaining the key information in the video, and improving the efficiency and accuracy of the model.

[0035] In one implementation, convolution operations are respectively performed on the frame sequence through a normal convolutional layer and a dilated convolutional layer to extract the first local feature and the second local feature, and the first local feature and the second local feature are fused to obtain a local fusion feature that can reflect local motion details and broader spatio-temporal relationships. This multi-scale feature fusion strategy not only enriches the feature representation but also improves the model's perception ability of local motion, making action recognition more accurate.

[0036] Furthermore, based on the global spatio-temporal feature and the local fusion feature, the operation action of the target operator is determined, and the operation action of the target operator is evaluated.

[0037] The embodiment of the present application provides a method for identifying and evaluating operation actions in warehousing logistics based on machine vision. By acquiring the video stream of the warehousing area and performing skeleton feature extraction and frame extraction operations, the accuracy of feature extraction and processing efficiency are ensured. Joint data is determined and global spatio-temporal features are extracted to capture the long-term dynamics of actions. Local features are extracted and fused through a normal convolutional layer and a dilated convolutional layer to enhance the perception ability of local motion details. Combining global features and local features to determine operation actions and evaluate them, realizing accurate identification and effective evaluation of operation behaviors.

[0038] In an embodiment of the present application, spatio-temporal feature extraction is performed on the joint data to obtain global spatio-temporal features, including:

[0039] Constructing a spatio-temporal topology graph according to the joint data;

[0040] Performing feature extraction on the spatio-temporal topology graph through adaptive graph convolution to obtain topological features, and performing feature extraction on the spatio-temporal topology graph through temporal dilated convolution to obtain temporal features;

[0041] Fusing the topological features and the temporal features to obtain temporal-topological features, performing feature extraction on the temporal-topological features through a spatial transformer to obtain global relationship features, and performing feature extraction on the global relationship features through a temporal transformer to obtain global spatio-temporal features.

[0042] In this embodiment, the spatio-temporal data of the skeletal joints is transformed into a graph structure, that is, a spatio-temporal topological graph, to clarify the spatial connection and temporal continuity between joints. The spatial connection is, for example, between the elbow and the shoulder, and the temporal continuity is, for example, the same joint in consecutive frames. Exemplarily, in the "running" action, the spatio-temporal graph can represent the swinging trajectory of the leg joints (temporal edges) and the linkage relationship of the trunk joints (spatial edges), providing a structured input for subsequent feature extraction.

[0043] Topological features are obtained by performing feature extraction on the spatio-temporal topological graph through adaptive graph convolution (AGCN, Adaptive Graph Convolutional Network). Adaptive graph convolution can capture potential joint relationships outside natural anatomical connections, such as joints that are related to the action but not directly connected. Exemplarily, in the "waving" action, the adaptive graph convolution can identify the dynamic association between the hand and the shoulder, even if they are not directly connected, enhancing the representational ability of local topological features.

[0044] Temporal features are obtained by performing feature extraction on the spatio-temporal topological graph through temporal dilation convolution (TDCN, Temporal Dilation Convolutional Network). Temporal dilation convolution uses multi-scale dilation convolution to expand the temporal receptive field and capture long-distance temporal dependencies, such as the persistence and periodicity of actions. Exemplarily, in the "walking" action, the temporal dilation convolution can model the continuous swinging pattern of the leg from stepping to landing, avoiding the loss of long temporal information by short-time convolution.

[0045] The topological features and temporal features are fused to obtain spatio-temporal topological features. Global relationship features are obtained by performing feature extraction on the spatio-temporal topological features through a spatial transformer. The spatial transformer (i.e., the spatial Transformer) acts on the spatio-temporal topological features within a single frame and dynamically models the global dependencies between all joints through the self-attention mechanism, breaking through the limitations of traditional graph structures on natural anatomical connections. For example, in the "lifting heavy objects" action, even if the hand and the waist are not directly connected, the spatial Transformer can still capture the co-variations between them due to mechanical balance, identify the association strength between non-adjacent joints (such as calculating the attention weights between the hand and the waist), and thus extract global relationship features reflecting the overall body coordination.

[0046] The global spatio-temporal features are obtained by extracting the global relational features through a time transformer. The time transformer (i.e., the time Transformer), based on the global relational features output by the spatial Transformer, analyzes the cross-frame dynamic evolution of the same joint along the time dimension, captures the long-term temporal patterns of actions, such as the start, duration, and end phases of an action. For example, in the "continuous bending" scenario, the time Transformer identifies the postural deterioration caused by cumulative fatigue by comparing the angle change trends of the spinal joints in different frames, such as the bending amplitude increasing frame by frame, and finally generates the global spatio-temporal features that fuse spatio-temporal continuity, providing a complete temporal basis for action risk assessment.

[0047] In the embodiments of the present application, the spatio-temporal topological graph is constructed to explicitly model the spatio-temporal associations between joints, such as torso linkage and time trajectories, and the adaptive graph convolution is used to dynamically capture the action-related relationships outside the potential anatomical connections, such as the dynamic association between the hand and the shoulder when waving. At the same time, the time-domain dilated convolution is adopted to break through the short-term dependence limitation and accurately model the periodic long-range action patterns, such as the walking gait cycle, solving the problem in the related technologies that the short-term actions cannot be completely modeled, resulting in missing or misidentifying in the recognition process. Finally, through the hierarchical feature fusion of the spatio-temporal Transformer, the progressive abstraction from local topology to global spatio-temporal relationships is realized, significantly enhancing the model's modeling ability for complex motion patterns such as multi-joint coordination in running, and solving the limitations of traditional methods in spatial sparse connections and long-term temporal dependencies.

[0048] In an embodiment of the present application, the first local feature and the second local feature are obtained by performing convolution operations on the frame sequence through a normal convolution layer and a dilated convolution layer respectively, including:

[0049] The first motion feature is obtained by performing a convolution operation on the frame sequence through a normal convolution layer, and the first local feature is obtained by performing a convolution operation on the first motion feature again;

[0050] The second motion feature is obtained by performing a convolution operation on the frame sequence through a dilated convolution layer, and the second local feature is obtained by performing a max pooling operation on the second motion feature.

[0051] In this embodiment, a normal convolution layer, such as a 3D convolution layer of 3×3×3, is used to perform a convolution operation on the frame sequence to extract the first motion feature F1, and then a normal convolution layer is used again to perform a convolution operation on the first motion feature F1 to obtain the refined first local feature F1′, that is, the detail feature.

[0052] By performing two convolutional operations on the frame sequence using a 3×3×3 3D convolutional layer, the first motion feature F1 and the first local feature F1' are obtained respectively, which can effectively extract the detailed information of local motion. This operation method can capture the subtle motion changes between adjacent frames, thus providing rich local motion features for subsequent action recognition. This fine local motion feature extraction is crucial for understanding the dynamic changes of actions, especially when dealing with complex actions, and can significantly improve the model's perception ability of local motion.

[0053] Use a dilated convolutional layer, such as a 3×3×3 convolutional layer with a dilation rate of 2, to perform a convolutional operation on the frame sequence to obtain a motion feature with a larger receptive field, that is, the second motion feature F2. Perform a max pooling operation on the second motion feature F2 in the spatial dimension to obtain the downsampled second local feature F2', that is, the detail feature.

[0054] By performing a convolutional operation on the frame sequence using a 3×3×3 dilated convolutional layer with a dilation rate of 2, the second motion feature F2 is obtained, and further the second local feature F2' is obtained through a max pooling operation. The use of the dilated convolutional layer effectively expands the receptive field of the convolutional kernel, enabling the model to capture motion information in a larger range. This operation not only retains the details of local motion but also can capture broader spatio-temporal relationships, thus providing a more comprehensive motion feature representation for action recognition. This feature extraction method of expanding the receptive field helps the model better understand the overall dynamics of actions, especially when dealing with long-term actions, and can significantly improve the recognition accuracy.

[0055] In an embodiment of the present application, the first local feature and the second local feature are fused to obtain a local fusion feature, including:

[0056] Fuse the first local feature and the second local feature to obtain a first fusion feature;

[0057] Extract the appearance feature by using the ResNet-50 backbone network to perform feature extraction on the frame sequence, and fuse the first fusion feature and the appearance feature to obtain a second fusion feature;

[0058] Extract the spatial feature from the second fusion feature, and model the second fusion feature through a 3D convolutional layer to obtain a weight feature;

[0059] Perform an average pooling operation on the second fusion feature in the time dimension to obtain a spatial descriptor, model the spatial descriptor through a 2D convolutional layer to obtain spatial information, and activate the spatial information through a Sigmoid function to obtain a spatial weight feature;

[0060] Perform average pooling operation on the second fusion feature in the spatial dimension to obtain a temporal descriptor, model the temporal descriptor through a 1D convolutional layer to obtain temporal information, and activate the temporal information through a Sigmoid function to obtain a temporal weight feature;

[0061] Concatenate the spatial feature, weight feature, spatial weight feature, and temporal weight feature to obtain a local fusion feature.

[0062] In this embodiment, the first local feature F1′ and the second local feature F2′ are fused to obtain a first fusion feature. This fusion method combines motion features of different scales, enabling the model to capture both local details and global dynamics simultaneously. In addition, the ResNet-50 backbone network is used to extract features from the frame sequence to obtain appearance features, and the extracted appearance features are further fused with the first fusion feature to obtain a second fusion feature. This multi-scale feature fusion strategy not only enriches the feature representation but also improves the model's perception ability of motion information at different scales, effectively enhancing the robustness and accuracy of the model in processing complex actions. Especially when facing actions with different speeds and scales, it can significantly improve the recognition performance.

[0063] In one implementation, ResNet-50 is a deep residual convolutional neural network. By introducing residual blocks and skip connections, it solves the problem of gradient disappearance in the training of deep networks and can efficiently extract high-level semantic features of images. In this process, ResNet-50 extracts features from the video frame sequence and captures the global appearance information (such as color, texture, shape) and context semantic information (such as human pose, tool form) of the objects in the image through its 50-layer network structure (including 49 convolutional layers and 1 fully connected layer).

[0064] Extract spatial features from the second fusion feature, and model the second fusion feature through a 3D convolutional layer to obtain a weight feature. In one implementation, the "D" in the 3D convolutional layer refers to depth, that is, the size of the convolutional kernel (or filter) in the temporal dimension or depth dimension. By extracting spatial features from the second fusion feature and modeling these features through a 3D convolutional layer, a weight feature is obtained. This operation can effectively capture the spatial information of each frame in the video, and at the same time, through the modeling of the 3D convolutional layer, the expressive ability of these spatial features is further strengthened. This enhanced spatial feature extraction method helps the model better understand the static information of each frame in the video, thus providing richer spatial context for subsequent action recognition.

[0065] In machine learning and deep learning, modeling refers to learning and expressing the features of data through mathematical methods and algorithms in order to extract useful information from it. For example: By using a convolutional layer to model the features, spatial features or temporal features in the data can be extracted. Spatial descriptor: A spatial descriptor is a concise representation of the spatial information of each frame in a video. By performing average pooling operation on the second fused feature in the temporal dimension, the spatial descriptor of each frame can be obtained, which contains the information of all spatial positions in that frame. Temporal descriptor: A temporal descriptor is a concise representation of the temporal information of each frame in a video. By performing average pooling operation on the second fused feature in the spatial dimension, the temporal descriptor of each frame can be obtained, which contains the dynamic change information of that frame in the time series.

[0066] In one implementation, the spatial descriptor is obtained by performing average pooling operation on the second fused feature in the temporal dimension, then the spatial descriptor is modeled through a 2D convolutional layer, and the spatial weight feature is obtained by activation through a Sigmoid function. In this way, the importance of different spatial regions in each frame can be dynamically adjusted, enabling the model to pay more attention to those regions that are more meaningful for action recognition. This dynamic adjustment mechanism of spatial weights improves the model's perception ability of key spatial regions, thereby enhancing the accuracy of action recognition.

[0067] In one implementation, the temporal descriptor is obtained by performing average pooling operation on the second fused feature in the spatial dimension, then the temporal descriptor is modeled through a 1D convolutional layer, and the temporal weight feature is obtained by activation through a Sigmoid function. In this way, the importance of different time points can be dynamically adjusted, enabling the model to pay more attention to those time periods that are more meaningful for action recognition. This dynamic adjustment mechanism of temporal weights improves the model's perception ability of key time points, thereby enhancing the accuracy of action recognition.

[0068] Furthermore, the spatial feature, weight feature, spatial weight feature, and temporal weight feature are concatenated to obtain a richer local fused feature. This multi-dimensional feature fusion method combines the information of both spatial and temporal dimensions, enabling the model to more comprehensively understand the action features in the video. This fusion method not only enriches the feature representation but also improves the model's perception ability of information in different dimensions, thereby significantly improving the performance of action recognition.

[0069] In an embodiment of the present application, evaluating the operation actions of a target operator includes:

[0070] Calculating the similarity between the operation action and the standard action to obtain a risk score;

[0071] If the risk score < the first threshold, it is determined that the target operator is operating in a standard manner;

[0072] If the risk score is ≥ the first threshold, it is determined that the target operator is engaged in high-risk operations.

[0073] In this embodiment, by calculating the similarity between the operation actions and the standard actions to obtain the risk score, it is possible to evaluate in real time whether the actions of the operator conform to the specifications, ensure that potential high-risk behaviors are immediately discovered during the operation process, so as to take measures in a timely manner and reduce the risk of accidents. Real-time risk assessment improves the safety of the operation site and ensures that the operators can work in a safe environment.

[0074] In one implementation, the similarity calculation is performed by quantifying the similarity degree between two actions. For example: (1) First, key features such as the angle, speed, acceleration, etc. of the operation action and the standard action are extracted, and the similarity is evaluated by calculating the differences between these features. Common methods include Euclidean distance, cosine similarity, etc.; (2) A deep learning model, such as a convolutional neural network or a recurrent neural network, is used to learn the feature representation of the action, and the similarity is calculated through the feature vectors output by the model. This method can automatically learn the complex features of the action and improve the accuracy of the similarity calculation.

[0075] In an embodiment of the present application, the method further includes: in the case where it is determined that the target operator is engaged in high-risk operations, performing a target operation, and the target operation includes at least one of the following: issuing an alarm, pausing the operation equipment, recording high-risk operation information, providing safety tips and operation guidance.

[0076] In this embodiment, after determining the high-risk operation behavior, in addition to issuing an alarm or pausing the operation equipment, the following operations can also be taken: (1) Automatically record the detailed information of the high-risk operation, including time, location, operator identity, specific actions, etc., and generate a safety report, which can be used for subsequent safety analysis and accident investigation; (2) Provide immediate safety tips and operation guidance to the operator, such as voice guidance, and report the risk to the management, helping the operator correct the non-standard actions and avoid further risks.

[0077] As a specific implementation of the above-mentioned method for identifying and evaluating warehousing and logistics operation actions, an embodiment of the present application provides a device for identifying and evaluating warehousing and logistics operation actions. As Figure 2 shown, the device 200 for identifying and evaluating warehousing and logistics operation actions includes: a data preprocessing module 201, a global feature extraction module 202, a local feature extraction module 203, and an operation action evaluation module 204.

[0078] Among them, the data preprocessing module 201 is used to obtain the video stream of the warehousing area, extract the skeleton features from the video stream to obtain skeleton data, and extract frames from the video stream to obtain a frame sequence;

[0079] The global feature extraction module 202 is used to determine joint data based on the skeleton data, and perform spatio-temporal feature extraction on the joint data to obtain global spatio-temporal features;

[0080] The local feature extraction module 203 is used to perform convolution operations on the frame sequence through a common convolutional layer and a dilated convolutional layer respectively to obtain a first local feature and a second local feature, and fuse the first local feature and the second local feature to obtain a local fusion feature;

[0081] The operation action evaluation module 204 is used to determine the operation action of the target operator according to the global spatio-temporal features and the local fusion features, and evaluate the operation action of the target operator.

[0082] Furthermore, the joint data includes joint topological structure and joint motion data.

[0083] Furthermore, the global feature extraction module 202 is specifically used for:

[0084] Construct a spatio-temporal graph according to the joint data to obtain a spatio-temporal topological graph;

[0085] Extract topological features from the spatio-temporal topological graph through adaptive graph convolution, and extract temporal features from the spatio-temporal topological graph through temporal dilated convolution;

[0086] Fuse the topological features and the temporal features to obtain temporal topological features, extract global relationship features from the temporal topological features through a spatial transformer, and extract global spatio-temporal features from the global relationship features through a temporal transformer.

[0087] Furthermore, the local feature extraction module 203 is specifically used for:

[0088] Perform a convolution operation on the frame sequence through a common convolutional layer to obtain a first motion feature, and perform a convolution operation on the first motion feature again to obtain a first local feature;

[0089] Perform a convolution operation on the frame sequence through a dilated convolutional layer to obtain a second motion feature, and perform a max pooling operation on the second motion feature to obtain a second local feature.

[0090] Furthermore, the local feature extraction module 203 is specifically used for:

[0091] Fuse the first local feature and the second local feature to obtain a first fusion feature;

[0092] Extract appearance features from the frame sequence through a ResNet-50 backbone network, and fuse the first fusion feature and the appearance features to obtain a second fusion feature;

[0093] Feature extraction is performed on the second fusion feature to obtain spatial features, and the second fusion feature is modeled through a 3D convolutional layer to obtain weight features;

[0094] Average pooling operation in the time dimension is performed on the second fusion feature to obtain a spatial descriptor, the spatial descriptor is modeled through a 2D convolutional layer to obtain spatial information, and the spatial information is activated through a Sigmoid function to obtain spatial weight features;

[0095] Average pooling operation in the spatial dimension is performed on the second fusion feature to obtain a time descriptor, the time descriptor is modeled through a 1D convolutional layer to obtain time information, and the time information is activated through a Sigmoid function to obtain time weight features;

[0096] The spatial features, weight features, spatial weight features, and time weight features are concatenated to obtain local fusion features.

[0097] Furthermore, the job action evaluation module 204 is specifically configured to:

[0098] Calculate the similarity between the job action and the standard action to obtain a risk score;

[0099] If the risk score < the first threshold, it is determined that the target operator is performing the job in a standardized manner;

[0100] If the risk score ≥ the first threshold, it is determined that the target operator is performing a high-risk job.

[0101] Furthermore, the device further includes:

[0102] An execution module, configured to: in the case of determining that the target operator is performing a high-risk job, execute a target operation, where the target operation includes at least one of the following: issuing an alarm, pausing the job equipment, recording high-risk job information, providing safety tips and operation guidance.

[0103] The warehousing and logistics job action recognition and evaluation device 200 in the embodiments of the present application may be a computer device or a component in a computer device, such as an integrated circuit or a chip. The warehousing and logistics job action recognition and evaluation device 200 provided in the embodiments of the present application can implement Figure 1 each process implemented by the warehousing and logistics job action recognition and evaluation method embodiment, and for the sake of avoiding repetition, it will not be elaborated here.

[0104] The embodiments of the present application further provide a computer device, such as Figure 3As shown, the computer device 300 includes a processor 301 and a memory 302. A program or instruction that can run on the processor 301 is stored on the memory 302. When the program or instruction is executed by the processor 301, it implements each step of the recognition and evaluation method embodiment of the above warehousing and logistics operation actions, and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0105] The memory 302 can be used to store software programs and various data. The memory 302 mainly includes a first storage area for storing programs or instructions and a second storage area for storing data. Among them, the first storage area can store an operating system, application programs or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 302 can include volatile memory or non-volatile memory, or the memory 302 can include both volatile and non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synch link dynamic random access memory (SLDRAM), and a direct rambus random access memory (DRRAM). The memory 302 in the embodiments of the present application includes but is not limited to these and any other suitable types of memory.

[0106] The processor 301 can include one or more processing units; optionally, the processor 301 integrates an application processor and a modem processor. Among them, the application processor mainly processes operations related to the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above modem processor may not be integrated into the processor 301.

[0107] The embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, it implements each process of the above-described embodiment of the method for identifying and evaluating warehousing and logistics operation actions, and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0108] It should be noted that in this document, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, article or device including that element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0109] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the purpose of the present application and the scope protected by the claims, and all of them belong to the protection scope of the present application.

Claims

1. A method for identifying and evaluating the operation actions of warehousing logistics, characterized in that Including: Obtain the video stream of the storage area, extract the skeleton features from the video stream to obtain skeleton data, and extract frames from the video stream to obtain a frame sequence; Determine joint data according to the skeleton data, and extract spatio-temporal features from the joint data to obtain global spatio-temporal features; Perform convolution operations on the frame sequence through a normal convolution layer and a dilated convolution layer respectively to obtain a first local feature and a second local feature, and fuse the first local feature and the second local feature to obtain a local fusion feature; Determine the operation actions of the target operator according to the global spatio-temporal features and the local fusion features, and evaluate the operation actions of the target operator.

2. The method according to claim 1, wherein The joint data includes joint topological structure and joint motion data.

3. The method according to claim 1, characterized in that, The extracting spatio-temporal features from the joint data to obtain global spatio-temporal features includes: Construct a spatio-temporal graph according to the joint data to obtain a spatio-temporal topological graph; Extract topological features from the spatio-temporal topological graph through adaptive graph convolution, and extract temporal features from the spatio-temporal topological graph through temporal dilated convolution; Fuse the topological features and the temporal features to obtain a temporal topological feature, extract global relationship features from the temporal topological feature through a spatial transformer, and extract global spatio-temporal features from the global relationship feature through a temporal transformer.

4. The method according to claim 1, wherein The performing convolution operations on the frame sequence through a normal convolution layer and a dilated convolution layer respectively to obtain a first local feature and a second local feature includes: Perform a convolution operation on the frame sequence through a normal convolution layer to obtain a first motion feature, and perform a convolution operation on the first motion feature again to obtain a first local feature; Perform a convolution operation on the frame sequence through a dilated convolution layer to obtain a second motion feature, and perform a max pooling operation on the second motion feature to obtain a second local feature.

5. The method according to claim 1, characterized in that, The fusing the first local feature and the second local feature to obtain a local fusion feature includes: Fuse the first local feature and the second local feature to obtain a first fusion feature; Extract appearance features from the frame sequence through a ResNet-50 backbone network, and fuse the first fusion feature and the appearance features to obtain a second fusion feature; Extract spatial features from the second fusion feature, and model the second fusion feature through a 3D convolution layer to obtain weight features; Perform an average pooling operation on the second fusion feature in the time dimension to obtain a spatial descriptor, model the spatial descriptor through a 2D convolution layer to obtain spatial information, and activate the spatial information through a Sigmoid function to obtain spatial weight features; Perform an average pooling operation on the second fusion feature in the spatial dimension to obtain a temporal descriptor, model the temporal descriptor through a 1D convolution layer to obtain temporal information, and activate the temporal information through a Sigmoid function to obtain temporal weight features; The spatial feature, the weight feature, the spatial weight feature, and the temporal weight feature are concatenated to obtain the local fusion feature.

6. The method according to claim 1, characterized in that, The evaluation of the operation actions of the target operator includes: Calculating the similarity between the operation action and the standard action to obtain a risk score; If the risk score < the first threshold, it is determined that the target operator is operating in a standard manner; If the risk score ≥ the first threshold, it is determined that the target operator is performing a high-risk operation.

7. The method according to claim 6, wherein The method further includes: In the case of determining that the target operator is performing a high-risk operation, performing a target operation, where the target operation includes at least one of the following: issuing an alarm, pausing the operation equipment, recording high-risk operation information, providing safety tips, and operation guidance.

8. An identification and evaluation device for warehousing and logistics operation actions, characterized in that Including: A data preprocessing module, configured to obtain a video stream of a storage area, extract skeleton data from the video stream, and extract frames from the video stream to obtain a frame sequence; A global feature extraction module, configured to determine joint data based on the skeleton data, and extract spatio-temporal features from the joint data to obtain global spatio-temporal features; A local feature extraction module, configured to perform convolution operations on the frame sequence through a normal convolution layer and a dilated convolution layer respectively to obtain a first local feature and a second local feature, and fuse the first local feature and the second local feature to obtain a local fusion feature; An operation action evaluation module, configured to determine the operation actions of the target operator according to the global spatio-temporal features and the local fusion features, and evaluate the operation actions of the target operator.

9. A computer device, characterized in that, Including a processor and a memory, where the memory stores a program or instruction that runs on the processor, and when the program or instruction is executed by the processor, the steps of the method for identifying and evaluating the operation actions of warehousing and logistics as described in any one of claims 1 to 7 are implemented.

10. A readable storage medium, on which a program or instructions are stored, characterized in that, When the program or instruction is executed by the processor, the steps of the method for identifying and evaluating the operation actions of warehousing and logistics as described in any one of claims 1 to 7 are implemented.

Images