Sorting behavior recognition method and device, equipment and storage medium
The sorting behavior recognition model constructed through the RetinaNet network solves the problem of insufficient monitoring of operators' sorting behavior in the existing technology, realizes rapid and accurate identification and alerts of sorting behavior, reduces the need for manual monitoring, and prevents work injuries .
Patent Information
- Application Number
- CN202411168763.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-12
AI Technical Summary
The existing automated sorting system insufficient monitoring of operator sorting behavior, resulting in non-operating behavior and improper sorting behaviors that may delay operations, damage goods or cause safety hazards.
The RetinaNet network is used to build a sorting behavior recognition model. By annotating and classifying the sorting images, the training set and test set are divided, the initial recognition model is trained, and the parameters are adjusted after the test, and the real-time sorting behavior is finally recognized and recognized. Alarm judgment.
It realizes rapid, accurate and automatic identification of operator sorting behaviors, reduces the need for manual monitoring, timely monitors unsafe behaviors, and prevents work-related injuries.
Smart Images

Figure CN118942160A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of behavior recognition technology, and in particular to a sorting behavior recognition method, device, equipment and storage medium. Background Art
[0002] In actual logistics operations, in order to improve sorting efficiency and accuracy, modern logistics distribution centers have begun to introduce automated equipment and intelligent systems. However, most of the existing automated sorting systems focus on the automatic identification and classification of goods, and do not adequately monitor the sorting behavior of operators, while the behavior of operators has a direct impact on sorting efficiency and safety. For example, non-operational behaviors such as rest and communication may cause delays in sorting operations, while improper sorting operations may damage goods or cause safety hazards.
[0003] Therefore, the prior art still needs to be improved and developed. Summary of the invention
[0004] The main purpose of the present invention is to solve the problem that the prior art does not adequately monitor the sorting behavior of operators, and improper sorting operations may damage goods or cause safety hazards.
[0005] A first aspect of the present invention provides a sorting behavior recognition method, comprising:
[0006] The collected sorting images are labeled and classified to obtain a pre-processed sorting image data set; the pre-processed sorting image data set is divided into a training set and a test set according to a predetermined ratio, and the training set and the test set both include conventional sorting operation image samples, non-operation behavior image samples, and violation behavior image samples; an initial recognition model is constructed based on the RetinaNet network; the initial recognition model is trained by the training set, the parameters of the initial recognition model are adjusted, and after being tested by the test set, a final sorting behavior recognition model is obtained; the real-time acquired sorting image to be recognized is input into the final sorting behavior recognition model, a detection result is output, and whether to issue an alarm is determined according to the detection result.
[0007] Optionally, in a first implementation of the first aspect of the present invention, the collected sorting images are labeled and classified to obtain a preprocessed sorting image data set, including the steps of: acquiring sorting images in different time periods and different scenes through a camera set in the sorting area; performing data amplification processing on the sorting images to obtain expanded image samples; and labeling and classifying the expanded image samples to obtain a preprocessed sorting image data set.
[0008] Optionally, in a second implementation of the first aspect of the present invention, data augmentation processing is performed on the sorted image to obtain an expanded picture sample, including the steps of: mirroring, rotating, scaling, cropping, translating or Gaussian noise processing on the sorted image to obtain a deformed picture sample; performing Mosaic processing and / or shearing and fusion processing on the sorted image and the deformed picture sample to obtain an expanded picture sample.
[0009] Optionally, in a third implementation of the first aspect of the present invention, the expanded image samples are labeled and classified to obtain a preprocessed sorting image data set, including the steps of: using the Labelme tool to perform different feature labeling on non-operational behaviors and illegal behaviors in the expanded image samples to obtain image samples labeled with different feature information; classifying the image samples according to different feature information to obtain conventional sorting operation image samples, non-operational behavior image samples and illegal behavior image samples, and the conventional sorting operation image samples, non-operational behavior image samples and illegal behavior image samples constitute the preprocessed sorting image data set.
[0010] Optionally, in a fourth implementation manner of the first aspect of the present invention, an initial recognition model is constructed based on a RetinaNet network, including the steps of: building a RetinaNet network framework, the RetinaNet network framework including a ResNet backbone network, a feature pyramid network, a classification and bounding box regression network, and a loss function module; the ResNet backbone network is stacked by multiple Residual modules, each Residual module includes multiple convolutional layers, batch normalization layers and ReLU activation functions, the Residual module uses shortcut connections to implement residual mapping, and inside the Residual module, the input X first undergoes a series of convolution and activation operations to obtain an output F(X), and at the same time, the input X is directly passed to the output end of the Residual module through a shortcut connection, and finally, the output of the Residual module is the element-by-element addition of F(X) and X, that is, H(X)=F(X)+X.
[0011] Optionally, in a fifth implementation manner of the first aspect of the present invention, the initial recognition model is trained using the training set, the parameters of the initial recognition model are adjusted, and after being tested using the test set, a final sorting behavior recognition model is obtained, including the steps of: assigning initial values to all parameters in the initial recognition model using a random initialization method; randomly selecting image samples from the training set and inputting them into the initial recognition model for training, calculating prediction results through forward propagation, and then calculating loss values according to a Focal Loss function; updating the parameters of the network using a back propagation algorithm so that the loss values gradually decrease; repeating the above training steps until a predetermined number of training rounds is reached to obtain a trained initial recognition model; evaluating the trained initial recognition model using a test set, adjusting hyperparameters according to the evaluation results, and obtaining a final sorting behavior recognition model.
[0012] Optionally, in a sixth implementation of the first aspect of the present invention, the loss value is calculated according to the Focal Loss function, including the steps of: using focal loss as the classification loss function, and measuring the difference between the category probability predicted by the initial recognition model and the true category through the classification loss function, and the calculation formula of Focal loss is as follows: L_cls = -α(1-p_t)^γlog(p_t), wherein p_t is the category probability predicted by the initial recognition model, α and γ are hyperparameters used to adjust the weights of samples of different difficulty levels, when p_t is close to 1, (1-p_t)^γ will become smaller, so that the loss value of the sample becomes smaller; conversely, when p_t is close to 0, (1-p_t)^γ will become larger, so that the loss value of the sample becomes larger; using smooth L1 loss as the regression loss function, and measuring the difference between the bounding box position predicted by the initial recognition model and the true bounding box position through the regression loss function, Smooth The calculation formula of L1 loss is as follows: L_reg = 0.5*(IoU-1)^2*1; if |IoU-1|<1, then L_reg = |IoU-1|-0.5, where IoU represents the IOU value of the predicted bounding box and the true bounding box. When the IoU value is close to 1, it means that the predicted box is very close to the true box, and the loss value is small at this time; when the IoU value deviates from 1, the loss value gradually increases.
[0013] The second aspect of the present invention provides a sorting behavior recognition device, comprising: a labeling and classification module, used to label and classify the collected sorting images to obtain a pre-processed sorting image data set; a data division module, used to divide the pre-processed sorting image data set into a training set and a test set according to a predetermined ratio, and the training set and the test set both include conventional sorting operation image samples, non-operation behavior image samples and violation behavior image samples; a model construction module, used to build an initial recognition model based on a RetinaNet network; a training module, used to train the initial recognition model through the training set, adjust the parameters of the initial recognition model, and obtain a final sorting behavior recognition model after testing the test set; a recognition module, used to input the real-time acquired sorting image to be recognized into the final sorting behavior recognition model, output a detection result, and determine whether to issue an alarm according to the detection result.
[0014] Optionally, in a first implementation of the second aspect of the present invention, the labeling and classification module includes: an acquisition unit, used to acquire sorting images in different time periods and different scenes through a camera set in the sorting area; an amplification unit, used to perform data amplification processing on the sorting images to obtain expanded image samples; and a classification unit, used to label and classify the expanded image samples to obtain a preprocessed sorting image data set.
[0015] Optionally, in a second implementation of the second aspect of the present invention, the amplification unit includes: a deformation subunit, used to mirror, rotate, scale, crop, translate or Gaussian noise process the sorted image to obtain a deformed picture sample; a transformation subunit, used to perform Mosaic processing and / or shearing and fusion processing on the sorted image and the deformed picture sample to obtain an expanded picture sample.
[0016] Optionally, in a third implementation of the second aspect of the present invention, the classification unit includes: a labeling subunit, used to perform different feature labeling on non-operational behaviors and illegal behaviors in the expanded image samples through the Labelme tool to obtain image samples labeled with different feature information; a classification subunit, used to classify the image samples according to different feature information to obtain conventional sorting operation image samples, non-operational behavior image samples and illegal behavior image samples, and the conventional sorting operation image samples, non-operational behavior image samples and illegal behavior image samples constitute the preprocessed sorting image data set.
[0017] Optionally, in a fourth implementation manner of the second aspect of the present invention, the model construction module includes: a building unit, building a RetinaNet network framework, the RetinaNet network framework includes a ResNet backbone network, a feature pyramid network, a classification and border regression network, and a loss function module; an introduction unit, used for the ResNet backbone network to be stacked with multiple Residual modules, each Residual module includes multiple convolutional layers, batch normalization layers and ReLU activation functions, the Residual module realizes residual mapping by introducing shortcut connections, inside the Residual module, the input X first undergoes a series of convolution and activation operations to obtain an output F(X), and at the same time, the input X is directly passed to the output end of the Residual module through a shortcut connection, and finally, the output of the Residual module is the element-by-element addition of F(X) and X, that is, H(X)=F(X)+X.
[0018] Optionally, in a fifth implementation of the second aspect of the present invention, the training module includes: an initialization unit, used to assign initial values to all parameters in the initial recognition model using a random initialization method; a calculation unit, used to randomly select image samples from the training set and input them into the initial recognition model for training, calculate the prediction results through forward propagation, and then calculate the loss value according to the Focal Loss function; an updating unit, used to update the parameters of the network using a back propagation algorithm so that the loss value gradually decreases; a repetition unit, used to repeat the above training steps until a predetermined number of training rounds is reached to obtain a trained initial recognition model; an evaluation unit, used to evaluate the trained initial recognition model using a test set, adjust the hyperparameters according to the evaluation results, and obtain the final sorting behavior recognition model.
[0019] Optionally, in a sixth implementation manner of the second aspect of the present invention, the calculation unit includes: a classification loss calculation subunit, used to adopt focal loss as a classification loss function, and to measure the difference between the category probability predicted by the initial recognition model and the true category through the classification loss function, and the calculation formula of focal loss is as follows: L_cls = -α(1-p_t)^γlog(p_t), wherein p_t is the category probability predicted by the initial recognition model, α and γ are hyperparameters used to adjust the weights of samples of different difficulty levels, and when p_t is close to 1, (1-p_t)^γ will become smaller, so that the loss value of the sample becomes smaller; conversely, when p_t is close to 0, (1-p_t)^γ will become larger, so that the loss value of the sample becomes larger; a regression loss calculation subunit, used to adopt smooth L1 loss as a regression loss function, and to measure the difference between the bounding box position predicted by the initial recognition model and the true bounding box position through the regression loss function, Smooth The calculation formula of L1 loss is as follows: L_reg = 0.5*(IoU-1)^2*1; if |IoU-1|<1, then L_reg = |IoU-1|-0.5, where IoU represents the IOU value of the predicted bounding box and the true bounding box. When the IoU value is close to 1, it means that the predicted box is very close to the true box, and the loss value is small at this time; when the IoU value deviates from 1, the loss value gradually increases.
[0020] A third aspect of the present invention provides a sorting behavior recognition device, comprising: a memory and at least one processor, wherein the memory stores computer-readable instructions, and the memory and the at least one processor are interconnected through a line; the at least one processor calls the computer-readable instructions in the memory so that the sorting behavior recognition device performs each step of the sorting behavior recognition method as described above.
[0021] A fourth aspect of the present invention provides a computer-readable storage medium, in which computer-readable instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the various steps of the sorting behavior identification method described above.
[0022] Beneficial effects: In the technical solution of the present invention, the collected sorting images are annotated and classified to obtain a pre-processed sorting image data set; the pre-processed sorting image data set is divided into a training set and a test set according to a predetermined ratio, and the training set and the test set both include conventional sorting operation image samples, non-operation behavior image samples and violation behavior image samples; an initial recognition model is constructed based on the RetinaNet network; the initial recognition model is trained by the training set, the parameters of the initial recognition model are adjusted, and after the test set is tested, a final sorting behavior recognition model is obtained; the real-time acquired sorting image to be recognized is input into the final sorting behavior recognition model, the detection result is output, and it is determined whether to issue an alarm according to the detection result. The present invention constructs a final sorting behavior recognition model based on the RetinaNet network, and the final sorting behavior recognition model can quickly and accurately automatically identify the sorting behavior of the operator, which can reduce the need for manual monitoring and allow managers to focus on tasks that require more manual judgment; at the same time, unsafe behaviors that may cause accidents can be monitored, such as improper weightlifting postures or working in prohibited areas, and timely warnings can be issued to prevent work-related injuries. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 A first flow chart of the sorting behavior identification method provided by an embodiment of the present invention;
[0024] Figure 2 A second flow chart of the sorting behavior identification method provided by an embodiment of the present invention;
[0025] Figure 3 A third flow chart of the sorting behavior identification method provided by an embodiment of the present invention;
[0026] Figure 4 A fourth flow chart of the sorting behavior identification method provided by an embodiment of the present invention;
[0027] Figure 5 A fifth flow chart of the sorting behavior identification method provided by an embodiment of the present invention;
[0028] Figure 6 A sixth flow chart of the sorting behavior identification method provided by an embodiment of the present invention;
[0029] Figure 7 A schematic diagram of the structure of a sorting behavior recognition device provided by an embodiment of the present invention;
[0030] Figure 8 Another structural schematic diagram of the sorting behavior recognition device provided by an embodiment of the present invention;
[0031] Fig. 9A schematic diagram of the structure of a sorting behavior recognition device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0032] The embodiment of the present invention provides a sorting behavior recognition method, device, equipment and storage medium, which perform labeling and classification processing on the collected sorting images to obtain a pre-processed sorting image data set; divide the pre-processed sorting image data set into a training set and a test set according to a predetermined ratio, and the training set and the test set both include conventional sorting operation image samples, non-operation behavior image samples and violation behavior image samples; construct an initial recognition model based on a RetinaNet network; train the initial recognition model through the training set, adjust the parameters of the initial recognition model, and obtain a final sorting behavior recognition model after testing the test set; input the sorting image to be recognized obtained in real time into the final sorting behavior recognition model, output the detection result, and judge whether to issue an alarm according to the detection result. The present invention constructs a final sorting behavior recognition model based on the RetinaNet network. Based on the final sorting behavior recognition model, the sorting behavior of the operator can be quickly and accurately automatically recognized, which can reduce the need for manual monitoring and allow managers to focus on tasks that require more manual judgment. At the same time, unsafe behaviors that may lead to accidents can also be monitored, such as improper weightlifting postures or working in prohibited areas, and timely warnings can be issued to prevent work-related injuries.
[0033] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0034] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , the first embodiment of the sorting behavior recognition method in the embodiment of the present invention includes:
[0035] S100: labeling and classifying the collected sorting images to obtain a preprocessed sorting image data set.
[0036] Specifically, in this embodiment, a camera is installed in the parcel sorting area to obtain sorting images in different time periods and different scenes as samples; of course, the sorting video can also be obtained through the camera, and then the sorting video is decomposed frame by frame, and frame-by-frame images in different time periods and different scenes are selected as samples. After the sorting images are obtained, the collected sorting images can be annotated and classified manually or by machine to obtain a pre-processed sorting image data set, including sorting images, annotation information, classification labels, etc., so as to ensure the quality and diversity of the data set so as to train a more accurate model later.
[0037] S200: Divide the preprocessed sorting image data set into a training set and a test set according to a predetermined ratio, wherein the training set and the test set both include conventional sorting operation image samples, non-operation behavior image samples, and violation behavior image samples.
[0038] In this embodiment, the initial recognition model initially constructed needs to be trained with a large amount of data and verified before the final sorting behavior recognition model with accurate recognition can be obtained. Therefore, in this embodiment, the pre-processed sorting image data set needs to be divided into a training set and a test set according to a predetermined ratio. In order to ensure that there are enough sample data sets to train the initial recognition model, the number of the training set should be greater than the number of the test set. Preferably, the ratio of the number of the training set to the test set is 8:2-6:4, wherein the training set and the test set both include conventional sorting operation image samples, non-operation behavior image samples and violation behavior image samples, wherein the image area where the sorting operation is being carried out is marked as the "conventional sorting operation" category; the area where the operator is in a non-working state such as resting or talking is marked as the "non-operation behavior" category; and the behavior that violates the operating regulations, such as violent sorting, not operating according to the process, etc., is marked as the "violation behavior" category, corresponding to the conventional sorting operation image samples, non-operation behavior image samples and violation behavior image samples mentioned above.
[0039] S300, build an initial recognition model based on the RetinaNet network.
[0040] Specifically, RetinaNet is a pioneering target detection framework. With its concise and clear structure, strong scalability, and high detection accuracy, the RetinaNet network is mainly composed of the following three parts: Backbone network: ResNet (residual network) is usually used. The number of ResNet network layers can range from 18 to 152. The main difference lies in the different residual units / modules used or the number and proportion of stacked residual units / modules. FPN (feature pyramid network): FPN expands the size of high-level feature maps by upsampling and fuses them with low-level feature maps to form new feature maps. These feature maps contain information of different scales, which helps to improve the model's detection ability for multi-scale targets. The FPN in RetinaNet is slightly different from the original FPN. The main difference lies in the selection and generation method of the feature layer. Head part: The Head part of RetinaNet is divided into two branches, one is the classification subnet, which is used to predict the category to which each anchor belongs; the other is the bounding box regression subnet, which is used to predict the bounding box regression parameters of each anchor. This embodiment constructs an initial recognition model based on the RetinaNet network, which has the following advantages: RetinaNet effectively solves the problems of category imbalance and multi-scale target detection through the combination of focal loss and FPN, thereby improving the detection accuracy of the model; the introduction of FPN enables RetinaNet to simultaneously detect targets of different scales, thereby enhancing the generalization ability of the model; in short, constructing an initial recognition model based on the RetinaNet network has many advantages, including high detection accuracy, strong scalability, fast training speed, strong multi-scale detection capability, and easy implementation and deployment.
[0041] S400, training the initial recognition model through the training set, adjusting the parameters of the initial recognition model, and obtaining a final sorting behavior recognition model after testing with the test set.
[0042] S500: input the real-time acquired sorting image to be identified into the final sorting behavior recognition model, output the detection result, and determine whether to issue an alarm according to the detection result.
[0043] In this embodiment, after the initial recognition model is trained and tested by the training set and the test set, the final sorting behavior recognition model is obtained. Based on the model, the sorting image to be identified can be detected in real time and quickly, so as to obtain whether the sorting image to be identified is a conventional sorting operation image sample, a non-operation behavior image sample, or a violation behavior image sample. If the operator is not detected in the specified time and area for sorting, it means that the sorting image to be identified is a non-operation behavior image sample. At this time, an early warning is issued to remind the management personnel to arrange personnel to perform sorting operations. If the sorting image to be identified is identified as a violation behavior image sample, an early warning is issued to correct the behavior of the operator. The present invention constructs a final sorting behavior recognition model based on the RetinaNet network. Based on the final sorting behavior recognition model, the sorting behavior of the operator can be quickly and accurately automatically identified, which can reduce the need for manual monitoring and allow the management personnel to focus on tasks that require more manual judgment; at the same time, unsafe behaviors that may cause accidents can also be monitored, such as improper weightlifting postures or working in prohibited areas, and timely warnings can be issued to prevent work injuries.
[0044] See also Figure 2 , the second embodiment of the sorting behavior recognition method in the embodiment of the present invention includes:
[0045] S110, acquiring sorting images in different time periods and scenes through cameras set in the sorting area;
[0046] S120, performing data amplification processing on the sorted image to obtain an expanded image sample;
[0047] S130: labeling and classifying the expanded image samples to obtain a preprocessed sorting image data set.
[0048] Specifically, in this embodiment, a camera can be installed in the sorting area to ensure that the equipment can clearly capture the behavior images or videos of the operators in the sorting area in different time periods and different scenes. If the video is taken, the sorting video needs to be decomposed frame by frame, and frame-by-frame images in different time periods and different scenes are selected as samples; since the number of sorting images initially obtained is limited, it is not conducive to training the constructed network, so it is necessary to perform data amplification processing on the obtained sorting images by data amplification to obtain more expanded image samples; then these expanded image samples are annotated and classified with special marks, and the processed expanded image samples are placed in the corresponding directory to form a pre-processed sorting image data set. This embodiment trains the model through a large number of annotated and classified pre-processed sorting image data sets, which can significantly improve the model's recognition accuracy of sorting behavior; at the same time, the data amplification processing enables the model to be exposed to more diverse image samples during the training process, so that it can maintain a high recognition ability when facing various complex situations in practical applications.
[0049] See also Figure 3 , a third embodiment of the sorting behavior recognition method in the embodiment of the present invention includes:
[0050] S121, mirroring, rotating, scaling, cropping, translating or Gaussian noise processing the sorted image to obtain a deformed image sample;
[0051] S122: Perform mosaic processing and / or shearing and fusion processing on the sorted images and the deformed image samples to obtain expanded image samples.
[0052] Specifically, in this embodiment, the initial sorting image is first processed by mirroring, rotating, scaling, cropping, translating or Gaussian noise by commonly used data augmentation means to generate a new image. Although this processing method is a brand new image for the network model and can achieve the purpose of expanding the data set, the deformed image sample processed in this way does not change the label of the original image, and the training effect is not good. Therefore, this embodiment further performs Mosaic processing and / or shear fusion processing on these deformed image samples and the initial sorting image, so that the label of the expanded image sample obtained may be different from the label of the original image, which is conducive to the training of the network model and obtains a network model with better prediction effect. Specifically, Mosaic is a data augmentation technology that can create new training samples by combining two or more images together. As an example: randomly select four images; perform data augmentation operations on each image, including flipping, scaling, color gamut change, etc.; splice these four images together to form a new image, and ensure that the size of each image is equal to the size of the final output image, thereby obtaining a new expanded image sample. The cutting and fusion processing of the image includes the following steps: randomly deleting part of the area of one image in the extended image sample; intercepting an area of the same size from another image in the extended image sample to fill the deleted part of the area of the previous image, and performing full-image soft fusion to generate a new image to achieve data augmentation. This embodiment adopts the CutMix method to achieve data enhancement. CutMix combines CutOut and Mixup. Compared with Cutout, CutMix changes the area deletion operation into intercepting an area of the same size as another image to fill the area, and changes the label of the new image at the same time; Mixup performs full-image soft fusion of the two images, and also soft fusion of the labels of the two images, thereby utilizing the information of the full image. Cutout only drops the image, so it cannot utilize the full image information, and of course does not change the label. Cutmix performs hard fusion of the two images, and adopts the label soft fusion strategy of mixup. Such processing ensures that CutMix does not change the distribution of the entire data set. This embodiment can obtain a more diverse and rich extended image sample set by performing deformation processing and Mosaic / cut fusion processing on the sorted image. This not only helps to improve the generalization and robustness of the model, but also provides strong support for more complex analysis tasks.
[0053] See also Figure 4 The fourth embodiment of the method for identifying the sorting behavior of express mail in the embodiment of the present invention includes:
[0054] S131, using the Labelme tool to perform different feature annotations on the non-operational behaviors and illegal behaviors in the expanded image samples, so as to obtain image samples annotated with different feature information;
[0055] S132. Classify the image samples according to different feature information to obtain regular sorting operation image samples, non-operation behavior image samples and violation behavior image samples, wherein the regular sorting operation image samples, non-operation behavior image samples and violation behavior image samples constitute the preprocessed sorting image data set.
[0056] In this embodiment, Labelme is an open source image annotation tool that supports multiple annotation methods such as polygons, rectangles, circles, etc., and is suitable for complex image annotation tasks. In this embodiment, the labeling software Labelme is used to perform feature annotation on the non-operation behavior and violation behavior areas on the image in the extended image sample. The feature annotation can be a frame selection of different shapes. For example, a special five-pointed star shape is used to frame the non-operation behavior area and serve as feature information, and a special triangle shape is used to frame the violation behavior area and serve as feature information. A corresponding label (such as "violent sorting violation behavior", "operator conversation behavior", etc.) is added to each annotated area, and the annotation result is saved; then the image samples can be classified according to the feature information, that is, the image samples annotated with a five-pointed star frame selection are non-operation behavior image samples, the image samples annotated with a triangle frame selection are violation behavior image samples, and the image samples not annotated with a five-pointed star frame selection and a triangle frame selection are regular sorting operation image samples. The regular sorting operation image samples, non-operation behavior image samples, and violation behavior image samples constitute the preprocessed sorting image data set. This embodiment can significantly improve the image recognition speed and accuracy in the security inspection process and reduce manual intervention and misjudgment through automatic or semi-automatic image annotation and classification technology.
[0057] See also Figure 5 , the fifth embodiment of sorting behavior recognition in the embodiment of the present invention includes:
[0058] S310, building a RetinaNet network framework, wherein the RetinaNet network framework includes a ResNet backbone network, a feature pyramid network, a classification and bounding box regression network, and a loss function module;
[0059] S320, the ResNet backbone network is composed of a plurality of stacked Residual modules, each of which includes a plurality of convolutional layers, a batch normalization layer, and a ReLU activation function. The Residual module realizes residual mapping by introducing a shortcut connection. Inside the Residual module, the input X first undergoes a series of convolution and activation operations to obtain an output F(X). At the same time, the input X is directly transmitted to the output end of the Residual module through a shortcut connection. Finally, the output of the Residual module is the element-by-element addition of F(X) and X, that is, H(X)=F(X)+X.
[0060] Specifically, the RetinaNet network framework consists of a ResNet backbone network, a feature pyramid network, a classification and bounding box regression network, and a loss function module. The ResNet backbone network is composed of multiple Residual modules stacked together. Each Residual module contains multiple convolutional layers, batch normalization layers, and ReLU activation functions. The Residual module implements residual mapping by introducing shortcut connections. Inside the Residual module, the input X first undergoes a series of convolution and activation operations to obtain the output F(X). At the same time, the input X is directly passed to the output of the Residual module through a shortcut connection. Finally, the output of the Residual module is the element-by-element addition of F(X) and X, that is, H(X)=F(X)+X.
[0061] In this embodiment, the FPN structure is introduced into the RetinaNet model. First, feature maps of different levels are generated through the backbone network, and these feature maps have different resolutions and semantic information. Then, the low-resolution high-level features are fused with the high-resolution low-level features by upsampling and lateral connection. The fused feature map contains rich multi-scale information, which can better adapt to the diversity and complexity of the operator's behavior in the sorting area and improve the detection performance of the model.
[0062] FPN (Feature Pyramid Network) includes two paths, top-down and bottom-up. In the top-down path, the output C2~C5 is used as input. First, the number of channels is adjusted by 1×1 convolution, and then C3~C5 is resized by deconvolution. Secondly, the C5 feature layer is upsampled and element-wise added and fused with C4 after deconvolution. The fusion result is added and fused with C3 after deconvolution after upsampling, and the fusion result is fused with C2; in the bottom-up path, the intermediate fusion results are first processed by 3×3 convolution to obtain C2', C3', C4', C5' respectively, and then the detail features C2', C3', 4', C5' are successively superimposed on the higher-level feature map by dense connection. The connection of feature maps of different layers is realized by feature reuse, so as to further enhance the propagation and utilization of low-level information such as contour, detail, and position in deeper feature maps. The feature pyramid network expands the size of high-level feature maps through upsampling and fuses them with low-level feature maps to form new feature maps. These feature maps contain information of different scales, which helps to improve the model's ability to detect multi-scale targets.
[0063] The head classification and bounding box regression network of RetinaNet is divided into two branches, one is the classification subnet, which is used to predict the category to which each anchor belongs; the other is the bounding box regression subnet, which is used to predict the bounding box regression parameters of each anchor.
[0064] This embodiment uses Focal Loss as the classification loss function. When calculating the classification loss, the positive and negative samples are calculated separately. The ECA channel attention module uses the global average pooling without dimensionality reduction to aggregate the convolution features, and then completes the cross-channel information interaction through one-dimensional convolution. GAP is performed on the input feature map, and then the convolution kernel size k is determined by the adaptive function to determine the coverage of the local interaction channel, so that the layer with a larger number of channels can interact more across channels. The sigmoid function is used to generate the weight of each channel, and then the features with channel attention combined with the channel weight are multiplied with the original input features channel by channel to generate weighted features.
[0065] See also Figure 6 The sixth embodiment of sorting behavior recognition in the embodiment of the present invention includes:
[0066] S410, using a random initialization method to assign initial values to all parameters in the initial recognition model;
[0067] S420, randomly selecting image samples from the training set and inputting them into the initial recognition model for training, calculating the prediction results through forward propagation, and then calculating the loss value according to the Focal Loss function;
[0068] S430, using the back propagation algorithm to update the parameters of the network so that the loss value gradually decreases;
[0069] S440, repeating the above training steps until a predetermined training round is reached to obtain a trained initial recognition model;
[0070] S450: Use the test set to evaluate the trained initial recognition model, adjust the hyperparameters according to the evaluation results, and obtain the final sorting behavior recognition model.
[0071] Specifically, before model training, a random initialization method is used to assign initial values to all parameters in the initial recognition model, such as setting the initial number of iterations (such as 5000 times), the initial learning rate (such as 10-5) and batch_size (such as 300), as well as setting the data set path parameters and category parameters, and using a parallel GPU for training; a batch of images and corresponding bounding box annotation data are randomly selected from the training set and input into the model for training; after feature extraction and prediction branch processing, the predicted category probability and bounding box position are obtained; according to the prediction results and the true annotations, the classification loss and regression loss are calculated, the classification loss uses focal loss, and the regression loss uses smooth L1 loss. Among them, focal loss is used as a classification loss function, and the classification loss function is used to measure the difference between the category probability predicted by the initial recognition model and the actual category. The calculation formula of focal loss is as follows: L_cls = -α(1-p_t)^γlog(p_t), where p_t is the category probability predicted by the initial recognition model, α and γ are hyperparameters used to adjust the weights of samples of different difficulty levels. When p_t is close to 1, (1-p_t)^γ will become smaller, making the loss value of the sample smaller; conversely, when p_t is close to 0, (1-p_t)^γ will become larger, making the loss value of the sample larger; smooth L1 loss is used as a regression loss function, and the regression loss function is used to measure the difference between the bounding box position predicted by the initial recognition model and the actual bounding box position. The calculation formula of smooth L1 loss is as follows: L_reg = 0.5*(IoU-1)^2*1if|IoU-1|<1, otherwise L_reg = |IoU-1|-0.5, where IoU represents the IoU value between the predicted bounding box and the true bounding box. When the IoU value is close to 1, it means that the predicted box is very close to the true box, and the loss value is small at this time; when the IoU value deviates from 1, the loss value will gradually increase.
[0072] Then, according to the gradient of the loss function, backpropagate to all parameters of the model, update the parameter values, and repeat the training until the preset number of training rounds is reached; use the test set to evaluate the performance of the trained model, calculate indicators such as accuracy, recall rate, mAP, etc. to evaluate the performance of the model; finally, test and verify the results, adjust hyperparameters such as learning rate, batch size, etc. to optimize the performance of the model, and obtain the final sorting behavior recognition model.
[0073] This embodiment solves the class imbalance problem existing in most target detection data sets based on the focal loss loss function. In traditional target detection tasks, the background area is usually much larger than the foreground area, which causes the model to overfit the background area during training and ignore the foreground area. Focal loss solves this problem by dynamically reducing the contribution of well-classified samples while emphasizing the importance of difficult-to-classify samples. Specifically, focal loss introduces a modulation factor based on the cross-entropy loss, which reduces the loss of easy-to-classify samples and increases the loss of difficult-to-classify samples.
[0074] The present invention constructs a final sorting behavior recognition model based on the RetinaNet network. Based on the final sorting behavior recognition model, the sorting behavior of the operator can be quickly and accurately automatically recognized, which can reduce the need for manual monitoring and allow managers to focus on tasks that require more manual judgment. At the same time, unsafe behaviors that may lead to accidents can also be monitored, such as improper weightlifting postures or working in prohibited areas, and timely warnings can be issued to prevent work-related injuries.
[0075] The above describes the sorting behavior recognition method in the embodiment of the present invention. The following describes the sorting behavior recognition device in the embodiment of the present invention. Figure 7 , an embodiment of the sorting behavior recognition device in the embodiment of the present invention includes:
[0076] The labeling and classification module 10 is used to label and classify the collected sorting images to obtain a pre-processed sorting image data set;
[0077] A data partitioning module 20, for partitioning the pre-processed sorting image data set into a training set and a test set according to a predetermined ratio, wherein the training set and the test set both include conventional sorting operation image samples, non-operation behavior image samples, and violation behavior image samples;
[0078] A model building module 30, used to build an initial recognition model based on the RetinaNet network;
[0079] A training module 40 is used to train the initial recognition model through the training set, adjust the parameters of the initial recognition model, and obtain a final sorting behavior recognition model after testing with the test set;
[0080] The recognition module 50 is used to input the real-time acquired sorting image to be recognized into the final sorting behavior recognition model, output the detection result, and determine whether to issue an alarm according to the detection result.
[0081] The present invention constructs a final sorting behavior recognition model based on the RetinaNet network. Based on the final sorting behavior recognition model, the sorting behavior of the operator can be quickly and accurately automatically recognized, which can reduce the need for manual monitoring and allow managers to focus on tasks that require more manual judgment. At the same time, unsafe behaviors that may lead to accidents can also be monitored, such as improper weightlifting postures or working in prohibited areas, and timely warnings can be issued to prevent work-related injuries.
[0082] See also Figure 8 Another embodiment of the sorting behavior recognition device in the embodiment of the present invention includes:
[0083] The labeling and classification module 10 is used to label and classify the collected sorting images to obtain a pre-processed sorting image data set;
[0084] A data partitioning module 20, for partitioning the pre-processed sorting image data set into a training set and a test set according to a predetermined ratio, wherein the training set and the test set both include conventional sorting operation image samples, non-operation behavior image samples, and violation behavior image samples;
[0085] A model building module 30, used to build an initial recognition model based on the RetinaNet network;
[0086] A training module 40 is used to train the initial recognition model through the training set, adjust the parameters of the initial recognition model, and obtain a final sorting behavior recognition model after testing with the test set;
[0087] The recognition module 50 is used to input the real-time acquired sorting image to be recognized into the final sorting behavior recognition model, output the detection result, and determine whether to issue an alarm according to the detection result.
[0088] In this embodiment, the labeling classification module 10 includes:
[0089] An acquisition unit 11 is used to acquire sorting images in different time periods and scenes through a camera arranged in the sorting area;
[0090] An amplification unit 12, used for performing data amplification processing on the sorted image to obtain an expanded image sample;
[0091] The classification unit 13 is used to label and classify the expanded image samples to obtain a pre-processed sorted image data set.
[0092] In this embodiment, the amplification unit 12 includes:
[0093] The deformation subunit 121 is used to perform mirroring, rotation, scaling, cropping, translation or Gaussian noise processing on the sorted image to obtain a deformed image sample;
[0094] The transformation subunit 122 is used to perform mosaic processing and / or cutting and fusion processing on the sorted image and the deformed picture sample to obtain an expanded picture sample.
[0095] In this embodiment, the classification unit 13 includes:
[0096] The labeling subunit 131 is used to label the non-operational behaviors and illegal behaviors in the expanded image samples with different features through the Labelme tool to obtain image samples labeled with different feature information;
[0097] The classification subunit 132 is used to classify the image samples according to different feature information to obtain regular sorting operation image samples, non-operation behavior image samples and violation behavior image samples, and the regular sorting operation image samples, non-operation behavior image samples and violation behavior image samples constitute the pre-processed sorting image data set.
[0098] In this embodiment, the model building module 30 includes:
[0099] Building unit 31, building a RetinaNet network framework, wherein the RetinaNet network framework includes a ResNet backbone network, a feature pyramid network, a classification and bounding box regression network, and a loss function module;
[0100] The introduction unit 32 is used for the ResNet backbone network to be stacked with multiple Residual modules, each of which includes multiple convolutional layers, batch normalization layers and ReLU activation functions. The Residual module realizes residual mapping by introducing shortcut connections. Inside the Residual module, the input X first undergoes a series of convolution and activation operations to obtain the output F(X). At the same time, the input X is directly passed to the output end of the Residual module through the shortcut connection. Finally, the output of the Residual module is the element-by-element addition of F(X) and X, that is, H(X)=F(X)+X.
[0101] In this embodiment, the training module 40 includes:
[0102] An initialization unit 41 is used to assign initial values to all parameters in the initial recognition model using a random initialization method;
[0103] A calculation unit 42 is used to randomly select image samples from the training set and input them into the initial recognition model for training, calculate the prediction results through forward propagation, and then calculate the loss value according to the Focal Loss function;
[0104] An updating unit 43 is used to update the parameters of the network using a back propagation algorithm so that the loss value gradually decreases;
[0105] A repeating unit 44 is used to repeat the above training steps until a predetermined training round is reached to obtain a trained initial recognition model;
[0106] The evaluation unit 45 is used to evaluate the trained initial recognition model using the test set, and adjust the hyperparameters according to the evaluation results to obtain the final sorting behavior recognition model.
[0107] In this embodiment, the loss calculation unit 43 includes:
[0108] The classification loss calculation subunit 431 is used to use focal loss as a classification loss function, and the classification loss function is used to measure the difference between the category probability predicted by the initial recognition model and the actual category. The calculation formula of focal loss is as follows: L_cls = -α(1-p_t)^γlog(p_t), where p_t is the category probability predicted by the initial recognition model, α and γ are hyperparameters used to adjust the weights of samples of different difficulty levels. When p_t is close to 1, (1-p_t)^γ will become smaller, so that the loss value of the sample becomes smaller; conversely, when p_t is close to 0, (1-p_t)^γ will become larger, so that the loss value of the sample becomes larger;
[0109] The regression loss calculation subunit 432 is used to use smooth L1 loss as the regression loss function, and the difference between the bounding box position predicted by the initial recognition model and the real bounding box position is measured by the regression loss function. The calculation formula of SmoothL1 loss is as follows: L_reg=0.5*(IoU-1)^2*1; if |IoU-1|<1, then L_reg=|IoU-1|-0.5, where IoU represents the IOU value of the predicted bounding box and the real bounding box. When the IoU value is close to 1, it means that the predicted box is very close to the real box, and the loss value is small at this time; when the IoU value deviates from 1, the loss value gradually increases.
[0110] above Figure 7 and Figure 8The sorting behavior recognition device in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The sorting behavior recognition device in the embodiment of the present invention is described in detail from the perspective of hardware processing.
[0111] Fig. 9 1 is a schematic diagram of the structure of a sorting behavior recognition device provided by an embodiment of the present invention. The sorting behavior recognition device 100 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 11 (for example, one or more processors) and a memory 12, and one or more storage media 13 (for example, one or more mass storage devices) storing application programs 133 or data 132. Among them, the memory 12 and the storage medium 13 can be short-term storage or permanent storage. The program stored in the storage medium 13 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the sorting behavior recognition device 100. Furthermore, the processor 11 can be configured to communicate with the storage medium 13 to execute a series of instruction operations in the storage medium 13 on the sorting behavior recognition device 100.
[0112] The sorting behavior recognition device 100 may also include one or more power supplies 14, one or more wired or wireless network interfaces 15, one or more input and output interfaces 16, and / or one or more operating systems 131, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that Figure 8 The device structure shown does not constitute a limitation on the sorting behavior recognition device 100, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0113] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the sorting behavior identification method.
[0114] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device, or unit can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.
[0115] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.
[0116] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A sorting behavior recognition method, characterized in that: The sorting behavior identification method comprises: Annotate and classify the collected sorting images to obtain a preprocessed sorting image data set; Dividing the preprocessed sorting image data set into a training set and a test set according to a predetermined ratio, wherein the training set and the test set both include conventional sorting operation image samples, non-operation behavior image samples, and violation behavior image samples; Build an initial recognition model based on the RetinaNet network; The initial recognition model is trained by the training set, the parameters of the initial recognition model are adjusted, and after being tested by the test set, a final sorting behavior recognition model is obtained; The real-time acquired sorting image to be identified is input into the final sorting behavior identification model, and the detection result is output, and it is determined whether to issue an alarm according to the detection result.
2. The sorting behavior recognition method according to claim 1, characterized in that: The collected sorting images are labeled and classified to obtain a preprocessed sorting image data set, including the following steps: The cameras set up in the sorting area are used to obtain sorting images in different time periods and scenes; Performing data amplification processing on the sorted images to obtain expanded image samples; The expanded image samples are labeled and classified to obtain a preprocessed sorted image data set.
3. The sorting behavior recognition method according to claim 2, characterized in that: Performing data augmentation processing on the sorted image to obtain an expanded image sample includes the following steps: Mirroring, rotating, scaling, cropping, translating or Gaussian noise processing are performed on the sorted image to obtain a deformed image sample; Mosaic processing and / or cutting and fusion processing are performed on the sorted images and the deformed picture samples to obtain expanded picture samples.
4. The sorting behavior recognition method according to claim 2, characterized in that: The expanded image samples are labeled and classified to obtain a preprocessed sorted image data set, including the following steps: Use the Labelme tool to label the non-operational behaviors and illegal behaviors in the expanded image samples with different features to obtain image samples labeled with different feature information; Image samples are classified according to different feature information to obtain conventional sorting operation image samples, non-operation behavior image samples and violation behavior image samples, which constitute the preprocessed sorting image data set.
5. The sorting behavior recognition method according to claim 1, characterized in that: Building an initial recognition model based on the RetinaNet network includes the following steps: Build a RetinaNet network framework, which includes a ResNet backbone network, a feature pyramid network, a classification and bounding box regression network, and a loss function module; The ResNet backbone network is composed of multiple stacked Residual modules, each of which contains multiple convolutional layers, batch normalization layers, and ReLU activation functions. The Residual module implements residual mapping by introducing shortcut connections. Inside the Residual module, the input X first undergoes a series of convolution and activation operations to obtain the output F(X). At the same time, the input X is directly passed to the output end of the Residual module through a shortcut connection. Finally, the output of the Residual module is the element-by-element addition of F(X) and X, that is, H(X)=F(X)+X.
6. The sorting behavior recognition method according to claim 1, characterized in that: The initial recognition model is trained by the training set, the parameters of the initial recognition model are adjusted, and after being tested by the test set, a final sorting behavior recognition model is obtained, including the steps of: Using a random initialization method to assign initial values to all parameters in the initial identification model; Randomly select image samples from the training set and input them into the initial recognition model for training, calculate the prediction results through forward propagation, and then calculate the loss value according to the Focal Loss function; Use the back propagation algorithm to update the parameters of the network so that the loss value gradually decreases; Repeat the above training steps until the predetermined training rounds are reached to obtain a trained initial recognition model; The trained initial recognition model is evaluated using the test set, and the hyperparameters are adjusted according to the evaluation results to obtain the final sorting behavior recognition model.
7. The sorting behavior recognition method according to claim 6, characterized in that: Calculate the loss value according to the Focal Loss function, including the following steps: Focal loss is used as the classification loss function, and the classification loss function is used to measure the difference between the category probability predicted by the initial recognition model and the actual category. The calculation formula of focal loss is as follows: L_cls = -α(1-p_t)^γlog(p_t), where p_t is the category probability predicted by the initial recognition model, α and γ are hyperparameters used to adjust the weights of samples of different difficulty levels. When p_t is close to 1, (1-p_t)^γ will become smaller, making the loss value of the sample smaller; conversely, when p_t is close to 0, (1-p_t)^γ will become larger, making the loss value of the sample larger; Smooth L1 loss is used as the regression loss function. The regression loss function is used to measure the difference between the bounding box position predicted by the initial recognition model and the true bounding box position. The calculation formula of Smooth L1 loss is as follows: L_reg = 0.5*(IoU-1)^2*1; if |IoU-1|<1, then L_reg = |IoU-1|-0.5, where IoU represents the IOU value of the predicted bounding box and the true bounding box. When the IoU value is close to 1, it means that the predicted box is very close to the true box, and the loss value is small at this time; when the IoU value deviates from 1, the loss value gradually increases.
8. A sorting behavior recognition device, characterized in that: include: The labeling and classification module is used to label and classify the collected sorting images to obtain a preprocessed sorting image data set; A data partitioning module, used for partitioning the pre-processed sorting image data set into a training set and a test set according to a predetermined ratio, wherein the training set and the test set both include conventional sorting operation image samples, non-operation behavior image samples, and violation behavior image samples; Model building module, used to build the initial recognition model based on the RetinaNet network; A training module, used to train the initial recognition model through the training set, adjust the parameters of the initial recognition model, and obtain a final sorting behavior recognition model after testing with the test set; The recognition module is used to input the real-time acquired sorting image to be recognized into the final sorting behavior recognition model, output the detection result, and determine whether to issue an alarm according to the detection result.
9. A sorting behavior recognition device, characterized in that: comprising a memory and at least one processor, wherein the memory has computer-readable instructions stored therein; The at least one processor calls the computer-readable instructions in the memory to execute the various steps of the sorting behavior identification method according to any one of claims 1-7.
10. A computer-readable storage medium having computer-readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, the various steps of the sorting behavior identification method according to any one of claims 1 to 7 are implemented.
