Multimodal-based Violent Sorting and Loading / Unloading Action Recognition Method, System and Device

Abstract

The present invention belongs to the field of action recognition, and particularly relates to a multi-modal based method, system and device for violent sorting and loading / unloading action recognition, aiming to solve the problem that the existing supervised detection methods for violent sorting and loading / unloading can only rely on supervisors' reminders or the transparency of luggage sorting monitoring, which requires a large amount of manpower. The present invention includes: obtaining the first behavior data and video data of the person to be recognized for actions; after being converted by a long short-term memory time series network and input into a data analysis algorithm model, obtaining the first motion state data; obtaining the second motion state data based on the multi-modal fusion of tracking box data, optical flow features and text information data; based on an end-to-end system with cloud collaboration and combining the first and second motion state data, obtaining the action recognition result of the person. The present invention combines the first motion state data with the action recognition detection result of multi-modal fusion to reduce the misdetection degree of violent behaviors in personnel sorting and unloading.

Description

Background Art

[0002] In the ground support procedures for flight departures and arrivals, the luggage sorting and handling work involves multiple links. Due to human reasons, the checked luggage of passengers is damaged, and the industry loses more than ten million yuan annually. The existing methods can only remind through other supervisors or make the luggage sorting monitoring transparent. It is supervised by passengers to prevent violent sorting and handling of luggage, but the effect is still not obvious enough, and it requires a large amount of manpower at the same time.

[0003] "Active Express Sorting Operation Assistance and Evaluation System and Method for Preventing Violent Sorting" proposes a method for preventing violent sorting of vulnerable items in the express delivery industry. However, due to the differences in sorting environments and sorted items, this method is not applicable to the luggage sorting scenario of civil aviation; "A Logistics Real-Time Monitoring Device Based on LoRa for Preventing Violent Sorting and Loss Prevention" proposes to monitor and reduce the loss of logistics express items and the damage caused by violent sorting through a logistics real-time monitoring device. However, the entire method involves overly complex hardware devices and is not applicable to the airport sorting environment; "A Logistics Storage Device for Preventing Breakage" proposes a logistics storage device for preventing breakage, including a feeding unit, a conveying unit, a storage unit, an output unit, and a controller, which is applicable to the protection of small items with regular shapes, but is not applicable to the large and irregular items that need to be protected during airport sorting.

[0004] The present invention aims to detect the violent sorting actions of loading and unloading personnel based on the AI deep learning algorithm, and transmit the results analyzed by the algorithm to the background in real time for managers to monitor and perform data analysis and statistics. After the edge-end bracelet APP receives the alarm information, it vibrates to remind the loading and unloading personnel to pay attention to safety. Based on this, the present invention provides a method, system, and device for identifying violent sorting and loading / unloading actions based on multi-modal. Summary of the Invention

[0005] To solve the above problems in the prior art, the present invention provides a method, system, and device for identifying violent sorting and loading / unloading actions based on multi-modal.

[0006] On the one hand, the present invention proposes a method for identifying violent sorting and loading / unloading actions based on multi-modal, and the method includes;

[0007] Obtain the first behavior data and video data of the person to be identified for action; the first behavior data is the motion data and heart rate data of the person; the actions to be identified include violent sorting and violent loading / unloading;

[0008] Convert the first behavior data into second behavior data, and input the second behavior data into a trained data analysis algorithm model to obtain the first motion state data;

[0009] Based on the video data, obtain multiple tracking box data, multiple optical flow features, and text information data. Perform multi-modal fusion on the multiple tracking box data and the text information data to obtain fusion data, and then fuse the fusion data with the optical flow features to obtain second motion state data; obtain the action recognition result of the person according to the first motion state data and the second motion state data.

[0010] In some preferred embodiments, the training method of the data analysis algorithm model is as follows:

[0011] Obtain multiple pieces of the first behavior data at time t; t > 1;

[0012] Obtain multiple pieces of the first behavior data at time t - 1, input them into a long short-term memory time series network, and obtain the predicted data corresponding to the first behavior data at time t;

[0013] Fuse the first behavior data and its corresponding predicted data, and use the fused data as the second behavior data;

[0014] Encode the second behavior data into M vectors of size 1xN, and construct an M*N matrix, where N is the sum of the number of the motion data and the number of the heart rate data;

[0015] Input the constructed M*N matrix into a logistic regression model for training after encoding, and use the trained logistic regression model as the data analysis algorithm model.

[0016] In some preferred embodiments, the motion data includes: the swinging direction data, swinging angular velocity data, swinging linear velocity data, swinging frequency data, and swinging amplitude data of the sorting and / or loading and unloading personnel.

[0017] In some preferred embodiments, the method for obtaining the tracking box data is as follows:

[0018] Obtain the video data of the person to be recognized for sorting and / or unloading actions as the first video stream data;

[0019] Extract the human body regions in each video frame of the first video stream data to construct the second video stream data;

[0020] Input the second video stream data into a target detection network to obtain the first target box data, and input the first target box data into a linear model to predict and obtain the second target box data, where the second target box data is the data of the next frame of the video frame where the first target box data is located;

[0021] Associate the first target detection box data and the second target box data through similarity calculation to obtain tracking box data corresponding to each frame of image;

[0022] Fuse and train the tracking box data with the optical flow feature algorithm to obtain an action recognition algorithm model.

[0023] In some preferred embodiments, the method for extracting the human body region in each video frame of the first video stream data is as follows:

[0024] Input the first video stream data into a cascaded sparse query algorithm based on high resolution to obtain rough position data of the human body region in the first video stream data, and calculate the accurate position region data of the human body region through sparse convolution as the second video stream data.

[0025] In some preferred embodiments, the method for obtaining the second motion state data is as follows: Input the text information data and the multiple tracking box data into a clip multi-modal algorithm model to obtain fusion data; Input the video data into an optical flow algorithm model to obtain optical flow features, and fuse the optical flow features and the fusion data again at the head layer of the optical flow algorithm model to obtain the second motion state data.

[0026] In some preferred embodiments, the method for obtaining the action recognition result of a person according to the first motion state data and the second motion state data is as follows:

[0027] Judge whether the first motion state data and the second motion state data are consistent;

[0028] If they are consistent, then judge whether both the first motion state data and the second motion state data are violent sorting and / or violent loading and unloading;

[0029] If not, do not output an alarm signal;

[0030] If so, upload the first video stream data corresponding to the second motion data to a first mobile terminal and output an alarm signal to a second mobile terminal;

[0031] If they are inconsistent, input the first motion state data and the second motion state data into a Gaussian mixture model according to the weighted average method and in combination with the historical first motion state data and second motion state data to obtain an action recognition result;

[0032] If it is not violent sorting and / or violent loading and unloading, do not output an alarm signal;

[0033] If it is violent sorting and / or violent loading and unloading, the first video stream data in the second motion data corresponding to the third motion data will be uploaded to the first mobile terminal, and an alarm signal will be output to the second mobile terminal.

[0034] On the other hand, the present invention proposes a multi-modal-based violent sorting and loading / unloading action recognition system, which includes a data acquisition module, a data model construction module, an action model construction module, and a judgment module;

[0035] The data acquisition module is configured to acquire first behavior data, and the first behavior data is the motion data and heart rate data of the loading and unloading personnel;

[0036] The first training module is configured to convert the first behavior data into second behavior data, input the second behavior data into a trained data analysis algorithm model, and obtain first motion state data;

[0037] The second training module is configured to, based on the video data, acquire multiple tracking box data, multiple optical flow features, and text information data, perform multi-modal fusion on the multiple tracking box data and the text information data to obtain fusion data, and then fuse the fusion data with the optical flow features to obtain second motion state data;

[0038] The judgment module is configured to obtain the action recognition result of the person according to the first motion state data and the second motion state data.

[0039] In the third aspect of the present invention, a storage device is proposed, in which multiple programs are stored, and the programs are suitable for being loaded and executed by a processor to implement a multi-modal-based violent sorting and loading / unloading action recognition method.

[0040] In the fourth aspect of the present invention, a processing device, a processor, is suitable for executing each program; a storage device is suitable for storing multiple programs; and it is characterized in that the programs are suitable for being loaded and executed by the processor to implement a multi-modal-based violent sorting and loading / unloading action recognition method.

[0041] The beneficial effects of the present invention:

[0042] Traditional violent behaviors of personnel sorting and unloading are greatly affected by the environment and hardware devices. Generally speaking, compared with traditional technologies, the advantages of the present invention are that it uses AI deep learning algorithms for behavior analysis. Compared with single traditional machine learning algorithms, the present invention integrates deep learning algorithms based on personnel action detection and multi-object tracking, greatly improving the recognition accuracy. Through the combination and optimization of the multi-modal action recognition network, the accuracy of action recognition is improved, the false detection degree of violent behaviors of personnel sorting and unloading is reduced, and the inference speed of the model is increased. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Other features, objects, and advantages of the present application will become more apparent by reading the following detailed description of non - limiting embodiments with reference to the accompanying drawings:

[0044] Figure 1 is an overall schematic diagram of the multi - modal based violent sorting and loading / unloading action recognition method of the present invention;

[0045] Figure 2 is a network flow chart for constructing the second motion state data in the multi - modal based violent sorting and loading / unloading action recognition method of the present invention;

[0046] Figure 3 is a flow chart for obtaining the second motion state data in the multi - modal based violent sorting and loading / unloading action recognition method of the present invention;

[0047] Figure 4 is a schematic diagram of the fusion of text information data and tracking frame data in the multi - modal based violent sorting and loading / unloading action recognition method of the present invention;

[0048] Figure 5 is a flow chart of the multi - modal based violent sorting and loading / unloading action recognition method of the present invention;

[0049] Figure 6 is a module structure diagram of the multi - modal based violent sorting and loading / unloading action recognition system of the present invention;

[0050] Figure 7 is a schematic diagram of the structure of a computer system of a server for implementing the method, system, and device embodiments of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0051] The present application will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only for explaining the related invention and not for limiting the invention. Additionally, it should be noted that for the sake of description, only parts related to the relevant invention are shown in the drawings.

[0052] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the drawings and embodiments.

[0053] As Figures 1-5 shown, referring to Figure 1 , the present invention provides a multi - modal based violent sorting and loading / unloading action recognition method, and the method includes:

[0054] Obtain the first behavior data and video data of the person to be subject to action recognition; the first behavior data is the movement data and heart rate data of the person; the actions to be recognized include violent sorting and violent unloading.

[0055] Convert the first behavior data into second behavior data, and input the second behavior data into the trained data analysis algorithm model to obtain the first motion state data.

[0056] Based on the video data, obtain multiple tracking box data, multiple optical flow features and text information data, perform multimodal fusion on the multiple tracking box data and the text information data to obtain fusion data, and then fuse the fusion data with the optical flow features to obtain the second motion state data.

[0057] Obtain the action recognition result of the person according to the first motion state data and the second motion state data.

[0058] Preferably, refer to Figure 2 , the training method of the data analysis algorithm model is as follows:

[0059] Obtain multiple pieces of the first behavior data at time t; t>1.

[0060] Obtain multiple pieces of the first behavior data at time t - 1, input them into the long short-term memory time series network, and obtain the predicted data corresponding to the first behavior data at time t.

[0061] Fuse the first behavior data and its corresponding predicted data, and use the fused data as the second behavior data.

[0062] Encode the second behavior data into M vectors of size 1xN, and construct an M*N matrix, where N is the sum of the number of the movement data and the number of the heart rate data.

[0063] Encode the constructed M*N matrix and input it into the logistic regression model for training, and use the trained logistic regression model as the data analysis algorithm model.

[0064] Among them, the method of encoding the constructed M*N matrix is: perform 01 encoding on the M*N matrix.

[0065] Preferably, the movement data includes: the swinging arm direction data, swinging arm angular velocity data, swinging arm linear velocity data, swinging arm frequency data, and swinging arm amplitude data of the sorting and / or loading and unloading personnel.

[0066] Preferably, refer to Figure 3 , the method for obtaining the tracking box data is as follows:

[0067] Obtain the video data of the person to be identified for sorting and / or unloading actions as the first video stream data;

[0068] Extract the human body regions in each video frame of the first video stream data to construct the second video stream data;

[0069] Input the second video stream data into the target detection network to obtain the first target box data, and input the first target box data into the linear model to predict and obtain the second target box data, where the second target box data is the data of the next frame of the video frame where the first target box data is located;

[0070] Perform data association on the first target detection box data and the second target box data through similarity calculation to obtain the tracking box data corresponding to each frame of image;

[0071] Fuse and train the tracking box data with the optical flow feature algorithm to obtain the action recognition algorithm model.

[0072] Among them, input a video stream of personnel sorting and loading / unloading, and the algorithm performs multi-target tracking and detection based on consecutive frames. For multi-target tracking, through the classic faster-RCNN target detection algorithm, detect the targets in each frame of the video, then input the detected box region into the linear model Kalman filter to predict their positions in the next frame, and finally perform Hungarian matching (data association) on the position prediction result and the target detection box through the IoU value (similarity calculation), so as to obtain the tracking box, and further obtain the unique identification ID of multiple personnel in each frame of image.

[0073] Among them, in terms of improving the algorithm speed, the present invention analyzes and optimizes the algorithm model training and inference processes respectively. In the training stage, we propose the Denosing denoising training method to improve the problem of instability based on Hungarian matching. Specifically: randomly add noise to the real labels with a certain probability, so in the calculation of the classic task model loss, an additional part of the reconstruction loss function MSE is added, and its calculation method is:

[0074]

[0075] Among them, the y i is the result of inputting the randomly noised image of each frame into the model for prediction, is the result of inputting the image of each frame without loading into the model for prediction.

[0076] Among them, the present invention proposes a tensor calculation method based on "low-tubal rank", and proposes a tensor singular value decomposition of "t-SVD", and its calculation method is as follows:

[0077]

[0078] Among them, the is a third-order tensor, which is the image feature corresponding to each frame of the image. fft represents the fast Fourier transform, SVD represents the feature singular value decomposition, U, S, and V respectively represent the left singular value, the singular value, and the right singular value; ifft represents the inverse Fourier transform.

[0079] Among them, the t-SVD decomposition used in the present invention is a method that optimizes the extraction of the image features corresponding to each frame of the image. While accelerating the feature extraction, it increases the accuracy of the extraction. Among the finally obtained U, S, and V, the feature vector S used in the present invention is used for fusion with the text information data.

[0080] Among them, the "low-tubal rank tSVD" tensor product operation based on the present invention takes into account the adjacent two-frame action timing information, expands one of the tensors into the form of a circulant matrix, and models the adjacent two-frame action information in chronological order, which is different from the previous unordered expansion by row / column;

[0081]

[0082] Among them, circ is the circulant matrix, and A is the first block vector; the A is the block vector corresponding to each frame of the image, specifically, the third-order tensor of each frame of the image is expanded along the third dimension, that is, the first block vector.

[0083] Based on the fast Fourier transform and the inverse transform, the present invention can expand the high-dimensional tensor into the form of a block diagonal matrix, which greatly reduces the computational time complexity. The calculation method is as follows:

[0084]

[0085] Among them, D is the second block vector; F is the fft vector; I is the identity matrix; is a third-order tensor; MatVec is a vector, circ: block circulant matrix;

[0086] Among them, MatVec vector: expanding the along the third dimension can obtain C blocks of H*W block vectors, which are combined into a vector form [A_1, A_2,..., A_c].

[0087] The circ block circulant matrix: performs a shift operation on the above MatVec vector

[0088] The above two matrices are: the transformation forms of the underlying tensors of the tSVD decomposition.

[0089] Preferably, the method for extracting the human body regions in each video frame of the first video stream data is as follows:

[0090] Input the first video stream data into a cascaded sparse query algorithm based on high resolution to obtain the rough position data of the human body regions in the first video stream data, and calculate the accurate position region data of the human body regions through sparse convolution as the second video stream data.

[0091] Preferably, refer to Figure 4 , the method for obtaining the second motion state data is as follows: Input the text information data and the multiple tracking box data into a clip multi-modal algorithm model to obtain fusion data; input the video data into an optical flow algorithm model to obtain optical flow features, and fuse the optical flow features and the fusion data again at the head layer of the optical flow algorithm model to obtain the second motion state data.

[0092] Among them, supervised learning is performed by constructing a similarity matrix in the process of obtaining the second motion state data.

[0093] In the present invention, "inputting the text information data and the multiple tracking box data into a clip multi-modal algorithm model" is a multi-modal information input that fuses text description features and video image features, constructs tens of millions of data sample pairs, and designs the model size with 100 million parameters, which has a good effect on few-shot / zero-shot transfer learning.

[0094] Preferably, the method for determining whether there is violent sorting by the loading and unloading personnel according to the first motion state data and the second motion state data is as follows:

[0095] Determine whether the first motion state data and the second motion state data are consistent;

[0096] If they are consistent, determine whether both the first motion state data and the second motion state data are violent sorting and / or violent loading and unloading;

[0097] If not, do not output an alarm signal;

[0098] If so, upload the first video stream data corresponding to the second motion data to a first mobile terminal and output an alarm signal to a second mobile terminal;

[0099] If they are inconsistent, input the first motion state data and the second motion state data into a Gaussian mixture model according to the weighted average method and in combination with the historical first motion state data and second motion state data to obtain an action recognition result;

[0100] If it is not violent sorting and / or violent loading and unloading, no alarm signal is output;

[0101] If it is violent sorting and / or violent loading and unloading, the first video stream data in the second motion data corresponding to the third motion data is uploaded to the first mobile terminal, and an alarm signal is output to the second mobile terminal.

[0102] Among them, the first mobile terminal is the screen displayed in the background, and the second mobile terminal is an intelligent wearable system set on the sorting and loading / unloading personnel, such as an intelligent bracelet, an intelligent watch, etc. The second mobile terminal reminds the sorting and loading / unloading personnel by means of alarm or vibration. Although the various steps are described in the above order in the above embodiments, those skilled in the art can understand that in order to achieve the effects of this embodiment, different steps do not have to be executed in such an order, and they can be executed simultaneously (in parallel) or in a reversed order. These simple changes are all within the protection scope of the present invention.

[0103] See Figure 6 , the multi-modal based violent sorting and loading / unloading action recognition system according to the second embodiment of the present invention, the system includes a data acquisition module 100, a first training module 200, a second training module 300, and a judgment module 400;

[0104] The data acquisition module 100 is configured to acquire first behavior data, and the first behavior data is the motion data and heart rate data of the loading / unloading personnel;

[0105] The first training module 200 is configured to convert the first behavior data into second behavior data, input the second behavior data into a trained data analysis algorithm model, and obtain first motion state data;

[0106] The second training module 300 is configured to, based on the video data, acquire a plurality of tracking box data, a plurality of optical flow features, and text information data, perform multi-modal fusion on the plurality of tracking box data and the text information data to obtain fusion data, and perform further fusion on the fusion data and the optical flow features to obtain second motion state data;

[0107] The judgment module 400 is configured to obtain an action recognition result of the personnel according to the first motion state data and the second motion state data.

[0108] Among them, a combination based on the clip multi-modal algorithm model, the optical flow algorithm model, the target detection network, and the linear model is used as the action recognition algorithm model. Both the data analysis algorithm model and the action recognition algorithm model are uploaded to the cloud and converted into tensorRT, and then sent to the edge side for model deployment and inference, so as to perform real-time judgment and analysis.

[0109] Among them, when there is violent sorting or violent loading and unloading, the video data is saved to the cloud.

[0110] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working process and related descriptions of the above-described system can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0111] It should be noted that the above-described multi-modal-based violent sorting and loading / unloading action recognition system provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiments can be combined into one module, or further split into multiple sub-modules to complete all or part of the functions described above. For the names of the modules and steps involved in the embodiments of the present invention, they are only used to distinguish each module or step and are not regarded as an improper limitation of the present invention.

[0112] A storage device according to the third embodiment of the present invention stores multiple programs: the programs are suitable for being loaded and executed by a processor to implement a multi-modal-based violent sorting and loading / unloading action recognition method.

[0113] A processing device according to the fourth embodiment of the present invention includes a processor and a storage device; the processor is suitable for executing each program; the storage device is suitable for storing multiple programs; the programs are suitable for being loaded and executed by the processor to implement a multi-modal-based violent sorting and loading / unloading action recognition method.

[0114] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working process and related descriptions of the above-described storage device and processing device can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0115] Those skilled in the art should be able to realize that the modules and method steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. The programs corresponding to the software modules and method steps can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the technical field. To clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in the form of electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0116] Reference is made below to Figure 7 , which shows a schematic structural diagram of a computer system of a server for implementing the method, system, and device embodiments of the present application. Figure 7 The server shown is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present application.

[0117] As Figure 7 shown, the computer system includes a central processing unit (CPU, Central Processing Unit) 701, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 702 or the program loaded from the storage section 708 into the random access memory (RAM, Random Access Memory) 703. In the RAM 703, various programs and data required for system operation are also stored. The CPU 701, ROM 702, and RAM 703 are connected to each other through a bus 704. The input / output (I / O, Input / Output) interface 705 is also connected to the bus 704.

[0118] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as required. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 710 as required so that a computer program read out therefrom is installed into the storage section 708 as required.

[0119] In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for performing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the method of the present application are executed. It should be noted that the computer-readable medium in the present application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or combined with an instruction execution system, apparatus, or device. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries the computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, and the computer-readable medium can send, propagate, or transmit a program for use by or combined with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0120] Computer program code for performing the operations of this application can be written in one or more programming languages or combinations thereof. The above-mentioned programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0121] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutively represented blocks can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0122] The terms "first", "second", etc. are used to distinguish similar objects, rather than to describe or represent a specific order or sequence.

[0123] The term "comprising" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or device / equipment that comprises a series of elements includes not only those elements but also other elements that are not explicitly listed, or also includes elements that are inherent to these process, method, article, or device / equipment.

[0124] So far, the technical solution of the present invention has been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A method for violent sorting and loading / unloading action recognition based on multi-modalities, characterized in that: The method includes: Obtaining first behavior data and video data of a person to be recognized for actions; the first behavior data is the motion data and heart rate data of the person; the actions to be recognized include violent sorting and violent unloading; the motion data includes: the swinging arm direction data, swinging arm angular velocity data, swinging arm linear velocity data, swinging arm frequency data, and swinging arm amplitude data of the sorting and / or loading / unloading personnel; Converting the first behavior data into second behavior data, and inputting the second behavior data into a trained data analysis algorithm model to obtain first motion state data; For the data analysis algorithm model, its training method is: Obtaining multiple pieces of the first behavior data at time t; t>1; Obtaining multiple pieces of the first behavior data at time t-1, and inputting them into a long short-term memory time series network to obtain prediction data corresponding to the first behavior data at time t; Fusing the first behavior data and its corresponding prediction data, and using the fused data as the second behavior data; Encoding the second behavior data into M vectors of size 1xN, and constructing an M*N matrix, where N is the sum of the number of the motion data and the number of the heart rate data; Encoding the constructed M*N matrix and inputting it into a logistic regression model for training, and using the trained logistic regression model as the data analysis algorithm model; Based on the video data, obtaining multiple tracking box data, multiple optical flow features, and text information data, performing multimodal fusion on the multiple tracking box data and the text information data to obtain fused data, and then fusing the fused data with the optical flow features to obtain second motion state data; obtaining the action recognition result of the person according to the first motion state data and the second motion state data; For the second motion state data, its obtaining method is: inputting the text information data and the multiple tracking box data into a clip multimodal algorithm model to obtain fused data; inputting the video data into an optical flow algorithm model to obtain optical flow features, and fusing the optical flow features and the fused data again at the head layer of the optical flow algorithm model to obtain second motion state data.

2. The multimodal-based violent sorting and loading / unloading action recognition method according to claim 1, characterized in that For the tracking box data, its obtaining method is: Obtaining the video data of a person to be recognized for sorting and / or loading / unloading actions as the first video stream data; Extracting the human body regions in each video frame of the first video stream data to construct second video stream data; Inputting the second video stream data into a target detection network to obtain first target box data, and inputting the first target box data into a linear model to predict second target box data, where the second target box data is the data of the next frame of the video frame where the first target box data is located; Performing data association on the first target box data and the second target box data through similarity calculation to obtain multiple tracking box data corresponding to each frame of the image.

3. The method for identifying violent sorting and loading / unloading actions based on multi-modalities according to claim 2, wherein The method for extracting the human body regions in each video frame of the first video stream data is: Input the first video stream data into a cascade sparse query algorithm based on high resolution to obtain the rough position data of the human body area in the first video stream data, and calculate the precise position area data of the human body area by means of sparse convolution as the second video stream data.

4. The method for identifying violent sorting and loading / unloading actions based on multi-modalities according to claim 2, wherein The method for obtaining the action recognition result of a person according to the first motion state data and the second motion state data is as follows: Judge whether the first motion state data and the second motion state data are consistent; If they are consistent, judge whether both the first motion state data and the second motion state data are violent sorting and / or violent loading and unloading; If not, do not output an alarm signal; If so, upload the first video stream data corresponding to the second motion state data to the first mobile terminal, and output the alarm signal to the second mobile terminal; If they are inconsistent, input the first motion state data and the second motion state data into a Gaussian mixture model according to the weighted average method and in combination with the historical first motion state data and the second motion state data to obtain the action recognition result; If it is not violent sorting and / or violent loading and unloading, do not output an alarm signal; If it is violent sorting and / or violent loading and unloading, upload the first video stream data in the second motion state data to the first mobile terminal, and output the alarm signal to the second mobile terminal.

5. The multi-modal based violent sorting and loading / unloading action recognition system according to any one of claims 1-4, characterized in that: The system includes a data acquisition module, a data model construction module, an action model construction module, and a judgment module; The data acquisition module is configured to acquire first behavior data, where the first behavior data is the motion data and heart rate data of the loading and unloading personnel; the motion data includes: the swing direction data, swing angular velocity data, swing linear velocity data, swing frequency data, and swing amplitude data of the sorting and / or loading and unloading personnel; The first training module is configured to convert the first behavior data into second behavior data, and input the second behavior data into a trained data analysis algorithm model to obtain the first motion state data; the training method of the data analysis algorithm model is as follows: Obtain multiple pieces of the first behavior data at time t; t>1; Obtain multiple pieces of the first behavior data at time t - 1, and input them into a long short-term memory time series network to obtain the predicted data corresponding to the first behavior data at time t; Fuse the first behavior data and its corresponding predicted data, and use the fused data as the second behavior data; Encode the second behavior data into M vectors of size 1xN, and construct an M*N matrix, where N is the sum of the number of the motion data and the number of the heart rate data; Encode the constructed M*N matrix and input it into a logistic regression model for training, and use the trained logistic regression model as the data analysis algorithm model; The second training module is configured to obtain multiple tracking box data, multiple optical flow features, and text information data based on the video data, perform multi-modal fusion on the multiple tracking box data and the text information data to obtain fusion data, and then fuse the fusion data with the optical flow features to obtain second motion state data; the method for obtaining the second motion state data is as follows: input the text information data and the multiple tracking box data into a clip multi-modal algorithm model to obtain fusion data; input the video data into an optical flow algorithm model to obtain optical flow features, and perform re-fusion of the optical flow features and the fusion data at the head layer of the optical flow algorithm model to obtain second motion state data; The judgment module is configured to obtain an action recognition result of a person according to the first motion state data and the second motion state data.

6. A storage device that stores multiple programs, characterized in that, The program is applicable to be loaded and executed by a processor to implement the multi-modal-based violent sorting and loading / unloading action recognition method according to any one of claims 1-4.

7. A processing device includes a processor and a storage device; the processor is adapted to execute each program; the storage device is adapted to store multiple programs; characterized in that, The program is applicable to be loaded and executed by a processor to implement the multi-modal-based violent sorting and loading / unloading action recognition method according to any one of claims 1-4.

Images