Appearance detection and action recognition device and method for densely packed objects in a narrow space

Abstract

The present invention provides an appearance detection action recognition device and method for dense objects in a small space. It first proposes that different computer vision algorithm models perform cyclic reading and processing on the original encoded video frame image data stored in the shared memory according to the time sequence, and use a multi-target tracking algorithm to correspond the ID of the dense object tracking to the appearance detection behavior recognition result of the object. The recognition result is regularly added to the data set for self-training of the model, which effectively reduces the coupling degree between systems and the data reading delay, and greatly improves the real-time, robustness and efficiency of video processing. It solves the problem that the existing methods for dense object appearance detection are low in efficiency and highly subjective.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and computer vision technology, and in particular relates to an appearance detection and action recognition device and method for densely packed objects in a narrow space. Background Art

[0002] In recent years, with the trend of industrial intelligent upgrading, more and more companies are trying to build smart factories through technologies such as robots and artificial intelligence. Factory surveillance cameras can generate several terabytes of effective factory video data every day, and most of these videos are only used to monitor workers' production. In fact, these factory video data contain a large number of workers and machine operation behaviors, as well as the production operation modes contained therein, which can be further used for motion recognition, workflow mining, abnormal event monitoring, etc. However, at present, the work quality assessment of production line workers or mechanical devices in certain production operation behaviors still relies on manual inspection or random sampling, which consumes a lot of manpower, material and financial resources, and does not effectively combine surveillance video with artificial intelligence technology.

[0003] It is known that there are few studies that attempt to automatically judge whether the appearance inspection of dense objects in a small space is compliant, and most methods still rely on manual monitoring. However, this manual inspection method is limited by manpower and subjective factors. When the factory capacity is expanded, the difficulty of judging whether the appearance inspection of dense objects is compliant increases significantly.

[0004] In the past, most of the technologies for recognizing human actions were to recognize the current actions of the human as a whole. For example, in Patent Document 1: Invention Disclosure CN115410268A Action Recognition Method and Action Recognition Method Based on Action Skeleton Point Data, only the action key point data was input into the model for recognition and the result was output. However, in the narrow space of the factory, multiple appearance detection actions need to be performed on multiple objects. The existing action recognition methods cannot directly recognize the actions of the monitoring video data and effectively bind the recognition results to the dense objects correctly. Summary of the invention

[0005] In order to overcome the problems existing in the above prior art, the purpose of the present invention is to provide an appearance detection action recognition device and method for dense objects in a small space. For the first time, different computer vision algorithm models are proposed to read and process the original encoded video frame image data stored in the shared memory in a cyclic manner according to the time sequence, and the multi-target tracking algorithm is used to correspond the ID of the dense object tracking to the appearance detection behavior recognition result of the object. The recognition result is regularly added to the data set for self-training of the model, which effectively reduces the coupling degree between systems and the data reading delay, and greatly improves the real-time, robustness and efficiency of video processing. The problem of low efficiency and strong subjectivity of the existing method for dense object appearance detection is solved.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] A method for appearance detection and action recognition of dense objects in a small space, comprising the following steps:

[0008] S1: The camera sends the original encoded video to the edge device and the background display system through multicast, and writes the frame image data to the shared memory in real time; if the write position reaches the end of the shared memory queue, it will start writing again from the head of the queue to overwrite the original frame image data, that is, the shared memory is written in a loop;

[0009] S2: cyclically reading from the shared memory according to the frame sequence information generated and transmitted by the camera, obtaining corresponding frame image data as input, wherein the frame sequence information specifies the specific storage location of the frame image data in the shared memory; performing target detection on the frame image data taken out from the shared memory using the damo-yolo target detection model, and outputting detection results such as detection frame coordinates and object categories to the target tracking model;

[0010] S3: Read frame image data from the shared memory according to the frame sequence information transmitted by the damo-yolo target detection model, combine the detection results, fuse the real-time position and texture changes of the objects to perform multi-target tracking processing and assign object IDs, and output the tracking results of dense object tracking coordinates and IDs;

[0011] S4: Use the action segmentation model to perform appearance detection action behavior segmentation, divide the cached long video obtained in the shared memory into isolated action segments, mark the start frame number start and the end frame number end of the segment, and forward it to the action recognition model through a message;

[0012] S5: The action recognition model obtains the isolated action segment image data from the shared memory according to the start frame number start and the end frame number end obtained by message forwarding, uses the spatiotemporal area attention model and the large convolution kernel to extract the spatiotemporal features of the video sequence, and then completes the action classification, outputs the recognition result and forwards the result by message;

[0013] S6: The backend display system receives the detection, tracking, behavior segmentation, and action recognition results of the front-end edge device through socket communication, and overlays the results on the real-time video.

[0014] S7: The recognition results are automatically cached to the storage device and fused with the original training data to generate a new data set. The new data set is triggered by a scheduled task to train the action recognition model using the incremental data set and automatically replace the latest action recognition model file.

[0015] Furthermore, in S1, the original coded video is I={I s} T , I s represents the sth frame, s represents the index value of each frame, and T represents the timestamp of the sth frame; the camera obtains the sth real-time image I with a size of M s And write the real-time image into a shared memory of N size applied in advance, wherein the shared memory can write n images in total, that is, N=nM.

[0016] Furthermore, in S2, the cyclic reading in the shared memory means that after the camera writes the shared memory, the target detection model reads the corresponding image data from the location of the shared memory according to the frame sequence information of the frame image data transmitted by the queue.

[0017] Further, in S2, the object detection result of the frame image data is calculated as shown in the following formula:

[0018] D t =F(I t );

[0019] Where D t Indicates the current frame I t The object detection result is specifically expressed as [c, w, h, x, y, thresh], where c represents the object category, w represents the detection box width, h represents the detection box height, x represents the x-axis coordinate of the center point of the detection box, y represents the y-axis coordinate of the center point of the detection box, and thresh represents the confidence of the detection box; t represents the index value of the image data of each frame in the shared memory; function F represents the current frame I t The operation process of the damo-yolo target detection model.

[0020] Furthermore, in S3, the calculation process of the real-time position and texture change of the fused object for multi-target tracking is as shown in the following formula:

[0021] X t =G(I t ,D t )

[0022] In the formula, X t Indicates the current frame I t The object tracking result is specifically expressed as [id, c, w, h, x, y, thresh], where id represents the object id assigned by the tracking algorithm, c represents the object category, w represents the width of the tracking detection box, h represents the height of the tracking detection box, x represents the x-axis coordinate of the center point of the tracking detection box, y represents the y-axis coordinate of the center point of the tracking detection box, and thresh represents the confidence of the tracking detection box; t represents the index value of the image data of each frame in the shared memory; function G represents the current frame I t And the detection result D of the frame t The operation process of the target tracking algorithm.

[0023] Furthermore, in S4, the action segmentation model performs a calculation process of appearance detection action behavior segmentation, as shown in the following formula:

[0024] {L 1 ,…,L n}=trim(V m )

[0025] (I start ,I end )=P({L 1 ,…,L n})

[0026] Where V m Indicates the cached long video data obtained from the shared memory; the trim function indicates the current cached video V m After the operation process of the action segmentation algorithm; L i Represents the action category output by each frame of the cached long video, i represents the number of categories set by the system; function P represents the action category L according to the frame data i Get the isolated action start frame I start and end frame I end The operation process.

[0027] Furthermore, in S5, the spatiotemporal feature extraction of the video sequence is performed using the spatiotemporal region attention model and the large convolution kernel, thereby completing the calculation process of action classification, as shown in the following formula:

[0028] Y=W(I start ,…,I end )

[0029] In the formula, Y represents the action recognition prediction result, I start ,…,I endIt indicates that the input image is all the picture data of the start point and end point defined by the action segmentation algorithm, which is obtained from the shared memory; the function W represents the operation process of the current input frame sequence through the action recognition algorithm.

[0030] Furthermore, in S6, the background display system receives the detection, tracking, behavior segmentation and action recognition results of the front-end edge device through socket communication, and at the same time obtains the image from the shared memory, aligns the image timestamp with the detection information timestamp, and overlays the results on the real-time video for display.

[0031] Furthermore, in S7, the system automatically caches the recognition results in the storage device, and merges them with the original training data to generate a new data set, which is triggered by a scheduled task, uses the incremental data set to train the motion recognition model, and automatically replaces the latest motion recognition model file.

[0032] Another object of the present invention is to provide a device, comprising a processor, a communication interface, a memory, a display and a communication bus, wherein the processor, the communication interface, the memory and the display communicate with each other via the communication bus;

[0033] Memory, used to store computer programs;

[0034] A display, used to display the execution results and real-time images of the processor;

[0035] The processor is used to implement the above method steps when executing the program stored in the memory.

[0036] Beneficial effects of the present invention:

[0037] (1) The video input frame access method of the present invention stores the video image in the shared memory, and the camera sampling and each algorithm model access the shared memory in time sequence. The camera writes to the shared memory in a loop, and each algorithm model reads the frame image data from the shared memory in an orderly manner and performs corresponding processing. Only a small amount of frame image processing results, such as the storage location of the frame in the shared memory, timestamp, etc., are transmitted between each model. The delay problem caused by the transmission and reading of large data such as images and other large data in each model is effectively reduced.

[0038] (2) The circular reading in the shared memory of the present invention means that after the camera writes to the shared memory, each algorithm model reads the picture according to the storage location of the picture frame in the shared memory. If the reading location reaches the end of the shared memory queue, it starts reading again from the head of the queue. The target detection model reads the picture from the shared memory for processing, and transmits a small amount of information such as the storage location of the frame in the shared memory and the detection results to the target tracking model through a queue. The target tracking model reads the picture from the shared memory according to the storage location of the frame in the shared memory, and processes it in combination with the detection results, and transmits a small amount of information such as the storage location of the frame in the shared memory and the tracking results through a queue. The action segmentation model obtains the cached long video data from the shared memory for processing, and transmits the isolated action start frame I through a queue. start and end frame I end The action recognition model reads the image sequence from the shared memory as the model input according to the transmitted isolated action start frame number start and end frame number end.

[0039] (3) The present invention introduces a target tracking model, which matches the ID of the object target tracking with the appearance detection behavior recognition result of the object, effectively solving the problem that it is difficult to match multiple results after appearance detection of dense objects in a small space, and achieves dynamic binding of dense objects and detection results.

[0040] (4) The present invention updates the data set by caching recognition data, uses scheduled tasks to iteratively train the model based on the new data set, and updates the action recognition model file in the system, effectively improving the accuracy and robustness of the model.

[0041] (5) Compared with the prior art, the present invention uses the existing 2D camera to sample the appearance inspection behavior of densely packed objects in a small space without adding any additional hardware equipment, and then uses the computer vision algorithm to automatically determine whether the current inspection behavior is compliant, thereby effectively reducing manpower and material resources and greatly increasing work efficiency and the objectivity of inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is the appearance detection and action recognition system framework for densely packed objects in a narrow space according to an embodiment of the present invention.

[0043] Figure 2 It is a schematic diagram of system data interaction in an embodiment of the present invention.

[0044] Figure 3 It is a schematic diagram of the system framework deployment constructed by an embodiment of the present invention.

[0045] Figure 4 It is a logic processing flow chart of object appearance detection according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0047] like Figures 1 to 3 As shown, an embodiment of the present invention provides an appearance detection and action recognition device and method for densely packed objects in a small space, wherein a camera, a target detection model, a target tracking model, a motion segmentation model, a motion recognition model, and an interface display are serially arranged and executed in chronological order, and shared memory data are accessed in parallel.

[0048] The specific steps include:

[0049] Step 1: The camera sends the original encoded video to the edge device and the background display system through multicast, and writes the frame image data to the shared memory in real time, so that the subsequent algorithm model can obtain the corresponding image data in order according to the frame sequence information. The camera only transmits the location of the image in the shared memory and the image timestamp to the target detection model through the queue, thereby reducing the delay time caused by the transmission of image data between models. The edge device is the server required for model operation and training, and the background display system includes a visualization interface and a display.

[0050] Step 2: According to the frame sequence information generated and transmitted by the camera, the corresponding frame image data is read from the shared memory in a loop, and the damo-yolo target detection model is used to perform target detection on the frame image data taken out from the shared memory, and the detection results such as the detection box coordinates and object category are output to the target tracking model.

[0051] Step 21: convert the original coded video data I = {I t} T Input to the pre-processing of the damo-yolo target detection model, I t represents the tth frame, t represents the index value of each frame, and T represents the timestamp. The input static image is scaled, cropped, denoised, etc. to meet the input requirements of the model.

[0052] In step 22, each cropped small piece of the image is input into a convolutional neural network for feature extraction, and each feature map is connected to form a feature map of the entire image.

[0053] Step 23, apply a fully connected layer to the large feature map to output a 13x13x(5+K) tensor, where K is the number of categories and 5 is the bounding box parameters (center coordinates, width, height, and confidence). For each 13x13 cell, a predicted bounding box is generated based on the bounding box parameters and category confidence. The predicted bounding box is threshold filtered and non-maximum suppressed to obtain the final detection result.

[0054] Step 3: Read frame image data from shared memory in a loop according to the frame sequence information transmitted by the damo-yolo target detection model, combine the detection results, integrate the real-time position and texture changes of the object to perform multi-target tracking processing and assign object IDs, and output tracking results such as dense object tracking coordinates and IDs.

[0055] Step 31, obtaining the original coded video I={I t} T Input tracking model processing.

[0056] Step 32, based on the detection frame and other information transmitted by the damo-yolo target detection model, use the feature extraction network to extract features of the image within each frame to obtain the feature vector of each target.

[0057] Step 33, use the Kalman filter to predict the state (position and speed) of each target, using an 8-dimensional state space vector represented by x = [cx, cy, w, h, vx, vy, vr, vh], and initialize each speed value to 0. As a direct observation model of the object state, (cx, cy) represents the center point coordinates of the detection box, the width w, the height h, and the relative speed corresponding to each image coordinate. The prediction is based on the state of the track at time t-1 to predict its state at time t, and the calculation formula is as follows:

[0058] x′=Fx (1)

[0059] P′=FPF T +Q (2)

[0060] In formula 1, x is the mean value of track at time t-1, F is called the state transfer matrix, and the formula predicts x' at time t:

[0061]

[0062] The dt in matrix F is the difference between the current frame and the previous frame.

[0063] In Formula 2, P is the covariance of track at time t-1, and Q is the noise matrix of the system, which represents the reliability of the entire system and is generally initialized to a very small value. This formula predicts P' at time t.

[0064] Step 34, use the Hungarian Algorithm to match the predicted state and the detected bounding box, and update the state of the tracker according to the matching result.

[0065] Step 35: For unmatched bounding boxes, use the cosine distance to calculate the similarity between them and the feature vectors of existing trackers. If the similarity is above a certain threshold, assign it to the corresponding tracker, otherwise create a new tracker. For unmatched trackers, if they have not been matched to a bounding box for multiple consecutive frames, delete them.

[0066] Step 36: Use Kalman filtering to update the position. The update is to correct the state of the track associated with the detection detected at time t to obtain a more accurate result. The calculation formula is as follows:

[0067] y=z-Hx′ (3)

[0068] S=HP′H T +R (4)

[0069] K=P′H T S -1 (5)

[0070] x=x′+Ky (6)

[0071] P=(I-KH)P′ (7)

[0072] In formula 3, z is the mean vector of detection, excluding the speed change value, that is, z = [cx, cy, w, h], H is called the measurement matrix, which maps the mean vector x' of track to the detection space. This formula calculates the mean error of detection and track;

[0073] In formula 4, R is the noise matrix of the detector, which is a 4x4 diagonal matrix. The values ​​on the diagonal are the two coordinates of the center point and the noise of width and height. It is initialized with arbitrary values. Generally, the noise of width and height is set greater than the noise of the center point. This formula first maps the covariance matrix P' to the detection space, and then adds the noise matrix R;

[0074] Formula 5 calculates the Kalman gain K, which is used to estimate the importance of the error;

[0075] Formula 6 and Formula 7 obtain the updated mean vector x and covariance matrix P.

[0076] Step 4: Use the action segmentation model to perform appearance detection action behavior segmentation, divide the cached long video into isolated action segments, mark the start frame number start and the end frame number end of the segment, and forward it to the action recognition model through a message.

[0077] Step 41, get the cached long video from the shared memory, and use a prediction generation stage to generate the initial action segmentation prediction. This stage uses a dual dilated layer (DDL) to capture both large-scale and small-scale temporal information;

[0078] Step 42, using multiple refinement stages to gradually improve the action segmentation prediction. Each refinement stage uses a self-supervised temporal convolutional network (SS-TCN), which can use the information of previous and next frames to correct the prediction of the current frame;

[0079] Step 43, average the outputs of all stages to obtain the final action segmentation result, and assign an action category label to each frame of the image. Perform action segmentation based on the action category label, and mark the start frame number start and the end frame number end of the isolated action segment.

[0080] Step 44, by forwarding a message, the start frame sequence number start and the end frame sequence number end are transmitted to the action recognition model.

[0081] Step 5: The action recognition model obtains the isolated action fragment image data from the shared memory according to the start frame number start and end frame number end obtained by message forwarding, uses the spatiotemporal area attention model and large convolution kernel to extract the spatiotemporal features of the video sequence, and then completes the action classification, outputs the recognition result and forwards the result by message.

[0082] Step 51, based on the obtained start frame number start and end frame number end, all image data of the action clip are obtained from the shared memory, the image is averagely sampled according to the NUM_FRAMES set by the model, and the image is randomly scaled and cropped after sampling.

[0083] Step 52: for each segment, sample different numbers of frames (e.g., 8 frames and 32 frames) from the slow path and the fast path, respectively. For each path, use a 3D convolutional neural network (e.g., ResNet-50) to extract features from the sampled frames to obtain a feature map.

[0084] Step 53, use the spatial attention mechanism to select the salient regions in the feature map. It calculates the attention score for the input feature map and uses the score as the weight coefficient for the values ​​of different channels at the same pixel position in the input feature map to generate an output feature map that only contains salient regions. The spatial attention score calculation formula is as follows:

[0085] M s (F) = f(conv k (concatenation(MaxPooling(F),AveragePooling(F))))

[0086] Where F is the input feature map, with a size of (B, C, T, H, W), where B refers to the batch size, C refers to the number of channels, T refers to the time dimension, H refers to the height, and W refers to the width. In the above formula, MaxPooling(·) and AveragePooling(·) refer to the use of maximum pooling and average pooling on F in the channel dimension, respectively, to obtain two tensors of size (B, 1, T, H, W), which are then concatenated (·) to obtain a tensor of size (B, 2, T, H, W). The conv in the above formula is k (·) refers to the width and height of the convolution kernel, which is specifically implemented as a k*k three-dimensional convolution operation. k is a hyperparameter that can be adjusted according to actual conditions. In the above formula, f(·) can be any activation function. In this invention, the sigmoid activation function is selected. The spatial weight M is thus obtained. s (F) M s The value of each pixel in (F) represents the importance of the corresponding position in the input feature F. Applying this module in the model can suppress irrelevant background information.

[0087] Step 54, using the spatial weight M s (F) Perform mask processing on the input feature F. The specific formula is as follows:

[0088]

[0089] After masking, the number of channels of the input features will not change.

[0090] Step 55, at a specific level, bidirectional channel-wise connections are used to fuse the feature maps of the slow path and the fast path. For each path, global average pooling is used to compress the features into a one-dimensional vector. The feature vectors of the two paths are concatenated to obtain the final video representation.

[0091] Step 56, use the fully connected layer (Fully Connected Layer) and the Softmax layer to classify the video representation to obtain the result of action recognition.

[0092] like Figure 2 As shown in the figure, the data interaction diagram explains in detail the data transmission method between each model and the front-end and back-end data communication method.

[0093] In step 6, the background display system receives the detection, tracking, behavior segmentation, and action recognition results of the front-end edge device through socket communication, and overlays the results on the real-time video.

[0094] Step 61, the camera pushes video data to the background display system in a multicast manner, and the background calls the camera driver to obtain frame image data and writes it into the shared memory.

[0095] Step 62, the detection, tracking, action recognition and other results share a socket connection, and the transmission data is distinguished by the result type field and related processing is performed.

[0096] Step 63, obtain frame image data from the shared memory according to the frame sequence information of the tracking result, and obtain the detection frame coordinates, tracking ID, log timestamp and other information of all objects in the frame. According to the action recognition result, obtain the recognition result of the object detection action and the corresponding tracking ID information.

[0097] Step 64, binding the detection result and the object position coordinates according to the tracking ID; aligning the tracking result timestamp with the image timestamp, and after alignment, performing frame processing according to the detection frame information, and displaying the processed image on the system interface.

[0098] In step 7, the system automatically caches the recognition results to the storage device and merges them with the original training data to generate a new data set. The new data set is triggered by a scheduled task to train the motion recognition model using the incremental data set and automatically replace the latest motion recognition model file.

[0099] like Figure 4As shown, an embodiment of the present invention provides a flow chart of the logic processing of object appearance detection, which introduces the processing logic of the system when the detection result is OK or NG, wherein the result OK means that the object appearance detection meets the appearance detection standard, and the result NG means that the object appearance detection behavior is unqualified. At this time, the operator or the robot arm needs to perform the appearance inspection of the object again, that is, the object re-inspection. The specific process is as follows:

[0100] The camera obtains real-time surveillance video and transmits it to the target detection model of the front-end edge device for target detection processing and output of detection results, which are then output to the target tracking model. The target tracking model makes a judgment based on the current trajectory status: if it is a re-inspection of an NG object, the trajectory status maintained by the target tracking model is not stable and continuous, but jumps. The system resets the trajectory and preferentially finds the missing ID from the currently maintained discontinuous trajectory for redistribution. After completion, the ID is stable and continuously increasing; if it is a new object detection, the trajectory status maintained by the current system is stable and increasing without jumps, then a new trajectory is generated incrementally and a new ID is assigned. The action segmentation model divides the continuous long video into isolated action segments, which are transmitted to the action recognition model for appearance detection compliance judgment, and the recognition results are transmitted to the background display system. The background receives the detection, tracking, behavior segmentation and action recognition results of the front-end edge device, and displays the results on the real-time video. The operator or mechanical equipment re-inspects the object with the NG result based on the result, and repeats the above operation process until the recognition result is OK.

[0101] The present invention analyzes the time consumption of data access between models and proposes for the first time to store video images in shared memory. Camera sampling and each algorithm model read frame image data from shared memory in chronological order and perform corresponding processing. Only frame sequence information is transmitted between models, including a small amount of data such as the position of the frame in shared memory and timestamp. This method effectively solves the delay problem caused by the transmission and reading of large data such as images in each model.

[0102] In the detection result matching stage of dense objects in a small space, the present invention innovatively introduces a target tracking model and proposes binding the target tracking ID with the appearance detection behavior recognition result of the object, which effectively solves the problem that multiple results of dense objects in a small space are difficult to match after appearance detection.

[0103] The present invention updates the data set by caching recognition data, uses scheduled tasks to iteratively train the model based on the new data set, and updates the action recognition model file in the system, effectively improving the accuracy and robustness of the model.

[0104] Compared with the existing technology, the present invention uses the existing 2D camera to sample the appearance detection behavior of dense objects without adding additional hardware equipment, and uses computer vision algorithms to automatically determine whether the current detection behavior is compliant, which effectively reduces manpower and material resources and greatly increases work efficiency and objectivity of inspection.

[0105] It should be understood that the above-mentioned embodiments are only illustrative. The embodiments described in the present invention may be implemented in hardware, software, firmware, middleware, microcode or any combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors and / or other electronic units designed to perform the functions described in the present invention or in combination thereof.

[0106] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0107] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for appearance detection and action recognition of densely packed objects in a small space, characterized in that: The following steps are involved: S1: The camera sends the original encoded video to the edge device and the background display system through multicast, and writes the frame image data to the shared memory in real time; if the write position reaches the end of the shared memory queue, it will start writing again from the head of the queue to overwrite the original frame image data, that is, the shared memory is written in a loop; S2: cyclically reading from the shared memory according to the frame sequence information generated and transmitted by the camera, obtaining corresponding frame image data as input, wherein the frame sequence information specifies the specific storage location of the frame image data in the shared memory; performing target detection on the frame image data taken out from the shared memory using the damo-yolo target detection model, and outputting the detection frame coordinates and the object category detection results to the target tracking model; S3: Read frame image data from the shared memory according to the frame sequence information transmitted by the damo-yolo target detection model, combine the detection results, fuse the real-time position and texture changes of the objects to perform multi-target tracking processing and assign object IDs, and output the tracking results of dense object tracking coordinates and IDs; S4: Use the action segmentation model to perform appearance detection action behavior segmentation, divide the cached long video obtained in the shared memory into isolated action segments, mark the start frame number start and the end frame number end of the segment, and forward it to the action recognition model through a message; S5: The action recognition model obtains the isolated action segment image data from the shared memory according to the start frame number start and the end frame number end obtained by message forwarding, uses the spatiotemporal area attention model and the large convolution kernel to extract the spatiotemporal features of the video sequence, and then completes the action classification, outputs the recognition result and forwards the result by message; S6: The backend display system receives the detection, tracking, behavior segmentation, and action recognition results of the front-end edge device through socket communication, and overlays the results on the real-time video. S7: The recognition results are automatically cached to the storage device and fused with the original training data to generate a new data set. The new data set is triggered by a scheduled task to train the action recognition model using the incremental data set and automatically replace the latest action recognition model file.

2. The appearance detection action recognition method for densely packed objects in a narrow space according to claim 1, characterized in that: In S1, the original coded video is I={I s } T , I s represents the sth frame, s represents the index value of each frame, and T represents the timestamp of the sth frame; the camera obtains the sth real-time image I with a size of M s And write the real-time image into a shared memory of N size applied in advance, wherein the shared memory can write n images in total, that is, N=nM.

3. The appearance detection action recognition method for densely packed objects in a narrow space according to claim 1, characterized in that: In S2, the cyclic reading in the shared memory means that after the camera writes into the shared memory, the target detection model reads the corresponding image data from the location of the shared memory according to the frame sequence information of the frame image data transmitted by the queue.

4. The appearance detection action recognition method for densely packed objects in a narrow space according to claim 1, characterized in that: In S2, the object detection result of the frame image data is calculated as shown in the following formula: D t =F(I t ); Where D t Indicates the current frame I t The object detection result is specifically expressed as [c, w, h, x, y, thresh], where c represents the object category, w represents the detection box width, h represents the detection box height, x represents the x-axis coordinate of the center point of the detection box, y represents the y-axis coordinate of the center point of the detection box, and thresh represents the confidence of the detection box; t represents the index value of the image data of each frame in the shared memory; function F represents the current frame I t The operation process of the damo-yolo target detection model.

5. The appearance detection action recognition method for densely packed objects in a narrow space according to claim 1, characterized in that: In S3, the calculation process of the real-time position and texture change of the fused object for multi-target tracking is shown in the following formula: X t =G(I t ,D t ) Where, X t Indicates the current frame I t The object tracking result is specifically expressed as [id, c, w, h, x, y, thresh], where id represents the object id assigned by the tracking algorithm, c represents the object category, w represents the width of the tracking detection box, h represents the height of the tracking detection box, x represents the x-axis coordinate of the center point of the tracking detection box, y represents the y-axis coordinate of the center point of the tracking detection box, and thresh represents the confidence of the tracking detection box; t represents the index value of the image data of each frame in the shared memory; function G represents the current frame I t And the detection result D of the frame t The operation process of the target tracking algorithm.

6. The appearance detection action recognition method for densely packed objects in a narrow space according to claim 1, characterized in that: In S4, the action segmentation model performs a calculation process of appearance detection action behavior segmentation, as shown in the following formula: {L1,…,L n }=trim(V m ) (YO start ,YO end )=P({L1,…,L n }) Where V m Indicates the cached long video data obtained from the shared memory; the trim function indicates the current cached video V m After the operation process of the action segmentation algorithm; L i Represents the action category output by each frame of the cached long video, i represents the number of categories set by the system; function P represents the action category L according to the frame data i Get the isolated action start frame I start and end frame I end The operation process.

7. The method for appearance detection and action recognition of densely packed objects in a narrow space according to claim 1, characterized in that: In S5, the spatiotemporal feature extraction of the video sequence using the spatiotemporal region attention model and the large convolution kernel is performed to complete the calculation process of action classification, as shown in the following formula: Y=W(I start ,...,I end ) In the formula, Y represents the action recognition prediction result, I start , ..., I end It indicates that the input image is all the picture data of the start point and end point defined by the action segmentation algorithm, which is obtained from the shared memory; the function W represents the operation process of the current input frame sequence through the action recognition algorithm.

8. The method for appearance detection and action recognition of densely packed objects in a narrow space according to claim 1, characterized in that: In S6, the background display system receives the detection, tracking, behavior segmentation and action recognition results of the front-end edge device through socket communication, and at the same time obtains the image from the shared memory, aligns the image timestamp with the detection information timestamp, and overlays the results on the real-time video for display.

9. The method for appearance detection and action recognition of densely packed objects in a narrow space according to claim 1, characterized in that: In S7, the system automatically caches the recognition results to the storage device, and merges them with the original training data to generate a new data set, which is triggered by a scheduled task, uses the incremental data set to train the action recognition model, and automatically replaces the latest action recognition model file.

10. A device is provided based on the method according to any one of claims 1 to 9, characterized in that: It includes a processor, a communication interface, a memory, a display and a communication bus, wherein the processor, the communication interface, the memory and the display communicate with each other through the communication bus; Memory, used to store computer programs; A display, used to display the execution results and real-time images of the processor; The processor is used to implement the method steps described in any one of claims 1 to 9 when executing the program stored in the memory.

Images