A BERT behavior detection method and system based on binocular vision

Abstract

The application relates to a BERT behavior detection method and system based on binocular vision, which comprises the following steps: acquiring multi-angle video data of a to-be-detected object collected by multiple binocular stereo cameras, and recording a reference coordinate system of each camera in the binocular stereo camera; converting the video data into binocular stereo image data, and sequentially performing instance segmentation and feature extraction matching on the image data to form point cloud data, and converting the point cloud data to the reference coordinate system; performing feature optimization based on the converted point cloud data and a BERT network, training an LSTM network based on the optimized features, and finally obtaining a behavior detection model; and extracting individual behavior and group behavior from the point cloud data according to the behavior detection model. Compared with the prior art, the application has the advantages of fewer samples, strong universality, high detection accuracy and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to a BERT behavior detection method and system based on binocular vision. BACKGROUND

[0002] Behavior analysis detection is an important research direction in the field of computer vision, which aims to analyze and detect specific actions in real-time video streams or historical video data, providing support and solutions for video surveillance, social security and industrial safety production, etc. With the continuous progress of technology, behavior analysis detection plays an increasingly important role in practical applications.

[0003] However, there are some challenges in most current behavior detection algorithm systems. First, these algorithms usually require a large amount of training data set to learn behavior characteristics and patterns, but it is costly to obtain high-quality data sets, especially for some specific behavior types or complex scenarios. Second, the universality of behavior detection algorithms is low, and needs to be fine-tuned for different application scenarios. This is because different application scenarios may involve different behavior types, environmental conditions and monitoring devices, and the algorithm needs to be adjusted and optimized according to the specific situation. SUMMARY

[0004] The purpose of the present application is to provide a BERT behavior detection method and system based on binocular vision to improve detection accuracy.

[0005] The purpose of the present application can be achieved by the following technical solutions:

[0006] A BERT behavior detection method based on binocular vision, comprising the following steps:

[0007] Obtain multiple binocular stereo cameras to collect multi-angle video data of the object to be detected, and record the reference coordinate system of each camera in the binocular stereo camera;

[0008] Convert the video data into binocular stereo image data, and sequentially perform instance segmentation and feature extraction matching on the image data to form point cloud data, and convert the point cloud data to the reference coordinate system;

[0009] Optimize the features based on the converted point cloud data and BERT network, and train the LSTM network based on the optimized features to finally obtain a behavior detection model;

[0010] According to the behavior detection model, individual behavior and group behavior are extracted from the point cloud data.

[0011] Further, the process of forming point cloud data specifically includes:

[0012] transforming the video data into binocular stereo image data;

[0013] instance segmentation is performed on left and right image data in the binocular stereo image data, and an instance identification id is assigned;

[0014] the same id in the left and right images is regarded as the same object, and feature extraction and matching are performed on the same object in the left and right images to form three-dimensional point cloud data.

[0015] Further, YOLOv8 segment is used for instance segmentation.

[0016] Further, the specific steps of the feature optimization include:

[0017] mapping the point cloud data to a low-dimensional feature vector;

[0018] performing feature fusion on the low-dimensional feature vector to construct a high-level feature vector;

[0019] inputting the high-level feature vector into a BERT network to output a BERT feature vector;

[0020] storing the BERT feature vector in a pre-constructed historical feature library, and extracting a plurality of BERT feature vectors from the historical feature library in chronological order to construct a time series feature vector.

[0021] Further, the point cloud data is sequentially mapped to a low-dimensional feature vector through convolution and fully connected network.

[0022] Further, the high-level feature vector is aligned before being input into the BERT network, and the corresponding mask mask is recorded.

[0023] Further, the specific steps of obtaining the behavior detection model include:

[0024] inputting the time series feature vector into an LSTM network to obtain a network output result;

[0025] based on the network output result, a loss value is calculated using a loss function;

[0026] based on the loss value, the model parameters of the LSTM network are updated using a backpropagation algorithm;

[0027] repeat the above steps to iteratively train the LSTM network until the loss value is less than a preset threshold or the number of iterations reaches a preset upper limit, and finally obtain a behavior detection model.

[0028] Further, in the process of updating the model parameters of the LSTM network, the valid output bits are selected according to the mask, and then the loss value is calculated and the model parameters are updated.

[0029] Further, the specific steps of extracting individual behavior and group behavior include:

[0030] Quantize the behavior detection model to obtain a behavior detection quantization model;

[0031] Input the point cloud data into the behavior detection quantization model, and extract valid feature parameters related to individual behavior and group behavior based on the mask;

[0032] Based on the valid feature parameters, individual behavior and group behavior are extracted respectively.

[0033] The application also provides a detection system based on the above-mentioned BERT behavior detection method based on binocular vision, characterized by comprising:

[0034] A data acquisition module is configured to acquire multi-angle video data of a to-be-detected object collected by a plurality of binocular stereo cameras, and record the reference coordinate system of each camera in the binocular stereo camera;

[0035] A data preprocessing module is configured to convert the video data into binocular stereo image data, and sequentially perform instance segmentation and feature extraction matching on the image data to form point cloud data, and convert the point cloud data to the reference coordinate system;

[0036] A data training module is configured to perform feature optimization based on the converted point cloud data and the BERT network, and train the LSTM network based on the optimized features, and finally obtain a behavior detection model;

[0037] A model inference module is configured to extract individual behavior and group behavior from the point cloud data based on the behavior detection model.

[0038] Compared with the prior art, the application has the following beneficial effects:

[0039] (1) In the present application, the reference coordinate system is calibrated during data acquisition and algorithm deployment, the point cloud data is transformed, the difference between the point cloud data obtained in different scenes is reduced, the generality of the algorithm system is improved, a plurality of point cloud data containing key information are extracted through instance segmentation and feature extraction matching, the prior knowledge of the BERT network model is used to optimize the features of individual behavior, individual behavior and group behavior, and then the trained behavior detection model is used for inference analysis, thereby improving the detection accuracy.

[0040] (2) The application can extract effective features from a small amount of samples by converting, instance segmentation, instance matching and feature optimization of the collected video data, so that the application can realize behavior detection function based on a small amount of sample data, without a large amount of sample data.

[0041] (3) The application analyzes high-level feature vectors through the BERT network to obtain BERT feature vectors and reduce the requirement for the number of data sets when training the model.

[0042] (4) The application quantizes the behavior detection model, and the quantization technology can reduce the size of the model and speed up the inference process of the model, thereby improving the efficiency of behavior detection.

[0043] (5) The application is easy to deploy, has high accuracy, can adapt to different application scenarios, has strong versatility, and is easy to migrate learning behavior information outside the training set. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is a flowchart of the detection method of the application;

[0045] Figure 2 is a structural schematic diagram of the detection system of the application;

[0046] Figure 3 is a processing process of the model training module of the application;

[0047] Figure 4 is a processing process of the model inference module of the application. DETAILED DESCRIPTION

[0048] The application will be described in detail below in combination with the drawings and specific embodiments. The embodiments are implemented on the premise of the technical solution of the application, and detailed implementation modes and specific operation processes are given, but the protection scope of the application is not limited to the following embodiments.

[0049] The application provides a BERT behavior detection method based on binocular vision, as shown in Figure 1 The method comprises the following steps:

[0050] Step 1, acquiring multiple binocular stereo cameras to collect multi-angle video data of a to-be-detected object, and recording the reference coordinate system of each camera in the binocular stereo camera.

[0051] This step is a data acquisition process, specifically: in this embodiment, six binocular stereo cameras are used to shoot video data of specific actions from different positions (angles), and the video data is decoded at equal intervals, and a reference coordinate system is established for each camera (the coordinate system xy plane is the ground plane, and the z axis is perpendicular to the ground plane), and the reference coordinate system corresponding to each camera is recorded.

[0052] Step 2, convert the video data into binocular stereo image data, and sequentially perform instance segmentation and feature extraction matching on the image data to form point cloud data, and convert the point cloud data to the reference coordinate system.

[0053] This step is a data preprocessing process, in which the video captured by the binocular stereo camera is converted into a photo, and the photo captured by the left lens is called the left photo, and the photo captured by the right lens is called the right photo. YOLOv8 segment is used for instance segmentation on the left and right photos to detect the detected object. Based on the maximum and minimum values of the pixel longitudinal coordinates of the detected object, the lower boundary and the upper boundary of the detected object are solved; based on the maximum and minimum values of the pixel transverse coordinates of the detected object, the right boundary and the left boundary of the detected object are solved, so as to determine the size of the boundary box of the detected object. The upper and lower boundary coordinates and the left and right boundary coordinates of the detected object are averaged respectively to determine the center coordinates of the detected object. And according to the boundary box size of the detected object, the center position coordinates of the boundary box in the left and right photos are used to assign an identification id to the instance. The objects with the same identification id in the left and right photos are regarded as the same object, and the feature extraction matching of the same object is performed to form three-dimensional point cloud data, and finally the point cloud data is converted into point cloud data in the reference coordinate system.

[0054] Step 3, based on the converted point cloud data and the BERT network, the features are optimized, and the LSTM network is trained based on the optimized features, and finally the behavior detection model is obtained.

[0055] This step is a model training process, specifically: in the model training process, mainly based on the above-mentioned three-dimensional point cloud data converted to the reference coordinate system, the feature optimization is performed for model training. As shown in Figure 3 each detected object in the same picture is used to construct a feature vector; each feature vector is mapped to a corresponding low-bit feature vector through convolution, full connection network, etc.; these low-bit feature vectors are fused to construct a high-level feature vector F1, and the high-level feature vector F1 is used as the input of the BERT network to obtain the BERT feature vector F2 output by the BERT network; the BERT feature vector F2 is stored in the historical feature library, and N BERT feature vectors F2 are extracted from the historical feature library in time sequence (the time sequence length is N in this paper), a time sequence feature vector is constructed, and is fed into the LSTM network, and the loss value is calculated using the loss function according to the output result, and the loss value is used to update the model weight parameters of the LSTM network using the back propagation algorithm; repeat this process until the loss value is less than a specified value, and the iteration reaches the upper limit, and finally the behavior detection model is obtained. After the behavior detection model is constructed, the trained model can be directly used for behavior detection.

[0056] Before the F1 high-level feature vector is fed into the BERT network in the above, vector alignment is performed, and the corresponding mask mask is recorded; when the model weight parameters are updated, the valid output bits are selected according to the mask mask, and then the loss value is calculated and the model parameters are updated. The calculation formula of the model loss function is:

[0057]

[0058] Where k is the detection behavior type; n is the maximum number of detection objects allowed in the same picture, divided into individual objects, interactive objects and group objects; m is the mask code, if the detection object is n valid, then m n = 1, otherwise m n = 0; y is the label, if the category is i, then y i = 1, otherwise y i = 0; p i is the probability of category i.

[0059] In the above, the historical feature library timely saves new data according to the time sequence length requirement, and timely deletes the old data.

[0060] In the above training process, only the neural network other than the BERT network is updated (the BERT network parameters are always in a frozen state).

[0061] Step 4, extracting individual behavior and group behavior from the point cloud data according to the behavior detection model.

[0062] This step is the model inference process, specifically: before the model performs inference, the behavior detection model is quantized using a tensorrt architecture to obtain a behavior detection quantization model; the three-dimensional point cloud data of the detected object is used as input, and the quantization model is used for analysis. As Figure 4 shown, when the LSTM network data result is decomposed, the mask mask recorded in the above is used to extract effective feature parameters related to individual behavior and group behavior, and individual behavior and group behavior are extracted in the order of individual behavior and group behavior in the effective feature parameters.

[0063] Embodiment 2

[0064] The embodiment provides a detection system of a BERT behavior detection method based on binocular vision, comprising:

[0065] A data acquisition module is configured to acquire multi-angle video data of a to-be-detected object collected by a plurality of binocular stereo cameras, and record a reference coordinate system of each camera in the binocular stereo camera.

[0066] The data preprocessing module is configured to convert the video data into binocular stereo image data, and sequentially perform instance segmentation and feature extraction matching on the image data to form point cloud data, and convert the point cloud data into the reference coordinate system.

[0067] The data training module is configured to perform feature optimization based on the converted point cloud data and the BERT network, and train the LSTM network based on the optimized features to finally obtain a behavior detection model.

[0068] The model inference module is configured to extract individual behavior and group behavior from the point cloud data according to the behavior detection model.

[0069] The remaining embodiments are as described in Embodiment 1.

[0070] If the above functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the prior art that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0071] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code. The solutions in the embodiments of the present application can be implemented in various computer languages, such as object-oriented programming languages Java and interpreted scripting language JavaScript.

[0072] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0073] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0074] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0075] While preferred embodiments of the application have been described, modifications and variations can be apparent to those skilled in the art once aware of the general underlying concepts. Accordingly, the appended claims intend to embrace all such modifications and variations as fall within the scope of the application.

[0076] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A BERT behavior detection method based on binocular vision, characterized in that, The method comprises the following steps: acquiring multi-angle video data of an object to be detected collected by a plurality of binocular stereo cameras, and recording a reference coordinate system of each camera in the binocular stereo cameras; converting the video data into binocular stereo image data, and sequentially performing instance segmentation and feature extraction matching on the image data to form point cloud data, and converting the point cloud data to the reference coordinate system; performing feature optimization based on the converted point cloud data and a BERT network, and training an LSTM network based on the optimized features to finally obtain a behavior detection model; the specific steps of the feature optimization include: mapping the point cloud data to a low-dimensional feature vector; performing feature fusion on the low-dimensional feature vector to construct a high-level feature vector; inputting the high-level feature vector into the BERT network to output a BERT feature vector; storing the BERT feature vector in a pre-constructed historical feature library, and simultaneously extracting a plurality of BERT feature vectors from the historical feature library in chronological order to construct a time sequence feature vector; extracting individual behavior and group behavior from the point cloud data according to the behavior detection model.

2. The BERT behavior detection method based on binocular vision according to claim 1, characterized in that, The process of forming point cloud data specifically includes: converting the video data into binocular stereo image data; performing instance segmentation on left and right image data in the binocular stereo image data and assigning instance identifiers id; regarding the ids in the left and right images as the same object, and performing feature extraction matching on the same object in the left and right images to form three-dimensional point cloud data. 3.The BERT behavior detection method based on binocular vision of claim 2, characterized in that, YOLOv8 segment is used for instance segmentation.

4. The BERT behavior detection method based on binocular vision according to claim 1, characterized in that, The point cloud data is sequentially mapped to a low-dimensional feature vector through convolution and fully connected network.

5. The binocular vision-based BERT behavior detection method according to claim 1, wherein, Before inputting the high-level feature vector into the BERT network, vector alignment is performed, and the corresponding mask mask is recorded.

6. The binocular vision-based BERT behavior detection method according to claim 1, wherein, The specific steps of obtaining the behavior detection model include: inputting the time sequence feature vector into the LSTM network to obtain network output results; based on the network output results, calculating a loss value using a loss function; based on the loss value, updating the model parameters of the LSTM network using a back propagation algorithm; repeat the above steps to iteratively train the LSTM network until the loss value is less than a preset threshold or the number of iterations reaches a preset upper limit, and finally obtain the behavior detection model.

7. The binocular vision-based BERT behavior detection method according to claim 5, characterized in that, During the process of updating the model parameters of the LSTM network, the effective output bits are selected according to the mask mask, and then the loss value is calculated and the model parameters are updated. 8.The BERT behavior detection method based on binocular vision of claim 5, wherein, The specific steps of extracting individual behavior and group behavior include: quantizing the behavior detection model to obtain a behavior detection quantization model; inputting the point cloud data into the behavior detection quantization model, and extracting effective feature parameters related to individual behavior and group behavior based on the mask mask; based on the effective feature parameters, individual behavior and group behavior are extracted.

9. A detection system based on the binocular vision-based BERT behavior detection method according to any one of claims 1-8, characterized in that, It includes: a data acquisition module for acquiring multi-angle video data of an object to be detected collected by a plurality of binocular stereo cameras, and recording a reference coordinate system of each camera in the binocular stereo cameras; The data preprocessing module is configured to convert the video data into binocular stereo image data, and sequentially perform instance segmentation and feature extraction matching on the image data to form point cloud data, and convert the point cloud data to the reference coordinate system; The data training module is configured to perform feature optimization based on the converted point cloud data and a BERT network, and train an LSTM network based on the optimized features, and finally obtain a behavior detection model; The model inference module is configured to extract individual behavior and group behavior from the point cloud data according to the behavior detection model.

Images