Behavior recognition method and device, equipment, storage medium and computer program product

Abstract

The invention discloses a behavior recognition method and device, equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring multi-modal data; the multi-modal data comprises image data and text data related to a job task; inputting the multi-modal data into a pre-trained classification and recognition model to obtain a classification and recognition result; the classification identification result comprises first classification identification information and second classification identification information, the classification identification model comprises a first network model and a second network model, the first network model is related to the first classification identification information, and the second network model is related to the second classification identification information; fusing the first classification identification information and the second classification identification information to obtain a target identification result; the target recognition result represents a recognition result for a dangerous behavior in the job task.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a behavior recognition method, device, equipment, storage medium and computer program product. Background Art

[0002] With the construction and safe operation of communication equipment, it is necessary to carry out on-site operations in various production activities such as infrastructure construction and equipment maintenance. Among them, how to use artificial intelligence, edge computing, sensors, the Internet of Things and other technologies to realize on-site analysis and real-time judgment of various data of on-site operations, and to achieve high-accuracy identification of violations at a relatively low cost, is the key and difficulty of carrying out on-site operation safety management and control.

[0003] At present, most of the solutions in the related technologies use a single classification algorithm such as target detection (for example, deploying cameras on site for detection) or traditional classifiers (for example, video analysis devices based on artificial intelligence) to complete the identification of dangerous behaviors in work tasks. However, since the actual identification scenarios required are complex and diverse, often containing different complex backgrounds, it is difficult for a single algorithm model to obtain accurate and robust recognition results, resulting in low recognition accuracy and insufficient robustness. In addition, due to various factors such as the many environmental interference factors in on-site operations, easy occlusion, the target to be identified may not be significant enough, and the samples trained on the corresponding model are atypical or insufficient, the overall accuracy of the identification solution of the related technology is not high, and there are many cases of missed detection and false detection, which can easily lead to accidents and make it difficult to maintain and ensure the safety of on-site workers. Summary of the invention

[0004] In order to solve the technical problems existing in the related technologies, the embodiments of the present application provide a behavior recognition method, device, equipment, storage medium and computer program product.

[0005] To achieve the above purpose, the technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a behavior recognition method, the method comprising:

[0007] Acquire multimodal data; the multimodal data includes image data and text data related to the task;

[0008] Inputting the multimodal data into a pre-trained classification recognition model to obtain a classification recognition result; the classification recognition result includes first classification recognition information and second classification recognition information, the classification recognition model includes a first network model and a second network model, the first network model is related to the first classification recognition information, and the second network model is related to the second classification recognition information;

[0009] The first classification identification information and the second classification identification information are fused to obtain a target identification result; the target identification result represents an identification result for the dangerous behavior in the operation task.

[0010] In a second aspect, an embodiment of the present application further provides a behavior recognition device, the device comprising:

[0011] An acquisition unit, used for acquiring multimodal data; the multimodal data includes image data and text data related to the operation task;

[0012] A processing unit, configured to input the multimodal data into a pre-trained classification recognition model to obtain a classification recognition result; the classification recognition result includes first classification recognition information and second classification recognition information, the classification recognition model includes a first network model and a second network model, the first network model is related to the first classification recognition information, and the second network model is related to the second classification recognition information;

[0013] A fusion unit is used to fuse the first classification identification information and the second classification identification information to obtain a target recognition result; the target recognition result represents the recognition result of the dangerous behavior in the operation task.

[0014] In a third aspect, an embodiment of the present application further provides a behavior recognition device, comprising: a processor and a memory for storing a computer program that can be run on the processor;

[0015] Wherein, when the processor is used to run the computer program, it executes the steps of the behavior recognition method described in the embodiment of the present application.

[0016] In a fourth aspect, an embodiment of the present application further provides a storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the behavior recognition method described in the embodiment of the present application are implemented.

[0017] In a fifth aspect, an embodiment of the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the behavior recognition method described in the embodiment of the present application.

[0018] The behavior recognition method, apparatus, device, storage medium and computer program product provided in the embodiments of the present application obtain multimodal data; the multimodal data includes image data and text data related to the work task; the multimodal data is input into a pre-trained classification recognition model to obtain a classification recognition result; the classification recognition result includes first classification recognition information and second classification recognition information, the classification recognition model includes a first network model and a second network model, the first network model is related to the first classification recognition information, and the second network model is related to the second classification recognition information; the first classification recognition information and the second classification recognition information are fused to obtain a target recognition result; the target recognition result represents the recognition result of the dangerous behavior in the work task. The behavior recognition method of the embodiment of the present application is adopted, by using two different types of models, namely, a first network model and a second network model, to classify and identify the acquired multimodal data respectively, to obtain first classification recognition information and second classification recognition information, and to fuse the first classification recognition information and the second classification recognition information to obtain the final target recognition result, that is, to identify dangerous behaviors in the operation task. In this way, the differences and complementarities between different models are fully utilized, and the classification recognition information output by different models is fused to obtain more accurate and reliable recognition results, which can improve the accuracy and robustness of dangerous behavior identification in the operation task, and at the same time can also improve the level of intelligence of on-site operation safety management, help prevent and reduce the occurrence of accidents, and better maintain and ensure the safety of on-site workers. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flowchart of a first solution for identifying illegal behaviors in the related art;

[0020] Figure 2 A flowchart of the second solution for identifying illegal behaviors in the related art;

[0021] Figure 3 The process diagram of the behavior recognition method of the present application embodiment is as follows: Figure 1 ;

[0022] Figure 4 A schematic diagram of the structure of a first network model according to an embodiment of the present application;

[0023] Figure 5 A schematic diagram of the structure of a second network model according to an embodiment of the present application;

[0024] Figure 6 The process diagram of the behavior recognition method of the present application embodiment is as follows: Figure 2 ;

[0025] Figure 7 The process diagram of the behavior recognition method of the present application embodiment is as follows: Figure 3 ;

[0026] Figure 8 A schematic diagram of the training process of the second network model according to an embodiment of the present application;

[0027] Fig. 9 The process diagram of the behavior recognition method of the present application embodiment is as follows: Figure 4 ;

[0028] Fig.10 A schematic diagram of the structure of a behavior recognition device according to an embodiment of the present application;

[0029] Fig.11 This is a schematic diagram of the hardware composition structure of the behavior recognition device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] The present application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0032] With the construction and safe operation of communication equipment, it is necessary to carry out on-site operations in various production activities such as infrastructure construction and equipment maintenance. In practical applications, due to the large number of on-site operation points, large number of construction personnel, and complex environment, the on-site operation safety control is scattered, the personnel and equipment control is not strong enough, and the safety supervision manpower is insufficient. At present, on-site operation safety control based on various monitoring terminals is generally carried out. For example, based on the joint use of various sensors, video image recognition systems, wearable equipment and tools, the illegal behaviors (also called dangerous behaviors) of on-site operations are automatically identified and alarmed to improve the safety control level of on-site operations.

[0033] Among them, how to use artificial intelligence, edge computing, sensors, the Internet of Things and other technologies to realize on-site analysis and real-time judgment of various data of field operations, and to achieve high-accuracy identification of violations of field operations at a relatively low cost, is the key and difficulty of carrying out safety management and control of field operations such as communication installation and maintenance.

[0034] To address this issue, the more mainstream approach in related technologies is to deploy cameras or AI-based video analysis devices on site to identify people on site in video images, and specifically distinguish between violations such as people not wearing helmets or seat belts, and immediately analyze the violations and issue warnings.

[0035] In the related art, there are two exemplary solutions for identifying traffic violations:

[0036] Figure 1 The present invention is a flow chart of a scheme 1 for identifying illegal behaviors in the related art, and the scheme relates to a method for identifying illegal behaviors in power field operations by integrating multimodal data analysis, and specifically includes the following steps: collecting historical data of target power field operation images, illegal behaviors and digital work tickets; based on a pre-trained target recognition model, performing target recognition on the historical data of the collected operation images to obtain target recognition results; extracting feature information from the historical data of the digital work tickets; establishing a bidirectional transformer encoder (BERT, Bidirectional Encoder Representation from Transformers) classification model, taking the target recognition results of the historical data of the operation images and the feature information in the historical data of the digital work tickets as input data, and the historical data of the illegal behaviors as label data, training the BERT model to obtain a trained illegal behavior recognition model; obtaining a current operation image of the target power field and a digital work ticket corresponding to the current operation, performing target recognition on the current operation image through a pre-trained target recognition model to obtain a current target recognition result, extracting the feature information of the digital work ticket corresponding to the current operation and inputting it into the illegal behavior recognition model together with the current target recognition result to obtain the illegal behavior recognition result. It can be seen that the first solution uses the image recognition output as the feature information of the text model, fails to fully exploit the effective information in the image data, and the performance of the BERT model used in the text processing model is relatively limited.

[0037] Figure 2 This is a flow chart of the second scheme for identifying illegal behaviors in the related art. This scheme involves a method and system for identifying dangerous behaviors in a smart construction site, which specifically includes the following steps: obtaining a first example construction site monitoring video data cluster, combining the video semantic features and dangerous behavior label data of the example construction site monitoring video data, performing dangerous behavior recognition learning on the basic construction site dangerous behavior recognition network, generating a construction site dangerous behavior recognition network after dangerous behavior recognition learning, combining the second example construction site monitoring video data cluster, performing extended dangerous behavior recognition learning on the construction site dangerous behavior recognition network after dangerous behavior recognition learning, and if the learned construction site dangerous behavior recognition network meets the network convergence conditions, then the learned construction site dangerous behavior recognition network is used as the target construction site dangerous behavior recognition network. It can be seen that this scheme 2 mainly uses video and semantic information for behavior recognition, and does not use models with better performance such as large visual models.

[0038] It can be seen that most of the solutions of related technologies use a single classification algorithm such as target detection or traditional classifiers to complete the identification of dangerous behaviors in work tasks. However, due to the complexity and diversity of the actual identification scenarios required, which often contain different complex backgrounds, it is difficult for a single algorithm model to obtain accurate and robust recognition results, resulting in low recognition accuracy and insufficient robustness. In addition, due to various factors such as the many environmental interference factors in on-site operations, easy occlusion, the target to be identified may not be significant enough, and the samples trained by the corresponding artificial intelligence model are atypical or insufficient, the overall accuracy of the identification solution of the related technology is not high, and there are many cases of missed detection and false detection, which can easily lead to accidents and make it difficult to maintain and ensure the safety of on-site workers.

[0039] Based on this, an embodiment of the present application proposes a behavior recognition method. In various embodiments of the present application, by utilizing two different types of models, namely a first network model and a second network model, the acquired multimodal data are classified and recognized respectively to obtain first classification recognition information and second classification recognition information, and the first classification recognition information and the second classification recognition information are fused to obtain the final target recognition result, that is, to identify dangerous behaviors in the operation task. In this way, the differences and complementarities between different models are fully utilized, and the classification recognition information output by different models is fused to obtain more accurate and reliable recognition results, which can improve the accuracy and robustness of dangerous behavior identification in the operation task, and at the same time can also improve the level of intelligence of on-site operation safety management, help prevent and reduce the occurrence of accidents, and better maintain and ensure the safety of on-site workers.

[0040] The present application embodiment provides a behavior recognition method, which is applied to a behavior recognition device. Figure 3 The process diagram of the behavior recognition method of the present application embodiment is as follows: Figure 1 ;like Figure 3 As shown, the behavior recognition method includes:

[0041] Step 301: Acquire multimodal data.

[0042] In an embodiment of the present application, the multimodal data includes image data and text data related to the operation task, wherein, exemplarily, the operation task may be an on-site operation task involving communication equipment, such as infrastructure construction tasks and equipment maintenance tasks during the construction or safe operation of communication equipment.

[0043] Taking the operation task as an equipment maintenance task as an example, the image data related to the operation task can be the video image information of the construction personnel in the equipment maintenance task, and the text data related to the operation task can be the text image information of the construction personnel in the equipment maintenance task. The behavior recognition method of the embodiment of the present application can be applied to the scene of the on-site operation of communication equipment; that is, the behavior recognition method can be a dangerous behavior recognition method applied to the on-site operation of communication equipment.

[0044] Step 302: Input the multimodal data into a pre-trained classification and recognition model to obtain a classification and recognition result; the classification and recognition result includes first classification and recognition information and second classification and recognition information.

[0045] In an embodiment of the present application, the classification identification model includes a first network model and a second network model, the first network model is related to the first classification identification information, and the second network model is related to the second classification identification information.

[0046] It should be noted that the first network model and the second network model are two different types of network models, that is, the first network model and the second network model are two different types of classification and recognition models. Exemplarily, in actual application, the first network model and the second network model can be different types of classifiers, which are used to classify and recognize the input multimodal data respectively and obtain corresponding recognition results. Among them, the first network model can be used to classify and recognize multimodal data to obtain first classification recognition information, that is, the output result of the first network model is the first classification recognition information; the second network model can be used to classify and recognize multimodal data to obtain second classification recognition information, that is, the output result of the second network model is the second classification recognition information.

[0047] Based on this, in one embodiment, the multimodal data is input into a pre-trained classification and recognition model to obtain a classification and recognition result, including: inputting the multimodal data into the first network model to obtain first classification and recognition information output by the first network model; inputting the multimodal data into the second network model to obtain second classification and recognition information output by the second network model.

[0048] In actual application, in one embodiment, the multimodal data is input into the first network model to obtain the first classification identification information output by the first network model, including: inputting the multimodal data into the first network model to perform feature extraction on the multimodal data through the first network model to obtain multi-scale features, aggregating the multi-scale features to obtain aggregated features, and generating the first classification identification information based on the aggregated features.

[0049] Here, the first network model can be a small target detection model, which mainly performs classification and recognition on the acquired multimodal data (such as image data and text data related to the work task), wherein the small target detection model can select the YOLO-v10 model, that is, in the embodiment of the present application, the YOLO-v10 model can be used to perform classification and recognition processing on the input multimodal data.

[0050] It should be noted that the YOLO-v10 model is built on the basis of the previous version of the YOLO (You Only Look Once) model (an algorithm for target detection using convolutional neural networks) by eliminating non-maximum suppression (NMS) and optimizing various model components to classify and recognize multimodal data.

[0051] Based on this, in one embodiment, before inputting the multimodal data into the first network model to obtain the first classification identification information output by the first network model, the method further includes: generating the first network model.

[0052] Figure 4 This is a schematic diagram of the structure of the first network model of the embodiment of the present application. Taking the first network model as the YOLO-v10 model as an example, it mainly includes the following functional modules: a backbone module, a neck module, a one-to-many head module, a one-to-one head module, and a consistent matching (Consistent Match Metric) module of scale. Among them, the backbone module is responsible for feature extraction of multimodal data. Exemplarily, the backbone module can use an enhanced version of the cross-stage partial network (CSPNet, Cross Stage Partial Network) to improve gradient flow and reduce computational redundancy. The neck module may include a path aggregation network (PAN, Path Aggregation Network) layer, and the PAN layer is used to fuse features of different scales and pass the fused features to the head module. A one-to-many head module is used to generate multiple prediction results for each object during the model training process to provide rich supervision signals and improve the accuracy of learning. A one-to-one head module is used to generate an optimal prediction result for each object during the model reasoning process without NMS, thereby reducing delays and improving classification and recognition efficiency. The scale consistency matching module is used to perform scale consistency matching on the results of model training and the results of model inference.

[0053] Here, for inputting the multimodal data into the first network model and obtaining the first classification identification information output by the first network model, specifically, the multimodal data can be first input into the backbone network module of the first network model, and the multimodal data is subjected to feature extraction through the backbone network module to obtain multi-scale features; then, the multi-scale features are passed to the PAN layer of the first network model, and the multi-scale features are aggregated through the PAN layer to obtain aggregated features, that is, features of different scales are fused through the PAN layer to obtain fused aggregated features; next, the aggregated features are passed to the one-to-one head module of the first network model, and the aggregated features are inferred through the one-to-one head module to obtain the first classification identification information, and the first classification identification information is an optimal predicted classification identification result generated for the aggregated features.

[0054] In actual application, the behavior recognition device can optimize the parameters of the pre-trained first network model through the constructed training data set to realize the training process of the first network model.

[0055] Based on this, in one embodiment, generating the first network model includes: constructing a first training data set; generating the first training data set based on a labeled data sample set; and optimizing the parameters of the pre-trained first network model using the first training data set to obtain a trained first network model.

[0056] Here, the behavior recognition device can first construct a data sample set and label the data in the data sample set, and then construct a first training data set based on the labeled data sample set. After constructing the first training data set, the first training data set can be used to optimize the parameters of the pre-trained first network model, thereby obtaining a trained first network model.

[0057] It should be noted that labeling the data in the data sample set is actually labeling the data in the data sample set to obtain corresponding label data, that is, the labeled data sample set. For example, the label data may include but is not limited to the recognition and detection of actions such as not wearing a safety helmet, not having a guardian, and not wearing a safety belt correctly. In actual application, for constructing a first training data set based on the labeled data sample set, the first training data set can be constructed based on the labeled data sample set according to a set ratio. For example, the labeled data sample set is used to construct the first training data set and the first verification data set at a ratio of 80% and 20%.

[0058] In actual application, in one embodiment, the multimodal data is input into the second network model to obtain the second classification identification information output by the second network model, including: inputting the multimodal data into the second network model to generate multimodal feature sequence information based on the multimodal data through the second network model, and generating the second classification identification information based on the multimodal feature sequence information.

[0059] Here, the second network model may be a visual big model. Specifically, the second network model may be an open source visual big model, wherein the visual big model may select the CogVLM algorithm model, which mainly performs a deep fusion of language features and visual features to achieve a deep understanding of images and texts.

[0060] In actual application, since the multimodal data includes image data and text data, in the embodiment of the present application, the behavior recognition device encodes the image data and the text data separately.

[0061] Based on this, in one embodiment, the multimodal data is input into the second network model to generate multimodal feature sequence information based on the multimodal data through the second network model, including: inputting the multimodal data into the second network model to encode the image data through the first encoder in the second network model to obtain image feature sequence information, encoding the text data through the second encoder in the second network model to obtain text feature sequence information, and constructing the multimodal feature sequence information based on the image feature sequence information and the text feature sequence information.

[0062] Here, for constructing the multimodal feature sequence information based on the image feature sequence information and the text feature sequence information, exemplarily, the image feature sequence information and the text feature sequence information may be connected (also referred to as splicing) to obtain the multimodal feature sequence information.

[0063] Figure 5The schematic diagram of the structure of the second network model of the embodiment of the present application is taken as the CogVLM algorithm model, which mainly includes the following functional modules: a vision transformer (ViT) encoder (ViTencoder, corresponding to the aforementioned first encoder), a multilayer perceptron (MLP) adapter (MLPAdapter), a language encoder (corresponding to the aforementioned second encoder), and a pre-trained large language model (GPT, Generative Pre-Trained Transformer). Specifically, the image data (Patchified images) is input into the ViTencoder, and the image data is encoded by the ViT encoder to obtain image feature sequence information (Image features sequence), and the text data (Paired Text) is input into the language encoder (Word embedding), and the text data is encoded by the language encoder to obtain text feature sequence information (Text feature sequence). After the image data is input into the ViT encoder and the image data is encoded by the ViT encoder to obtain the image feature sequence information, the multilayer perceptron adapter (MLP Adapter) adapts the parameter difference between the visual transformer encoder and the pre-trained large language model to ensure that the parameters of the pre-trained large language model remain unchanged. After obtaining the image feature sequence information and the text feature sequence information, the image feature sequence information and the text feature sequence information are concatenated to obtain multimodal feature sequence information.

[0064] Step 303: The first classification identification information and the second classification identification information are integrated to obtain a target identification result; the target identification result represents an identification result for the dangerous behavior in the operation task.

[0065] In actual application, after obtaining the first classification identification information and the second classification identification information, the behavior recognition device can use evidence theory to fuse the first classification identification information and the second classification identification information to obtain the final target recognition result.

[0066] Based on this, in one embodiment, the first classification identification information and the second classification identification information are fused to obtain a target identification result, including: using evidence combination rules to fuse the first classification identification information and the second classification identification information to obtain the target identification result.

[0067] In actual application, in one embodiment, the first classification identification information and the second classification identification information are merged using the evidence combination rule to obtain the target identification result, including: converting the first classification identification information into a first evidence function, and converting the second classification identification information into a second evidence function; the first evidence function represents the credibility of the first network model, and the second evidence function represents the credibility of the second network model; using the evidence combination rule, the first evidence function and the second evidence function are merged to obtain a target evidence function; based on the target evidence function, the target identification result is determined.

[0068] Here, the behavior recognition device can convert the first classification recognition information and the second classification recognition information into corresponding evidence functions, respectively. The evidence functions can be represented by different mass functions m1 and m2. The behavior recognition device can convert the first classification recognition information into the first evidence function. Assume that the recognition framework is recorded as Θ={ω1,ω2...ω M}, then the first evidence function can be expressed as m1={m1(ω1),m1(ω2)...m1(ω M ),m1(Θ)}, and m1 is used to characterize the credibility of the first network model, where m1(ω i ), i=1,2...M is used to represent the proposition ω in the identification framework i The behavior recognition device can convert the second classification recognition information into a second evidence function, and the second evidence function can be expressed as m2 = {m2(ω1), m2(ω2)...m2(ω M ),m2(Θ)}, and m2 is used to characterize the credibility of the second network model, where m i (Θ), i = 1, 2 is used to represent the support for the entire identification framework, which can better quantify the uncertainty in the classification process and quantify the lack of evidence.

[0069] Here, in actual applications, if the image collection environment is not good during the audit identification task, it is easy to cause unclear image data problems, and for images of poor quality, uncertainty in category attribution is likely to occur during classification training. In order to solve this technical problem, in an embodiment of the present application, two different algorithm models are used for training and processing respectively, and the uncertainty of the data itself can be modeled from different angles. Then, the two sets of evidence are fused, that is, the first evidence function and the second evidence function are fused, which can negate some of the modeling uncertainties and further improve the recognition effect.

[0070] In practical applications, the behavior recognition device can use the probability conversion rules in the evidence theory framework to determine the target recognition result based on the target evidence function.

[0071] Based on this, in one embodiment, the target identification result is determined based on the target evidence function, including: performing probability conversion on the target evidence function to obtain a target prediction probability; the target prediction probability represents the prediction probability corresponding to the category of each dangerous behavior in the work task; and determining the target prediction probability as the target identification result.

[0072] Here, assuming that the target evidence function obtained by fusing the first evidence function m1 and the second evidence function m2 is m, the target evidence function m is converted into a probability using the following formula (1) to obtain the target prediction probability:

[0073]

[0074] Among them, P(ω j ) represents the transformed j The probability distribution of the target prediction probability; Θ = {ω1,ω2...ω M} represents the identification framework; m(A) represents the support of the fused evidence function m for proposition A in the identification framework, and |A| represents the number of elements in proposition A.

[0075] In actual application, after determining the target recognition result, the behavior recognition device can also output and display the target recognition result.

[0076] Based on this, in one embodiment, after the first classification identification information and the second classification identification information are merged to obtain a target identification result, the method further includes: outputting and displaying the target identification result; wherein the target identification result includes one or more of the following information: the category of dangerous behavior; the probability corresponding to each category of the dangerous behavior.

[0077] Here, in the embodiment of the present application, the categories of dangerous behaviors may include but are not limited to: not wearing a safety helmet, not having a guardian, not wearing a seat belt correctly, and other dangerous behaviors.

[0078] The present application embodiment also provides another behavior recognition method, which is applied to a behavior recognition device. Figure 6 The process diagram of the behavior recognition method of the present application embodiment is as follows: Figure 2 ;like Figure 6 As shown, the behavior recognition method includes:

[0079] Step 601: Acquire multimodal data.

[0080] In an embodiment of the present application, the multimodal data includes image data and text data related to the work task.

[0081] Step 602: Input the multimodal data into a first network model of a classification recognition model to obtain first classification recognition information output by the first network model.

[0082] In the embodiment of the present application, the first network model is related to the first classification identification information, and the second network model is related to the second classification identification information.

[0083] In one embodiment, the multimodal data is input into a first network model of a classification recognition model to obtain first classification recognition information output by the first network model, including: inputting the multimodal data into the first network model to perform feature extraction on the multimodal data through the first network model to obtain multi-scale features, aggregating the multi-scale features to obtain aggregated features, and generating the first classification recognition information based on the aggregated features.

[0084] In one embodiment, before inputting the multimodal data into the first network model of the classification recognition model to obtain the first classification recognition information output by the first network model, the method further includes: generating the first network model.

[0085] In one embodiment, generating the first network model includes: constructing a first training data set; generating the first training data set based on a labeled data sample set; and optimizing the parameters of the pre-trained first network model using the first training data set to obtain a trained first network model.

[0086] Step 603: input the multimodal data into the second network model of the classification and recognition model to obtain second classification and recognition information output by the second network model.

[0087] In one embodiment, the multimodal data is input into the second network model of the classification recognition model to obtain second classification recognition information output by the second network model, including: inputting the multimodal data into the second network model to generate multimodal feature sequence information based on the multimodal data through the second network model, and generating the second classification recognition information based on the multimodal feature sequence information.

[0088] In one embodiment, the multimodal data is input into the second network model to generate multimodal feature sequence information based on the multimodal data through the second network model, including: inputting the multimodal data into the second network model to encode the image data through the first encoder in the second network model to obtain image feature sequence information, encoding the text data through the second encoder in the second network model to obtain text feature sequence information, and constructing the multimodal feature sequence information based on the image feature sequence information and the text feature sequence information.

[0089] Step 604: Convert the first classification identification information into a first evidence function, and convert the second classification identification information into a second evidence function.

[0090] In an embodiment of the present application, the first evidence function represents the credibility of the first network model, and the second evidence function represents the credibility of the second network model.

[0091] Step 605: Using evidence combination rules, the first evidence function and the second evidence function are merged to obtain a target evidence function.

[0092] Step 606: Determine a target recognition result based on the target evidence function.

[0093] In the embodiment of the present application, the target recognition result represents the recognition result of the dangerous behavior in the operation task.

[0094] In one embodiment, the target identification result is determined based on the target evidence function, including: performing probability conversion on the target evidence function to obtain a target prediction probability; the target prediction probability represents the prediction probability corresponding to the category of each dangerous behavior in the work task; and determining the target prediction probability as the target identification result.

[0095] In one embodiment, after determining the target identification result based on the target evidence function, the method further includes: outputting and displaying the target identification result; wherein the target identification result includes one or more of the following information: the category of dangerous behavior; and the probability corresponding to each category of the dangerous behavior.

[0096] It should be noted that the specific processing process of the behavior recognition device to complete the behavior recognition has been described in detail above and will not be repeated here.

[0097] By adopting the technical solution of the embodiment of the present application, two different types of models, namely the first network model and the second network model, are used to classify and identify the acquired multimodal data respectively, to obtain first classification identification information and second classification identification information, and the first classification identification information and the second classification identification information are fused to obtain the final target recognition result, that is, to identify dangerous behaviors in the operation task. In this way, the differences and complementarities between different models are fully utilized, and the classification identification information output by different models is fused to obtain more accurate and reliable recognition results, which can improve the accuracy and robustness of the identification of dangerous behaviors in the operation tasks, and at the same time can also improve the level of intelligence of on-site operation safety management, help prevent and reduce the occurrence of accidents, and better maintain and ensure the safety of on-site workers.

[0098] The present application is described below in conjunction with application examples.

[0099] In response to the problems in related technologies such as low behavior recognition accuracy and insufficient robustness, as well as the presence of many missed detections and false detections, which easily lead to accidents and make it difficult to maintain and ensure the safety of on-site workers, the present application proposes a multimodal dangerous behavior recognition and analysis method and system for on-site operations of communication equipment. The method classifies and identifies the collected multimodal data based on a small target detection model (corresponding to the aforementioned first network model) and a large visual model (corresponding to the aforementioned second network model), obtains different results (corresponding to the aforementioned first classification identification information and the second classification identification information), and makes decision-making fusions on the different results. The behavior recognition performance is improved by fusing the differences and complementarities between different types of models, thereby being able to better serve the on-site operation tasks of communication equipment while improving the recognition accuracy and robustness, making it safer and more reliable.

[0100] Figure 7 The process diagram of the behavior recognition method of the present application embodiment is as follows: Figure 3 ,like Figure 7As shown, the main core modules of the scheme are a classification and recognition module and a decision fusion module. The main steps of the method are as follows: first, collect image text information (corresponding to the aforementioned multimodal data), wherein the image text information is related to the task; second, pass the image text information to the classification and recognition module, which mainly uses different types of classifiers (corresponding to the aforementioned first network model and the second network model) to classify and recognize the collected image text information, and obtain classification recognition information (corresponding to the aforementioned first classification recognition information and the second classification recognition information); then, pass the classification recognition information to the decision fusion module, that is, the decision fusion module receives the classification recognition information transmitted from the classification and recognition module, and uses evidence theory to fuse the classification recognition information to obtain the final recognition result (corresponding to the aforementioned target recognition result); finally, output and display the final recognition result.

[0101] It can be seen that the present application innovatively combines the classification and recognition module with the decision fusion module, and applies it to the on-site operation of communication equipment. It uses the target detection small model and the visual large model framework for decision recognition fusion, and outputs the final recognition result, which provides strong technical support for the on-site safety monitoring of communication equipment maintenance. This method not only improves the level of intelligence of on-site work safety management, but also can better identify dangerous behaviors in on-site operation of communication equipment, and also helps to prevent and reduce the occurrence of accidents, and better maintain and ensure the safety of on-site workers.

[0102] The specific implementation of the classification recognition module and the decision fusion module of the present application is described below.

[0103] 1. Classification and recognition module

[0104] The classification and recognition module in the present application mainly processes the acquired multimodal data (including image and text data), wherein the image and text data are classified and recognized using a small target detection model and a large visual model respectively.

[0105] 1) Small target detection model

[0106] For image text data, the target detection model can be used for recognition processing. The target detection model can use the latest target detection model YOLO-v10 model. The YOLO-v10 model is built on the basis of the previous version of the YOLO model by eliminating NMS and optimizing various model components, such as Figure 4 As shown in the figure, the main optimization and improvement modules of the YOLO-v10 model can be divided into the following four aspects:

[0107] Backbone network: responsible for feature extraction of multimodal data, which uses an enhanced version of CSPNet to improve gradient flow and reduce computational redundancy. Neck: responsible for aggregating features of different scales and passing the aggregated features to the head, where the neck includes a PAN layer to achieve effective multi-scale feature fusion. The head includes a one-to-one head and a one-to-many head, where the one-to-many head: generates multiple prediction results for each object during model training to provide rich supervision signals and improve learning accuracy. One-to-one head: generates an optimal prediction result for each object during model inference, without the need for NMS, thereby reducing latency and improving classification and recognition efficiency.

[0108] The main training and use process of the YOLO-v10 model is as follows: 1. Data collection and labeling: Based on the target detection model YOLO-v10, a data sample set is constructed, and the data in the data sample set is manually labeled, that is, the data in the data sample set is labeled to obtain the corresponding label data, where the label data includes but is not limited to the recognition detection of actions such as not wearing a safety helmet, not having a guardian, and not wearing a safety belt correctly. 2. The labeled data sample set is used to construct a training set (corresponding to the first training data set mentioned above) and a validation set at a ratio of 80% and 20%, and cross-validation is performed. The training and data are used to train the detection and recognition model, the model parameters are optimized, and the validation set is used to verify the model. After the YOLO-v10 model is trained, and after the model is deployed and used, the model reasoning is performed on the collected multimodal data, and a set of detection and recognition results can be obtained, which is recorded as the recognition decision result a (corresponding to the first classification recognition information mentioned above).

[0109] 2) Visual big model

[0110] The visual big model in this application mainly adopts an open source visual big model. By inputting the currently collected image text data into the visual big model, the analysis and recognition results of the current image text data can be obtained, and the category probability attribution information of each category of results can be obtained. Among them, the visual big model can adopt some currently common open source models, such as the CogVLM algorithm model.

[0111] like Figure 5 As shown in the figure, the CogVLM algorithm model mainly consists of four basic components: ViT encoder, MLP adapter, GPT and language encoder. Among them, the ViT encoder is responsible for extracting high-level features from image data and mapping them to the same dimensional space as GPT. The MLP adapter is responsible for adapting the parameter differences between the ViT encoder and GPT and keeping the parameters of GPT unchanged. The language encoder is responsible for extracting text features from text data. GPT is responsible for processing cross-modal sequences and outputting the final result, that is, connecting the image feature sequence information and the text feature sequence information to obtain multimodal feature sequence information.

[0112] Figure 8 Schematic diagram of the training process of the second network model of the embodiment of the present application, such as Figure 8 As shown in the figure, after encoding the image data and text data respectively, the encoded image features and text features are passed to the channel-by-channel normalization (LayerNorm) module, and the encoded image features and text features are normalized in the feature dimension (i.e., channel dimension) by the LayerNorm module, so that the LayerNorm is suitable for small batches or single samples through this channel-by-channel normalization method, and the complexity and calculation amount of the model are also reduced; next, the normalized image features (Image features) and text features (Text features) are passed to the QKV matrix (matrix) module respectively, and the query (Q, Query), key (K, Key), and value (V, Value) vectors of the normalized image features and text features are calculated by the QKV matrix module, and the calculated Q, K, and V vectors are passed to the multi-head attention (Muti-head Attention) module, the Q, K, V vectors processed by the attention mechanism are passed to the LayerNorm module, and the Q, K, V vectors processed by the attention mechanism are normalized again by the LayerNorm module to obtain the corresponding image features and text features, and the corresponding image features and text features are passed to the fully connected feedforward neural network (FFN, Fully-ConnectedFeedforward Neural Network) respectively. By enhancing the interaction between visual encoding and language encoding in each attention layer and FFN layer, and improving the visual comprehension ability, a set of detection and recognition results can also be obtained through the image comprehension ability of the CogVLM algorithm model, which is recorded as the recognition decision result b (corresponding to the second classification recognition information mentioned above).

[0113] 2. Decision fusion module

[0114] In this application, the input data samples are respectively decided by the target detection small model and the visual large model, and the corresponding recognition decision results a and b are obtained respectively. Then, the recognition decision results a and b are passed to the decision fusion module, and the recognition decision results a and b are fused through the decision fusion module to obtain the final more accurate fusion recognition result (corresponding to the aforementioned target recognition result).

[0115] This application mainly aims at auditing and identifying dangerous actions in the operation and maintenance of communication equipment, and the output result is the identified dangerous action category and the corresponding probability. The specific implementation of the decision fusion module is described below.

[0116] In this application, after obtaining the output results of the target detection small model and the visual large model, the output results of the two models can be converted into a set of mass functions m1 = {m1(ω1), m1(ω2)...m1(ω M ),m1(Θ)}, m2={m2(ω1),m2(ω2)...m2(ω M ), m2(Θ)}, which are used to represent the degree of trust of different classifiers on sample classes. i (Θ), i = 1, 2 is used to represent the support for the entire recognition framework, which can better quantify the uncertainty in the classification process and the lack of evidence. In addition, the use of two different classification algorithms, the small target detection model and the large visual model, can better process and characterize the original data information, i.e., multimodal data, and obtain two different sets of mass functions m1 and m2. Then, using the Dempster combination rule, the two sets of mass functions are fused through the following formula (2):

[0117]

[0118] Among them, m1(B) and m2(C) represent the support of m1 and m2 to their respective propositions B and C in their respective identification frameworks, respectively. K = ∑ B∩C＝A m1(B)m2(C) represent the conflict coefficients; m(A) represents the support of the fused evidence function m for proposition A in the identification framework.

[0119] It should be noted that the two sets of evidence functions are fused in this application mainly to deal with the problem of unclear image data caused by poor image collection environment during the audit and recognition process. For images of poor quality, it is easy to encounter uncertainty in category attribution during classification training. In order to solve this technical problem, this application uses two different algorithm models (such as the first network model and the second network model) for training and processing respectively, which can model the uncertainty of the data itself from different angles. Then, the fusion of the two sets of evidence functions can negatively affect part of the modeling uncertainty and further improve the recognition results. After the two sets of evidence functions m1 and m2 are fused to obtain the target evidence function m, the target evidence function can be probability converted through the above formula (1) to obtain the target prediction probability, that is, the target evidence function is probability converted and the decision is completed, and the fusion classification recognition result is output.

[0120] Fig. 9 The process diagram of the behavior recognition method of the present application embodiment is as follows: Figure 4 ,like Fig. 9As shown, the overall implementation process of the method is as follows: first, image data and text data related to the job task are obtained, and then the image data and text data are passed to the target detection small model and the visual large model for classification and recognition to obtain recognition decision results a and b, and the recognition decision results a and b are passed to the decision fusion module, and the recognition decision results a and b are fused through the decision fusion module to obtain the final recognition result (i.e., the fused recognition result), and the obtained fused recognition result is sent.

[0121] The key point of this application is to innovatively combine the small target detection model and the large visual model, and apply them to audit identification scenarios such as dangerous behaviors in field operations of communication equipment. Among them, the large visual model can better adapt to the needs of the task by learning the subtle differences in a large amount of data, and has better prediction performance when processing large-scale data sets or unseen samples, and better play the high accuracy of the large model in dealing with complex scenes and its more powerful generalization ability; at the same time, it can give full play to the high efficiency and accuracy of the small target detection model in detection and identification in conventional target scenes. Through the fusion recognition module, the small target detection model and the large visual model are comprehensively utilized, the differences and complementarities of different types of models are integrated, and the different output results with certain differences and complementarities are fused using evidence combination rules to improve detection and identification, which can better deal with the uncertainty problems in classification and identification in complex task scenarios, so as to obtain more accurate and reliable detection and identification results.

[0122] This application proposes a multimodal dangerous behavior identification and analysis method and system for on-site operation of communication equipment. Compared with the methods of related technologies, this application has the following beneficial effects:

[0123] (1) The fusion recognition module in this application uses two different types of models for detection and recognition: a small target detection model (such as the YOLO-v10 model) and a large visual model (such as the CogVLM algorithm model). This makes better use of the high accuracy of the YOLO-v10 model in detection and recognition in conventional target scenarios and the strong generalization of the large visual model, and can more fully explore sample data information and more effectively characterize and process various complex tasks.

[0124] (2) The main core modules of this application are the classification and recognition module and the decision fusion module. The classification and recognition module mainly realizes the acquisition of the required recognition image information and text data, uses the classification and recognition module to perform classification and recognition, and outputs the recognition results; the decision fusion module receives the classification and recognition information transmitted from the classification and recognition module, makes full use of the differences and complementarities between different types of models, uses evidence theory to integrate the decision information between different types of models, and obtains more accurate and reliable recognition results through fusion decision.

[0125] (3) This application can be applied to dangerous behavior identification scenarios in on-site communication equipment maintenance work, and can better identify dangerous behaviors in on-site operations. It not only improves the level of intelligence in on-site operation safety management, but also helps prevent and reduce the occurrence of accidents to maintain and ensure the safety of staff.

[0126] In summary, it can be seen that the present application combines the small target detection model and the large visual model, and utilizes the fusion recognition module to bring out the respective advantages of the large visual model and the small target detection model, so as to better deal with the uncertainty problems in classification and recognition caused by poor image quality in complex scenes, and obtain more accurate and reliable recognition results, making the on-site work of communication equipment more explicit and intelligent, which can better promote the development of communication services.

[0127] In order to implement the behavior recognition method of the embodiment of the present application, the embodiment of the present application also provides a behavior recognition device, Fig.10 Schematic diagram of the structure of the behavior recognition device according to the embodiment of the present application. Fig.10 As shown, the behavior recognition device includes:

[0128] An acquisition unit 1001 is used to acquire multimodal data; the multimodal data includes image data and text data related to the operation task;

[0129] The processing unit 1002 is used to input the multimodal data into a pre-trained classification recognition model to obtain a classification recognition result; the classification recognition result includes first classification recognition information and second classification recognition information, the classification recognition model includes a first network model and a second network model, the first network model is related to the first classification recognition information, and the second network model is related to the second classification recognition information;

[0130] The fusion unit 1003 is used to fuse the first classification identification information and the second classification identification information to obtain a target recognition result; the target recognition result represents the recognition result of the dangerous behavior in the operation task.

[0131] In one embodiment, the processing unit 1002 includes a first processing subunit and a second processing subunit; wherein,

[0132] The first processing subunit is used to input the multimodal data into the first network model to obtain first classification identification information output by the first network model;

[0133] The second processing subunit is used to input the multimodal data into the second network model to obtain second classification identification information output by the second network model.

[0134] In one embodiment, the first processing subunit is specifically configured to:

[0135] The multimodal data is input into the first network model to perform feature extraction on the multimodal data through the first network model to obtain multi-scale features, the multi-scale features are aggregated to obtain aggregated features, and the first classification identification information is generated based on the aggregated features.

[0136] In one embodiment, the behavior recognition device further includes: a generating unit; wherein,

[0137] The generating unit is used to generate the first network model before the first processing subunit inputs the multimodal data into the first network model to obtain the first classification identification information output by the first network model.

[0138] In one embodiment, the generating unit is specifically used for:

[0139] Constructing a first training data set; the first training data set is generated based on a labeled data sample set;

[0140] The first training data set is used to optimize the parameters of the pre-trained first network model to obtain a trained first network model.

[0141] In one embodiment, the second processing subunit is specifically configured to:

[0142] The multimodal data is input into the second network model to generate multimodal feature sequence information based on the multimodal data through the second network model, and the second classification identification information is generated based on the multimodal feature sequence information.

[0143] In one embodiment, the second processing subunit is specifically configured to:

[0144] The multimodal data is input into the second network model, so as to encode the image data through the first encoder in the second network model to obtain image feature sequence information, and the text data is encoded through the second encoder in the second network model to obtain text feature sequence information, and the multimodal feature sequence information is constructed based on the image feature sequence information and the text feature sequence information.

[0145] In one embodiment, the fusion unit 1003 is specifically used for:

[0146] The first classification identification information and the second classification identification information are fused by using evidence combination rules to obtain the target identification result.

[0147] In one embodiment, the fusion unit 1003 includes a transformation subunit, a fusion subunit and a determination subunit; wherein,

[0148] The conversion subunit is used to convert the first classification identification information into a first evidence function, and convert the second classification identification information into a second evidence function; the first evidence function represents the credibility of the first network model, and the second evidence function represents the credibility of the second network model;

[0149] The fusion subunit is used to fuse the first evidence function and the second evidence function by using the evidence combination rule to obtain a target evidence function;

[0150] The determination subunit is used to determine the target recognition result based on the target evidence function.

[0151] In one embodiment, the determining subunit is specifically used to:

[0152] Performing probability conversion on the target evidence function to obtain a target prediction probability; the target prediction probability represents the prediction probability corresponding to the category of each dangerous behavior in the task;

[0153] The target prediction probability is determined as the target recognition result.

[0154] In one embodiment, the behavior recognition device further includes: an output display unit; wherein,

[0155] The output display unit is used for outputting and displaying the target recognition result after the fusion unit 1003 fuses the first classification recognition information and the second classification recognition information to obtain the target recognition result;

[0156] The target recognition result includes one or more of the following information: the category of dangerous behavior; and the probability corresponding to each category of the dangerous behavior.

[0157] In actual application, the acquisition unit 1001 can be implemented by a communication interface in the behavior recognition device; the processing unit 1002 and the fusion unit 1003 can be implemented by a processor in the behavior recognition device.

[0158] It should be noted that: the behavior recognition device provided in the above embodiment only uses the division of the above program modules as an example when performing behavior recognition. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the behavior recognition device provided in the above embodiment and the behavior recognition method embodiment belong to the same concept. The specific implementation process is detailed in the behavior recognition method embodiment, which will not be repeated here.

[0159] Based on the hardware implementation of the above program modules, and in order to implement the behavior recognition method of the embodiment of the present application, the embodiment of the present application also provides a behavior recognition device, Fig.11 Schematic diagram of the hardware structure of the behavior recognition device of the embodiment of the present application. Fig.11 As shown, the behavior recognition device 1100 includes:

[0160] Communication interface 1101, capable of exchanging information with other devices;

[0161] The processor 1102 is connected to the communication interface 1101 to implement information interaction with other devices and is used to execute the above-provided behavior recognition method when running a computer program, and the computer program is stored in the memory 1103.

[0162] Specifically, the communication interface 1101 is used to obtain multimodal data; the multimodal data includes image data and text data related to the operation task;

[0163] The processor 1102 is used to input the multimodal data into a pre-trained classification and recognition model to obtain a classification and recognition result; the classification and recognition result includes first classification and recognition information and second classification and recognition information, the classification and recognition model includes a first network model and a second network model, the first network model is related to the first classification and recognition information, and the second network model is related to the second classification and recognition information; the first classification and recognition information are fused with the second classification and recognition information to obtain a target recognition result; the target recognition result represents the recognition result of the dangerous behavior in the operation task.

[0164] In one embodiment, the processor 1102 is specifically configured to:

[0165] Inputting the multimodal data into the first network model to obtain first classification identification information output by the first network model;

[0166] The multimodal data is input into the second network model to obtain second classification identification information output by the second network model.

[0167] In one embodiment, the processor 1102 is specifically configured to:

[0168] The multimodal data is input into the first network model to perform feature extraction on the multimodal data through the first network model to obtain multi-scale features, the multi-scale features are aggregated to obtain aggregated features, and the first classification identification information is generated based on the aggregated features.

[0169] In one embodiment, the processor 1102 is further configured to generate the first network model before inputting the multimodal data into the first network model to obtain the first classification identification information output by the first network model.

[0170] In one embodiment, the processor 1102 is specifically configured to:

[0171] Constructing a first training data set; the first training data set is generated based on a labeled data sample set;

[0172] The first training data set is used to optimize the parameters of the pre-trained first network model to obtain a trained first network model.

[0173] In one embodiment, the processor 1102 is specifically configured to:

[0174] The multimodal data is input into the second network model to generate multimodal feature sequence information based on the multimodal data through the second network model, and the second classification identification information is generated based on the multimodal feature sequence information.

[0175] In one embodiment, the processor 1102 is specifically configured to:

[0176] The multimodal data is input into the second network model, so as to encode the image data through the first encoder in the second network model to obtain image feature sequence information, and the text data is encoded through the second encoder in the second network model to obtain text feature sequence information, and the multimodal feature sequence information is constructed based on the image feature sequence information and the text feature sequence information.

[0177] In one embodiment, the processor 1102 is specifically configured to:

[0178] The first classification identification information and the second classification identification information are fused by using evidence combination rules to obtain the target identification result.

[0179] In one embodiment, the processor 1102 is specifically configured to:

[0180] Converting the first classification identification information into a first evidence function, and converting the second classification identification information into a second evidence function; the first evidence function represents the credibility of the first network model, and the second evidence function represents the credibility of the second network model;

[0181] Using the evidence combination rule, the first evidence function and the second evidence function are merged to obtain a target evidence function;

[0182] Based on the target evidence function, the target recognition result is determined.

[0183] In one embodiment, the processor 1102 is specifically configured to:

[0184] Performing probability conversion on the target evidence function to obtain a target prediction probability; the target prediction probability represents the prediction probability corresponding to the category of each dangerous behavior in the task;

[0185] The target prediction probability is determined as the target recognition result.

[0186] In one embodiment, the communication interface 1101 is also used to output and display the target identification result after the processor 1102 fuses the first classification identification information and the second classification identification information to obtain the target identification result; wherein the target identification result includes one or more of the following information: the category of dangerous behavior; the probability corresponding to each category of the dangerous behavior.

[0187] It should be noted that the specific processing process of the communication interface 1101 and the processor 1102 can be understood by referring to the above-mentioned behavior recognition method.

[0188] Of course, in actual application, the various components in the behavior recognition device 1100 are coupled together through the bus system 1104. It can be understood that the bus system 1104 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1104 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Fig.11 Various buses are labeled as bus system 1104 .

[0189] The memory 1103 in the embodiment of the present application is used to store various types of data to support the operation of the behavior recognition device 1100. Examples of such data include: any computer program used to operate on the behavior recognition device 1100.

[0190] The behavior recognition method disclosed in the above embodiment of the present application can be applied to the processor 1102, or implemented by the processor 1102. The processor 1102 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above behavior recognition method can be completed by the hardware integrated logic circuit or software instructions in the processor 1102. The above-mentioned processor 1102 may be a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 1102 can implement or execute the various behavior recognition methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the behavior recognition method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 1103, and the processor 1102 reads the information in the memory 1103 and completes the steps of the above behavior recognition method in combination with its hardware.

[0191] In an exemplary embodiment, the behavior recognition device 1100 can be implemented by one or more application-specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD), field-programmable gate array (FPGA), general-purpose processor, controller, microcontroller (MCU), microprocessor, or other electronic components to execute the aforementioned behavior recognition method.

[0192] It can be understood that the memory 1103 of the embodiment of the present application can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory, a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAMbus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 1103 described in the embodiments of the present application is intended to include but is not limited to these and any other suitable types of memories.

[0193] In an exemplary embodiment, the present application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, for example, a memory 1103 storing a computer program, and the computer program can be executed by a processor 1102 in the behavior recognition device 1100 to complete the steps of the behavior recognition method described in the above embodiment of the present application. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface storage, optical disk, or CD-ROM.

[0194] In an exemplary embodiment, the present application also provides a computer program product, including a computer program, which can be executed by the processor 1102 in the behavior recognition device 1100 to complete the steps of the behavior recognition method described in the above-mentioned embodiment of the present application.

[0195] It should be noted that: "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0196] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0197] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A behavior recognition method, characterized in that: The method comprises: Acquire multimodal data; the multimodal data includes image data and text data related to the task; Inputting the multimodal data into a pre-trained classification recognition model to obtain a classification recognition result; the classification recognition result includes first classification recognition information and second classification recognition information, the classification recognition model includes a first network model and a second network model, the first network model is related to the first classification recognition information, and the second network model is related to the second classification recognition information; The first classification identification information and the second classification identification information are fused to obtain a target identification result; the target identification result represents an identification result for the dangerous behavior in the operation task.

2. The method according to claim 1, characterized in that The step of inputting the multimodal data into a pre-trained classification and recognition model to obtain a classification and recognition result includes: Inputting the multimodal data into the first network model to obtain first classification identification information output by the first network model; The multimodal data is input into the second network model to obtain second classification identification information output by the second network model.

3. The method according to claim 2, characterized in that The step of inputting the multimodal data into the first network model to obtain first classification identification information output by the first network model includes: The multimodal data is input into the first network model to perform feature extraction on the multimodal data through the first network model to obtain multi-scale features, the multi-scale features are aggregated to obtain aggregated features, and the first classification identification information is generated based on the aggregated features.

4. The method according to claim 2 or 3, characterized in that: Before inputting the multimodal data into the first network model to obtain first classification identification information output by the first network model, the method further includes: generating the first network model; The generating the first network model comprises: Constructing a first training data set; the first training data set is generated based on a labeled data sample set; The first training data set is used to optimize the parameters of the pre-trained first network model to obtain a trained first network model.

5. The method according to claim 2, characterized in that: The step of inputting the multimodal data into the second network model to obtain second classification identification information output by the second network model includes: The multimodal data is input into the second network model to generate multimodal feature sequence information based on the multimodal data through the second network model, and the second classification identification information is generated based on the multimodal feature sequence information.

6. The method according to claim 5, characterized in that The step of inputting the multimodal data into the second network model to generate multimodal feature sequence information based on the multimodal data through the second network model includes: The multimodal data is input into the second network model, so as to encode the image data through the first encoder in the second network model to obtain image feature sequence information, and the text data is encoded through the second encoder in the second network model to obtain text feature sequence information, and the multimodal feature sequence information is constructed based on the image feature sequence information and the text feature sequence information.

7. The method according to claim 1, characterized in that The step of fusing the first classification identification information and the second classification identification information to obtain a target identification result includes: The first classification identification information and the second classification identification information are fused by using evidence combination rules to obtain the target identification result.

8. The method according to claim 7, characterized in that The using of the evidence combination rule to fuse the first classification identification information and the second classification identification information to obtain the target identification result includes: Converting the first classification identification information into a first evidence function, and converting the second classification identification information into a second evidence function; the first evidence function represents the credibility of the first network model, and the second evidence function represents the credibility of the second network model; Using the evidence combination rule, the first evidence function and the second evidence function are merged to obtain a target evidence function; Based on the target evidence function, the target recognition result is determined.

9. The method according to claim 8, characterized in that The determining the target recognition result based on the target evidence function includes: Performing probability conversion on the target evidence function to obtain a target prediction probability; the target prediction probability represents the prediction probability corresponding to the category of each dangerous behavior in the task; The target prediction probability is determined as the target recognition result.

10. The method according to claim 1, characterized in that After fusing the first classification identification information and the second classification identification information to obtain a target identification result, the method further includes: Outputting and displaying the target recognition result; The target recognition result includes one or more of the following information: the category of dangerous behavior; and the probability corresponding to each category of the dangerous behavior.

11. A behavior recognition device, characterized in that: The device comprises: An acquisition unit, used for acquiring multimodal data; the multimodal data includes image data and text data related to the operation task; A processing unit, configured to input the multimodal data into a pre-trained classification recognition model to obtain a classification recognition result; the classification recognition result includes first classification recognition information and second classification recognition information, the classification recognition model includes a first network model and a second network model, the first network model is related to the first classification recognition information, and the second network model is related to the second classification recognition information; A fusion unit is used to fuse the first classification identification information and the second classification identification information to obtain a target recognition result; the target recognition result represents the recognition result of the dangerous behavior in the operation task.

12. A behavior recognition device, characterized in that: include: a processor and a memory for storing a computer program capable of running on said processor; Wherein, when the processor is used to run the computer program, it executes the steps of the method described in any one of claims 1 to 10.

13. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer program product comprising a computer program, characterized in that The computer program implements the steps of the method according to any one of claims 1 to 10 when executed by a processor.

Images