Abnormal behavior identification method, device, electronic device and storage medium

By combining video and audio data, using action, tone, and speech recognition models to identify abnormal behaviors, the problems of high false alarm rate and missed alarm rate in the prior art are solved, and high accuracy abnormal behavior recognition in group scenarios are achieved.

CN118656771BActive Publication Date: 2024-12-06INSPUR SUZHOU INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411147484.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-12-06
Estimated Expiration
2044-08-21

AI Technical Summary

Technical Problem

The prior art has high false alarm rates and missed alarm rates in the recognition of abnormal behaviors, making it difficult to accurately identify abnormal behaviors in group scenarios.

Method used

Multimedia data is used to combine video and audio data, and through the action recognition model, tone recognition model and speech recognition model, object object identification, action recognition results and speech recognition results of the object are obtained, abnormal behavior is determined based on confidence, and alarm information is generated.

Benefits of technology

It improves the accuracy of abnormal behavior recognition, reduces the false alarm rate and false alarm rate, especially in the scenarios where there are many group members.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118656771B_ABST
    Figure CN118656771B_ABST
Patent Text Reader

Abstract

The present application provides an abnormal behavior recognition method, device, electronic device and storage medium, which relates to the field of computer technology, and includes: obtaining multimedia data corresponding to multiple objects; inputting the video data in the multimedia data into an action recognition model to obtain the object identification and action recognition results of each object; inputting the audio data in the multimedia data into a timbre recognition model to obtain the object identification and audio segmentation of each object, and inputting the audio segmentation of each object into a speech recognition model to obtain the speech recognition results of each object; based on the object identification, action recognition results and speech recognition results of each object, determining the abnormal behavior recognition results of the multimedia data. The method and device provided by the present application improve the accuracy of abnormal behavior recognition and effectively reduce the false alarm rate and missed alarm rate.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an abnormal behavior identification method, device, electronic device and storage medium. Background Art

[0002] Abnormal behavior of some members of a group may cause physical and psychological harm to other members. For example, bullying in schools may have adverse effects on students' learning and life, and even on the safety and teaching atmosphere of the campus.

[0003] Related technologies use deep learning technology to identify abnormal behavior based on video or audio. When these methods are used in scenarios where abnormal behavior of a large number of members is to be identified, they usually have high false alarm rates and missed alarm rates.

[0004] Therefore, how to improve the accuracy of abnormal behavior identification has become a technical problem that needs to be solved urgently in the industry. Summary of the invention

[0005] The present application provides an abnormal behavior recognition method, device, electronic device and storage medium, which are used to solve the technical problem of how to improve the accuracy of abnormal behavior recognition.

[0006] The present application provides a method for identifying abnormal behavior, comprising:

[0007] Obtain multimedia data corresponding to multiple objects;

[0008] Inputting the video data in the multimedia data into the action recognition model to obtain the object identification and action recognition results of each object;

[0009] Inputting the audio data in the multimedia data into a timbre recognition model to obtain an object identifier and an audio segment of each object, and inputting the audio segment of each object into a speech recognition model to obtain a speech recognition result of each object;

[0010] Based on the object identification, action recognition result and voice recognition result of each object, an abnormal behavior recognition result of the multimedia data is determined.

[0011] In some embodiments, the step of inputting the audio data in the multimedia data into a timbre recognition model to obtain an object identifier and an audio segment of each object, and inputting the audio segment of each object into a speech recognition model to obtain a speech recognition result of each object includes:

[0012] Performing time-frequency conversion on the audio data in the multimedia data to obtain a frequency domain conversion result corresponding to the audio data;

[0013] Inputting the frequency domain conversion result into the timbre recognition model to obtain object identification of each object, and separating the audio data based on the object identification to obtain audio segments of each object;

[0014] The audio segments of each object are input into the speech recognition model to obtain the speech recognition results of each object.

[0015] In some embodiments, the determining of the abnormal behavior recognition result of the multimedia data based on the object identification, action recognition result and voice recognition result of each object includes:

[0016] When the action recognition result of any object is an abnormal action and the speech recognition result of any object is an abnormal speech, determining that any object has abnormal behavior;

[0017] Based on the object identification of any one of the objects, first warning information is generated.

[0018] In some embodiments, the determining of the abnormal behavior recognition result of the multimedia data based on the object identification, action recognition result and voice recognition result of each object includes:

[0019] When the action recognition result of any object is an abnormal action or the speech recognition result of any object is an abnormal speech, obtaining the confidence of the action recognition result of any object output by the action recognition model and the confidence of the speech recognition result of any object output by the speech recognition model;

[0020] Determining an abnormal behavior determination score of the any object based on the confidence level of the action recognition result of the any object and the confidence level of the speech recognition result of the any object;

[0021] When the abnormal behavior determination score is greater than a preset score threshold, determining that any of the objects has abnormal behavior;

[0022] Based on the object identification of any one of the objects, second warning information is generated.

[0023] In some embodiments, the determining of the abnormal behavior recognition result of the multimedia data based on the object identification, action recognition result and voice recognition result of each object includes:

[0024] When the action recognition result of any object is a normal action and the voice recognition result of any object is a normal voice, it is determined that any object does not have abnormal behavior.

[0025] In some embodiments, after determining the abnormal behavior recognition result of the multimedia data based on the object identification, action recognition result and voice recognition result of each object, the method further includes:

[0026] Determining an object identifier of the target object based on the identity information of the target object;

[0027] Determining a motion recognition result and a speech recognition result of the target object based on the object identifier of the target object and the abnormal behavior recognition result of the multimedia data;

[0028] Based on the action recognition result of the target object, determining the video data corresponding to the target object in the multimedia data;

[0029] Based on the speech recognition result of the target object, determining the audio data corresponding to the target object in the multimedia data;

[0030] Determining a trajectory curve of the target object based on the video data and audio data corresponding to the target object in the multimedia data;

[0031] Based on the trajectory curve and time information of the target object, the video data and audio data corresponding to the target object in the multimedia data are reported.

[0032] In some embodiments, the acquiring multimedia data corresponding to the plurality of objects includes:

[0033] Separating the video and audio in the multimedia data to obtain video data and audio data;

[0034] Based on the time information in the multimedia data, the pictures in the video data and the audio segments in the audio data are aligned.

[0035] In some embodiments, the region where the plurality of objects are located includes a first region and a second region;

[0036] The acquiring of multimedia data corresponding to the multiple objects includes:

[0037] Acquire the video data based on a camera set in the first area;

[0038] The audio data is acquired based on the microphone arranged in the second area.

[0039] In some embodiments, inputting the video data in the multimedia data into the action recognition model includes:

[0040] Dividing the video data in the multimedia data into continuous picture sets;

[0041] Slicing each picture in the picture set, determining a slice corresponding to each picture and time information and position coding information corresponding to the slice;

[0042] The slices corresponding to each image and the time information and position coding information corresponding to the slices are input into the action recognition model.

[0043] In some embodiments, the training methods of the action recognition model, the timbre recognition model and the speech recognition model include independent training and joint training.

[0044] In some embodiments, the action recognition model, the timbre recognition model and the speech recognition model include multiple cascaded feature extraction modules.

[0045] In some embodiments, the feature extraction module includes a first layer normalization submodule, an attention submodule, a second layer normalization submodule, and a fully connected layer submodule connected in sequence;

[0046] The attention submodule includes a plurality of attention feature extraction units arranged in parallel.

[0047] The present application provides an abnormal behavior recognition device, comprising:

[0048] A data acquisition unit, used to acquire multimedia data corresponding to a plurality of objects;

[0049] A video recognition unit, used to input the video data in the multimedia data into an action recognition model to obtain an object identification and action recognition result of each object;

[0050] An audio recognition unit, used for inputting the audio data in the multimedia data into a timbre recognition model to obtain an object identifier and an audio segment of each object, and inputting the audio segment of each object into a speech recognition model to obtain a speech recognition result of each object;

[0051] The behavior recognition unit is used to determine the abnormal behavior recognition result of the multimedia data based on the object identification, action recognition result and voice recognition result of each object.

[0052] The present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the abnormal behavior identification method is implemented when the processor executes the computer program.

[0053] The present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, and the computer program implements the abnormal behavior identification method when executed by a processor.

[0054] The abnormal behavior recognition method, device, electronic device and storage medium provided by the present application obtain multimedia data corresponding to multiple objects; input video data in the multimedia data into an action recognition model to obtain object identification and action recognition results of each object; input audio data in the multimedia data into a timbre recognition model to obtain object identification and audio segmentation of each object, and input audio segmentation of each object into a speech recognition model to obtain speech recognition results of each object; determine abnormal behavior recognition results of the multimedia data based on the object identification, action recognition results and speech recognition results of each object; because the audio data is processed according to the timbre recognition model and the speech recognition model during the recognition process, the speech recognition results of different objects can be effectively determined, thereby improving the accuracy of abnormal behavior recognition using audio data; because the combination of audio data and video data is used for abnormal behavior recognition, the deficiency of using only video data or audio data for recognition can be compensated, and the object identification of the object is simultaneously recognized during the recognition process, the accuracy of abnormal behavior recognition can be improved in scenes with a large number of group members, and the false alarm rate and missed alarm rate are effectively reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0056] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1 This is one of the flow charts of the abnormal behavior identification method provided in this application.

[0058] Figure 2 It is a flowchart of the abnormal behavior secondary analysis method provided by this application.

[0059] Figure 3 It is a schematic diagram of the model architecture provided by this application.

[0060] Figure 4 It is a flowchart of the model training method provided in this application.

[0061] Figure 5 It is a structural diagram of the feature extraction module provided in this application.

[0062] Figure 6 This is the second flow chart of the abnormal behavior identification method provided by this application.

[0063] Figure 7 It is a structural schematic diagram of the abnormal behavior identification device provided in this application.

[0064] Figure 8 It is a structural schematic diagram of the electronic device provided by this application. DETAILED DESCRIPTION

[0065] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0066] It should be noted that the terms "first", "second" etc. in the present application are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0067] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0068] Related technologies are used to identify abnormal behaviors. If video recognition is used, it is necessary to analyze the set rules and single-frame images; if audio recognition is used, it cannot be applied to scenarios where many members are speaking; the false alarm rate and missed alarm rate of the above two methods are relatively high.

[0069] In order to solve the shortcomings of related technologies, Figure 1 This is one of the flow charts of the abnormal behavior identification method provided by this application, such as Figure 1 As shown, the method includes step 110, step 120, step 130 and step 140.

[0070] Step 110: Acquire multimedia data corresponding to multiple objects.

[0071] Specifically, the execution subject of the abnormal behavior identification method provided in the embodiment of the present application is an abnormal behavior identification device. The device can be implemented by software, such as an abnormal behavior identification program running in a computer; or it can be a device that executes the abnormal behavior identification method, such as a mobile terminal, a tablet computer, a desktop computer or a server.

[0072] The applicable scenario of the method provided in the embodiment of the present application is to perform security monitoring on multiple objects and identify abnormal behaviors that may exist in some of the objects. For example, in a campus environment, it is possible to identify bullying behaviors that may exist in some students. Bullying behavior refers to aggressive bullying behavior, such as physical or verbal attacks.

[0073] The object refers to the monitored object, such as students, teachers and staff on campus. Multimedia data refers to files obtained by collecting data on the monitored object in various media forms, such as audio data, video data or audio and video integrated files.

[0074] Devices such as cameras and microphones can be set up in scenes or areas where multiple objects are located. Cameras can be used to capture images of multiple objects to generate video data. Microphones can be used to capture sounds of multiple objects to generate audio data. Cameras equipped with microphones can be used to capture images and sounds of multiple objects at the same time to generate integrated audio and video files.

[0075] Step 120: Input the video data in the multimedia data into the action recognition model to obtain the object identification and action recognition results of each object.

[0076] Specifically, the object identifier refers to an identifier used to distinguish the identity of the object, such as a student number or a teacher number, etc. The action recognition result refers to the specific work identified after analyzing the action of the object in the video data, such as running, jumping or other actions.

[0077] Deep learning technology can be used to build an action recognition model with a neural network as the initial model. The action recognition model is used to identify the object identification and action of each object in the video data. The initial model can select a convolutional neural network, a fully convolutional neural network, a recurrent neural network, or some more complex neural networks, such as a Transformer model, etc. The embodiment of the present application does not specifically limit the selection of the initial model.

[0078] The action recognition model can be obtained by pre-training. Specifically, the action recognition model can be obtained by the following training method: First, video data containing object activities is collected to obtain training samples. Secondly, the object identity and the actions corresponding to each object in the video data are identified, and the object identification and action recognition results of each object are determined to obtain sample labels. Then, the initial model is trained according to the video data to improve the initial model's prediction ability for the object identification and action recognition results of each object in the video data, and finally the action recognition model is obtained.

[0079] Step 130: input the audio data in the multimedia data into a timbre recognition model to obtain an object identifier and an audio segment of each object, and input the audio segment of each object into a speech recognition model to obtain a speech recognition result of each object.

[0080] Specifically, timbre refers to the characteristics of sound, which can be used to distinguish the sounds made by different objects. Audio segmentation refers to the fragments obtained by segmenting audio data. Speech recognition results refer to the speech content obtained after recognizing the speech of an object in an audio segment or audio data, which can be embodied in the form of text.

[0081] Deep learning technology can be used to build a timbre recognition model and a speech recognition model using a neural network as the initial model. The timbre recognition model is used to recognize the timbre of each object in the audio data and obtain the object identification and audio segmentation of each object. The speech recognition model is used to recognize the speech of each object.

[0082] The initial model selection and training process of the timbre recognition model is similar to that of the action recognition model. The difference is that the training samples are the audio of each object, and the sample labels are the timbre identifiers of each object. The timbre identifiers correspond one-to-one to the object identifiers, and the specific training process will not be repeated here.

[0083] The initial model selection and training process of the speech recognition model is similar to that of the action recognition model. The difference is that the training samples are the audio of each object, and the sample labels are the text corresponding to the audio. The specific training process will not be repeated here.

[0084] Step 140: Determine abnormal behavior recognition results of multimedia data based on the object identification, action recognition results, and voice recognition results of each object.

[0085] Specifically, the abnormal behavior identification result may include whether there are objects with abnormal behavior, as well as the object identifiers of these objects and the specific types of abnormal behavior.

[0086] Based on the action recognition results and speech recognition results of each object, it can be determined whether each object has abnormal behavior and the specific type of abnormal behavior. For example, the action recognition results can be matched with different types of abnormal behaviors. If the matches are consistent, the action recognition results are determined to be specific types of abnormal behaviors. The speech recognition results can be matched with abnormal speech keywords corresponding to different types of abnormal behaviors. If the matches are consistent, the speech recognition results are determined to be abnormal speech corresponding to specific types of abnormal behaviors.

[0087] Finally, an abnormal behavior recognition result is generated based on the object with abnormal behavior, the abnormal behavior corresponding to the object, and the object identifier.

[0088] The abnormal behavior recognition method provided by the embodiment of the present application obtains multimedia data corresponding to multiple objects; inputs video data in the multimedia data into an action recognition model to obtain object identification and action recognition results of each object; inputs audio data in the multimedia data into a timbre recognition model to obtain object identification and audio segmentation of each object, and inputs audio segmentation of each object into a speech recognition model to obtain speech recognition results of each object; determines abnormal behavior recognition results of the multimedia data based on the object identification, action recognition results and speech recognition results of each object; because the audio data is processed according to the timbre recognition model and the speech recognition model during the recognition process, the speech recognition results of different objects can be effectively determined, thereby improving the accuracy of abnormal behavior recognition using audio data; because the combination of audio data and video data is used for abnormal behavior recognition, the deficiency of using only video data or audio data for recognition can be compensated, and the object identification of the object is simultaneously recognized during the recognition process, the accuracy of abnormal behavior recognition can be improved in scenes with a large number of group members, effectively reducing the false alarm rate and the missed alarm rate.

[0089] It should be noted that each implementation method of the present application can be freely combined, the order can be changed, or it can be executed separately, and does not need to rely on or depend on a fixed execution order.

[0090] In some embodiments, inputting audio data in multimedia data into a timbre recognition model to obtain object identification and audio segmentation of each object, and inputting the audio segmentation of each object into a speech recognition model to obtain speech recognition results of each object include:

[0091] Performing time-frequency conversion on the audio data in the multimedia data to obtain a frequency domain conversion result corresponding to the audio data;

[0092] The frequency domain conversion result is input into the timbre recognition model to obtain the object identification of each object, and the audio data is separated based on the object identification to obtain the audio segmentation of each object;

[0093] The audio segments of each object are input into the speech recognition model to obtain the speech recognition results of each object.

[0094] Specifically, time-frequency conversion refers to the process of converting the original audio signal from the time domain to the frequency domain, which can be used to analyze the spectral characteristics of the speech signal.

[0095] Audio data in multimedia data is usually stored in the form of time series, and can be converted to frequency domain by Fourier transform or Mel frequency cepstrum coefficients to obtain frequency domain conversion results, such as spectrograms. The frequency domain conversion results are input into the timbre recognition model, which extracts features for classification or regression, and outputs the object identifier of each object, as well as the start and end time of the segment corresponding to the object identifier in the audio data. The audio data is separated according to the object identifier to obtain audio segments of each object.

[0096] The audio segments of each object are respectively input into the speech recognition model, and the speech recognition model recognizes the audio segments of each object to obtain the speech recognition results of each object.

[0097] The abnormal behavior identification method provided in the embodiment of the present application first separates the audio files of different persons according to the timbre recognition model, and performs speech recognition on the separated files, which can effectively determine the meaning of the voices of different persons and improve the accuracy of abnormal behavior identification using audio data.

[0098] In some embodiments, based on the object identification, action recognition result and voice recognition result of each object, determining the abnormal behavior recognition result of the multimedia data includes:

[0099] When the action recognition result of any object is an abnormal action and the speech recognition result of any object is an abnormal speech, determining that any object has abnormal behavior;

[0100] Based on the object identifier of any object, first warning information is generated.

[0101] Specifically, the action recognition result can be matched with the preset abnormal action recognition result to determine whether the action recognition result of any object is an abnormal action. The voice recognition result can be matched with the preset abnormal voice recognition result to determine whether the voice recognition result of any object is an abnormal voice.

[0102] If the action recognition result of any object can be determined as an abnormal action and the voice recognition result of the object can be determined as an abnormal voice according to the abnormal behavior recognition result, it can be determined that the object has abnormal behavior. A first alarm information can be generated according to the object identification of the object for prompting and alarming relevant personnel.

[0103] The abnormal behavior recognition method provided in the embodiment of the present application determines that an object has abnormal behavior when the action recognition result of any object is an abnormal action and the speech recognition result of the object is an abnormal speech, thereby improving the accuracy of abnormal behavior recognition.

[0104] In some embodiments, based on the object identification, action recognition result and voice recognition result of each object, determining the abnormal behavior recognition result of the multimedia data includes:

[0105] When the action recognition result of any object is an abnormal action or the speech recognition result of any object is an abnormal speech, obtain the confidence of the action recognition result of any object output by the action recognition model and the confidence of the speech recognition result of any object output by the speech recognition model;

[0106] Determining an abnormal behavior determination score for any object based on the confidence level of the action recognition result of any object and the confidence level of the speech recognition result of any object;

[0107] When the abnormal behavior determination score is greater than a preset score threshold, it is determined that any object has abnormal behavior;

[0108] Based on the object identification of any object, second warning information is generated.

[0109] Specifically, when the action recognition result of any object is an abnormal action or the voice recognition result of any object is an abnormal voice, it is impossible to directly determine whether abnormal behavior exists based on the action recognition result and the voice recognition result of the same object at the same time.

[0110] Confidence is a measure of how confident a model is in its predictions. Confidence is usually expressed as a probability value, reflecting how certain the model is about a prediction. For example, in speech recognition, confidence can represent the model's confidence level in the recognized word or sentence, usually ranging from 0 (no confidence) to 1 (complete confidence). A higher confidence value means that the model believes the prediction is more accurate.

[0111] For any object, the confidence of the action recognition result of the object output by the action recognition model and the confidence of the speech recognition result of the object output by the speech recognition model can be obtained. The abnormal behavior judgment score of the object can be determined by weighted summation, which is expressed as follows:

[0112]

[0113] In the formula, To determine the score for abnormal behavior, The confidence level of the speech recognition result of the object output by the speech recognition model. is the confidence of the action recognition result of the object output by the action recognition model, and They are the weighting coefficients of speech recognition results and action recognition results respectively.

[0114] It is understandable that the higher the abnormal behavior determination score, the higher the possibility that the object has abnormal behavior; the lower the abnormal behavior determination score, the lower the possibility that the object has abnormal behavior. A preset score threshold can be set, and when the abnormal behavior determination score is greater than the preset score threshold, it is determined that the object has abnormal behavior.

[0115] When it is determined that the object has abnormal behavior, second alarm information is generated according to the object identifier of the object, which is used to prompt and alarm relevant personnel.

[0116] The abnormal behavior recognition method provided in the embodiment of the present application determines the abnormal behavior judgment score of any object based on the confidence level of the action recognition result of the object and the confidence level of the speech recognition result of the object, and then determines that the object has abnormal behavior, thereby improving the accuracy of abnormal behavior recognition.

[0117] In some embodiments, based on the object identification, action recognition result and voice recognition result of each object, determining the abnormal behavior recognition result of the multimedia data includes:

[0118] When the action recognition result of any object is a normal action and the voice recognition result of any object is a normal voice, it is determined that any object does not have abnormal behavior.

[0119] Specifically, if the action recognition result of any object is a normal action and the voice recognition result of any object is a normal voice, it can be determined that the object does not have abnormal behavior.

[0120] The abnormal behavior recognition method provided in the embodiment of the present application jointly determines whether the object has abnormal behavior through action recognition results and voice recognition results, thereby improving the accuracy of abnormal behavior recognition.

[0121] In some embodiments, after determining the abnormal behavior recognition result of the multimedia data based on the object identification, action recognition result, and voice recognition result of each object, the method further includes:

[0122] Determine an object identifier of the target object based on the identity information of the target object;

[0123] Determine the action recognition result and the speech recognition result of the target object based on the object identification of the target object and the abnormal behavior recognition result of the multimedia data;

[0124] Based on the action recognition result of the target object, determine the video data corresponding to the target object in the multimedia data;

[0125] Based on the speech recognition result of the target object, determining the audio data corresponding to the target object in the multimedia data;

[0126] Determine a trajectory curve of the target object based on the video data and audio data corresponding to the target object in the multimedia data;

[0127] Based on the trajectory curve and time information of the target object, the video data and audio data corresponding to the target object in the multimedia data are reported.

[0128] Specifically, the method provided in the embodiment of the present application can also bind identity information with object identification to monitor the behavior trajectory of each object.

[0129] The target object is the object that needs to be monitored. The object identification of the target object can be determined based on the identity information of the target object. According to the object identification of the target object, the abnormal behavior recognition results of the multimedia data can be searched to determine the action recognition results and speech recognition results of the target object.

[0130] According to the target object's action recognition result, the corresponding video data of the target object in the multimedia data is determined, and according to the target object's voice recognition result, the corresponding audio data of the target object in the multimedia data is determined. These video data and audio data are arranged in chronological order, and the trajectory curve of the target object can be determined according to the position of the camera corresponding to each video data and the position of the microphone corresponding to each audio data.

[0131] According to the trajectory curve and time information of the target object, the video data and audio data corresponding to the target object in the multimedia data can be sorted and reported for secondary analysis by relevant personnel.

[0132] Figure 2 is a flow chart of the abnormal behavior secondary analysis method provided by this application, such as Figure 2 As shown, the method includes:

[0133] Step 1: Obtain input information. Preferably, the input information includes the picture information, voice information and corresponding name of the school personnel.

[0134] Step 2: Input the information into the security detection system, and obtain the name information of the person ID (object identifier) ​​in the picture corresponding to the sound through the timbre recognition model and the action recognition model, so that all information is organically unified. Preferably, based on this ID information, when new video or audio data is input into the system, the name corresponding to the person ID can be identified at the same time.

[0135] Step 3: Get the personnel ID.

[0136] Step 4: Get the personnel ID through the timbre, and determine whether the personnel is a reported person through the personnel ID. If not, the person is classified as a key person. If so, determine whether the person is a key person according to the personnel situation. If the person is a key person, proceed to the next step, otherwise proceed to step 11;

[0137] Step 5: Focus on monitoring this person.

[0138] Step 6: Identify and save the monitoring track and action of the ID under the camera.

[0139] Step 7: Identify and save the monitoring track and audio of the ID under the microphone.

[0140] Step 8: Draw the trajectory curve of the person.

[0141] Step 9: According to the time information and trajectory curve, the video and audio data of the person are reported to the monitoring room at the same time.

[0142] Step 10: Manually conduct secondary analysis on the suspicious video and audio data of key personnel reported.

[0143] Step 11: If the person is not a key personnel, directly enter the security detection system for reasoning analysis.

[0144] Step 12: The program ends.

[0145] The abnormal behavior recognition method provided in the embodiment of the present application determines the object identification based on the identity information, can completely characterize the trajectory curve of each object, and can upload the corresponding video and audio information to the monitoring room, which is convenient for secondary determination of suspicious video and audio, and improves the efficiency of abnormal behavior recognition and monitoring.

[0146] In some embodiments, obtaining multimedia data corresponding to a plurality of objects includes:

[0147] Separating the video and audio in the multimedia data to obtain video data and audio data;

[0148] Based on the time information in the multimedia data, the pictures in the video data and the audio segments in the audio data are aligned.

[0149] Specifically, when the multimedia data is an integrated audio and video file, the video and audio in the multimedia data can be separated to obtain video data and audio data, and each picture in the video data and each audio segment in the audio data can be aligned according to the time information in the multimedia data.

[0150] The abnormal behavior recognition method provided in the embodiment of the present application aligns the pictures in the video data and the audio segments in the audio data according to the time information in the multimedia data, which is conducive to accurately identifying abnormal actions or abnormal voices and improves the accuracy of abnormal behavior recognition.

[0151] In some embodiments, the region where the plurality of objects are located includes a first region and a second region;

[0152] Get multimedia data corresponding to multiple objects, including:

[0153] Acquire video data based on a camera set in the first area;

[0154] Audio data is acquired based on the microphone set in the second area.

[0155] Specifically, the area where the multiple objects are located can be divided into a first area and a second area. The first area is an area where the use of a camera for video acquisition is allowed, such as some public areas, etc. The second area is an area where the use of a camera for video acquisition is not allowed, such as some public areas where image acquisition is not convenient for privacy reasons, etc.

[0156] A camera may be arranged in the first area to obtain video data recording the actions of each object, and a microphone may be arranged in the second area to obtain audio data containing the voices emitted by each object.

[0157] The abnormal behavior identification method provided in the embodiment of the present application obtains different types of multimedia data for different areas, and detects and identifies abnormal behaviors by combining video and audio, which effectively avoids monitoring of areas where cameras cannot be installed, can monitor each object more comprehensively, and improves the accuracy of abnormal behavior identification.

[0158] In some embodiments, inputting video data in multimedia data into an action recognition model includes:

[0159] Dividing video data in multimedia data into continuous picture sets;

[0160] Slice each picture in the picture set, determine the slice corresponding to each picture and the time information and position coding information corresponding to the slice;

[0161] The slices corresponding to each image and the time information and position encoding information corresponding to the slices are input into the action recognition model.

[0162] Specifically, the video data in the multimedia data is divided into a continuous set of pictures according to the time information. Large images may contain too many details, making processing and calculation difficult. Cutting the pictures into smaller blocks (slices) allows the model to process the images more efficiently.

[0163] Each picture in the picture set is sliced, the slice corresponding to each picture is determined, and time information and position coding information are added to each slice. The position coding information is used to represent the position of the slice in the corresponding picture. The slices corresponding to each picture and the time information and position coding information corresponding to the slices are input into the action recognition model, so that the action recognition model can perform local learning in different picture areas, which is conducive to capturing more local features.

[0164] The abnormal behavior recognition method provided in the embodiment of the present application inputs the slices corresponding to each image and the time information and position coding information corresponding to the slices into the action recognition model, so that the action recognition model can learn more local features and avoid the loss of information in the image.

[0165] In some embodiments, the training methods of the action recognition model, the timbre recognition model and the speech recognition model include independent training and joint training.

[0166] Specifically, the action recognition model, timbre recognition model and speech recognition model in the present application can be trained in two ways. One is independent training. The training methods of each model have been described in the above embodiments and will not be repeated here. The other is joint training, that is, training these models simultaneously in one training process, and these models work together on the abnormal behavior recognition task.

[0167] Figure 3 This is a schematic diagram of the model architecture provided by this application, such as Figure 3 As shown, the method provided in this application uses a motion recognition model, a timbre recognition model and a speech recognition model.

[0168] The three models are set up independently and can be trained independently during training to improve the detection accuracy of each model; they can also be trained jointly to increase the training efficiency of the model, link information, and improve the overall recognition efficiency.

[0169] In order to improve the overall performance of the three models, the loss function for joint training of the three models can be determined, which can be expressed as:

[0170]

[0171] In the formula, is the joint training loss function, is the training loss function of the action recognition model, is the training weight coefficient of the action recognition model; is the training loss function of the timbre recognition model, is the training weight coefficient of the timbre recognition model; is the training loss function of the speech recognition model, is the training weight coefficient of the speech recognition model.

[0172] It is understandable that the larger the training weight, the greater the impact of a single model on the overall performance. The training weights of each model can be adjusted according to environmental factors. For example, when the lighting conditions are sufficient or the environmental noise is low, you can set Greater than , Greater than .

[0173] By adjusting the training weight coefficient, the joint training loss function constrains the joint training of the three models, thereby improving the overall performance of the method provided in the present application and improving the accuracy of abnormal behavior identification.

[0174] Figure 4 It is a flowchart of the model training method provided in this application, such as Figure 4 As shown, the method includes:

[0175] Step 1: Get input information.

[0176] Step 2: Separate the audio information and the video information.

[0177] Preferably, the audio is divided into different audio segments according to the time period, and the video is divided into a continuous picture set according to the time period. Preferably, the audio segment adopts a time of 10s (seconds), and the video is divided into a picture set of continuous pictures corresponding to 10s; further, the pictures corresponding to 10s are generally 250 pictures, and to simplify the calculation, 50 pictures are equally spaced as the picture set.

[0178] Step 3: Divide the separated audio segment into two parts and process them simultaneously, one of which enters the speech recognition model.

[0179] Step 4: Train the speech entering the speech recognition model to obtain the text information corresponding to the audio.

[0180] Step 5: Convert another part of the audio segment from time domain information to frequency domain information to facilitate the distinction of different timbres.

[0181] Step 6: Input the converted audio data into the timbre recognition model for training.

[0182] Step 7: The corresponding person ID and the audio data separated from the ID can be identified through the timbre recognition model. Preferably, different person IDs are determined according to different voices, and when a new timbre appears, it is marked as a new ID to ensure that each person ID corresponds to its own voice.

[0183] Step 8: Input the separated image set into the action recognition model for training.

[0184] Step 9: Through the action recognition model, the corresponding person ID and action information can be obtained.

[0185] Preferably, the detection error of the model is reduced. Here, the person ID is the corresponding person ID information and action information in the picture set, which is bound to the specific person name information, so that the fixed person information can be accurately identified.

[0186] The abnormal behavior recognition method provided in the embodiment of the present application includes a model training method including independent training and joint training, which can not only increase the training efficiency of the model, but also link information to improve the detection efficiency of the model. For example, the tone recognition model and the action recognition model both contain personnel ID information, and the output of the model can be verified by the ID information detected by the two models, thereby improving the accuracy of the model.

[0187] In some embodiments, the action recognition model, the timbre recognition model, and the speech recognition model include multiple cascaded feature extraction modules.

[0188] Specifically, the action recognition model, the timbre recognition model and the speech recognition model can adopt the same model structure. In the adopted model structure, multiple cascaded feature extraction modules can be adopted. The number of feature extraction modules in each model is different.

[0189] The abnormal behavior identification method provided in the embodiment of the present application cascades multiple feature extraction modules, so that the model can capture more complex and diverse features, thereby improving the feature extraction capability of input data and improving the performance of the model.

[0190] In some embodiments, the feature extraction module includes a first layer normalization submodule, an attention submodule, a second layer normalization submodule, and a fully connected layer submodule connected in sequence;

[0191] The attention submodule includes multiple attention feature extraction units arranged in parallel.

[0192] Specifically, Figure 5 It is a structural diagram of the feature extraction module provided by this application, such as Figure 5 As shown, the feature extraction module includes a first-layer normalization submodule, an attention submodule, a second-layer normalization submodule and a fully connected layer submodule which are connected in sequence, wherein a plurality of attention feature extraction units are arranged in parallel in the attention submodule.

[0193] The feature extraction module processes input data in the following ways:

[0194] Step 1: Obtain input features. Input features are determined based on input data, which is divided into audio data and video data.

[0195] Preferably, the audio is divided into different audio segments according to the time period, and the video is divided into a continuous picture set according to the time period. The audio segment adopts a time of 10s, and the video is divided into a picture set of continuous pictures corresponding to 10s. Furthermore, the pictures corresponding to 10s are generally 250 pictures. To simplify the calculation, 50 pictures are equally spaced as the picture set.

[0196] Preferably, the image is scaled to 448 448 (length width), slice the image, each slice size is 28 28 (length Normalize the audio data and add time and position coding information.

[0197] Step 2: Input the input features into the first layer normalization submodule for layer normalization operation.

[0198] Step 3: Input the input features into the attention submodule.

[0199] Where M is the number of attention feature extraction units in the attention submodule, which are arranged in parallel. Each attention feature extraction unit has the same dimension but different parameters.

[0200] Divide the input features into three identical groups of V (value vector), Q (query vector), and K (key vector). Perform matrix multiplication on Q and K. Since the shard sizes are the same, matrix transposition is not required.

[0201] Scale the results of Q and K. Scaling here uses division by a specific value, which is mainly used to prevent the dot product value from being too large. When calculating the dot product, if scaling is not performed, the dot product value may become very large as the vector dimension increases. This will cause the gradient of the softmax function to be very small, which in turn causes the gradient to disappear, affecting the training effect of the model; stable gradient: The scaling operation makes the dot product result more stable, ensuring that the softmax function works within a more appropriate range, thereby ensuring the training stability of the model. The result value is passed through the activation function softmax. The eigenvalue after softmax is matrix multiplied with V to obtain the output attention feature.

[0202] The whole process can be expressed by the formula:

[0203]

[0204] In the formula, is the output attention feature, is the dimension value of the key vector K.

[0205] Step 4: Add the output and input features obtained by the attention submodule to obtain new features;

[0206] Step 5: Input the new features into the second layer normalization submodule for layer normalization;

[0207] Step 6: Input the normalized features into the fully connected layer submodule.

[0208] Among them, the fully connected layer is set to two layers, and the activation function GELU and the anti-overfitting parameter dropout are set.

[0209] Step 7: Add the features after the fully connected layer and the new features.

[0210] Step 8: Get output features.

[0211] The output features are different according to the different settings of each model. For example, the speech recognition model corresponds to the text corresponding to the audio, the timbre recognition model corresponds to the person ID and the separated audio segment, and the action recognition model corresponds to the person ID and action information. In addition, to ensure the consistency of the output features and the input features, the blank information is filled with special symbols.

[0212] In the abnormal behavior identification method provided in the embodiment of the present application, the feature extraction module adopts a unified model structure to process video and audio information, simplifies the design of the model, fills the blank information in the output information with special symbols, facilitates the model cascading, and improves the detection effect of the model.

[0213] Figure 6 This is the second flow chart of the abnormal behavior identification method provided by this application. Figure 6 As shown, the method includes:

[0214] Step 1: Get input information.

[0215] Step 2: Determine whether the audio and video information are included at the same time. If yes, proceed to the next step; otherwise, proceed to step 14.

[0216] Step 3: Separate the audio information and the video information.

[0217] Preferably, the audio is divided into different audio segments according to the time period, and the video is divided into a continuous picture set according to the time period. Preferably, the audio segment adopts a time of 10s, and the video is divided into a picture set of continuous pictures corresponding to 10s. Furthermore, the pictures corresponding to 10s are generally 250 pictures. To simplify the calculation, 50 pictures are equally spaced as the picture set.

[0218] Step 4: Align the audio segment and the image set based on the time information.

[0219] Step 5: Convert the audio segment from time domain information to frequency domain information to distinguish different timbres.

[0220] Step 6: Input the converted audio data into the timbre recognition model to obtain the corresponding person ID.

[0221] Step 7: Separate all corresponding audio files according to the corresponding personnel ID.

[0222] Preferably, when a new person ID appears, the ID is marked as a key person.

[0223] Step 8: Input the separated audio file into the speech recognition model to obtain the text information corresponding to the audio.

[0224] Step nine: Check whether the corresponding text information contains sensitive words.

[0225] Step 10: Input the obtained picture set into the action recognition model.

[0226] Step 11: Through the action recognition model, the corresponding person ID and action information can be obtained.

[0227] Step 12: Detect whether the corresponding action information is a sensitive action.

[0228] Step 13: Further classification is performed based on the determinations of Steps 9 and 12:

[0229] If both are judged as yes, the alarm information will be reported directly;

[0230] If only one judgment is yes, the judgment score is determined based on the confidence level of the speech recognition result detected by the speech recognition model and the confidence level of the action recognition result detected by the action recognition model. If the judgment score is greater than the set threshold, an alarm is issued, otherwise no alarm is issued;

[0231] If both are judged as no, no alarm will be issued.

[0232] Step 14: If the input information only contains audio information, continue to execute; if it only contains video information, execute step 20.

[0233] Step 15: Convert the audio segment from time domain information to frequency domain information to facilitate the distinction of different timbres.

[0234] Step 16: Input the converted audio data into the timbre recognition model to obtain the corresponding person ID.

[0235] Step 17: Separate all corresponding audio files according to the corresponding personnel ID. Preferably, when a new personnel ID appears, the ID is marked as a key personnel.

[0236] Step 18: Input the separated audio file into the speech recognition model to obtain the text information corresponding to the audio;

[0237] Step 19: Detect whether the corresponding text information contains abnormal keywords (such as certain sensitive words); if so, issue an alarm, otherwise, do not issue an alarm.

[0238] Step 20: Divide the obtained video information into picture sets, and input the picture sets into the action recognition model.

[0239] Step 21: Through the action recognition model, the corresponding person ID and action information can be obtained.

[0240] Step 22: Detect whether the corresponding action information is an abnormal action (such as some sensitive actions); if so, issue an alarm, otherwise, do not issue an alarm.

[0241] Step 23: The program ends.

[0242] The device provided in the embodiment of the present application is described below. The device described below and the method described above can refer to each other.

[0243] Figure 7 is a schematic diagram of the structure of the abnormal behavior identification device provided by the present application, such as Figure 7 As shown, the device comprises:

[0244] A data acquisition unit 710 is used to acquire multimedia data corresponding to a plurality of objects;

[0245] The video recognition unit 720 is used to input the video data in the multimedia data into the action recognition model to obtain the object identification and action recognition results of each object;

[0246] The audio recognition unit 730 is used to input the audio data in the multimedia data into the timbre recognition model to obtain the object identification and audio segmentation of each object, and input the audio segmentation of each object into the speech recognition model to obtain the speech recognition result of each object;

[0247] The behavior recognition unit 740 is used to determine the abnormal behavior recognition result of the multimedia data based on the object identification, action recognition result and voice recognition result of each object.

[0248] The abnormal behavior recognition device provided by the embodiment of the present application obtains multimedia data corresponding to multiple objects; inputs video data in the multimedia data into an action recognition model to obtain object identification and action recognition results of each object; inputs audio data in the multimedia data into a timbre recognition model to obtain object identification and audio segmentation of each object, and inputs audio segmentation of each object into a speech recognition model to obtain speech recognition results of each object; determines abnormal behavior recognition results of the multimedia data based on the object identification, action recognition results and speech recognition results of each object; because the audio data is processed according to the timbre recognition model and the speech recognition model during the recognition process, the speech recognition results of different objects can be effectively determined, thereby improving the accuracy of abnormal behavior recognition using audio data; because the combination of audio data and video data is used for abnormal behavior recognition, it can make up for the shortcomings of using only video data or audio data for recognition, and the object identification of the object is simultaneously recognized during the recognition process, which can improve the accuracy of abnormal behavior recognition in scenes with a large number of group members, and effectively reduce the false alarm rate and missed alarm rate.

[0249] In some embodiments, the audio recognition unit is used to:

[0250] Performing time-frequency conversion on the audio data in the multimedia data to obtain a frequency domain conversion result corresponding to the audio data;

[0251] The frequency domain conversion result is input into the timbre recognition model to obtain the object identification of each object, and the audio data is separated based on the object identification to obtain the audio segmentation of each object;

[0252] The audio segments of each object are input into the speech recognition model to obtain the speech recognition results of each object.

[0253] In some embodiments, the behavior recognition unit is used to:

[0254] When the action recognition result of any object is an abnormal action and the speech recognition result of any object is an abnormal speech, determining that any object has abnormal behavior;

[0255] Based on the object identifier of any object, first warning information is generated.

[0256] In some embodiments, the behavior recognition unit is used to:

[0257] When the action recognition result of any object is an abnormal action or the speech recognition result of any object is an abnormal speech, obtain the confidence of the action recognition result of any object output by the action recognition model and the confidence of the speech recognition result of any object output by the speech recognition model;

[0258] Determine an abnormal behavior determination score for any object based on the confidence level of the action recognition result of any object and the confidence level of the speech recognition result of any object;

[0259] When the abnormal behavior determination score is greater than a preset score threshold, it is determined that any object has abnormal behavior;

[0260] Based on the object identifier of any object, second warning information is generated.

[0261] In some embodiments, the behavior recognition unit is used to:

[0262] When the action recognition result of any object is a normal action and the voice recognition result of any object is a normal voice, it is determined that any object does not have abnormal behavior.

[0263] In some embodiments, the behavior recognition unit is further configured to:

[0264] Determine an object identifier of the target object based on the identity information of the target object;

[0265] Determine the action recognition result and the speech recognition result of the target object based on the object identification of the target object and the abnormal behavior recognition result of the multimedia data;

[0266] Based on the action recognition result of the target object, determine the video data corresponding to the target object in the multimedia data;

[0267] Based on the speech recognition result of the target object, determining the audio data corresponding to the target object in the multimedia data;

[0268] Determine a trajectory curve of the target object based on the video data and audio data corresponding to the target object in the multimedia data;

[0269] Based on the trajectory curve and time information of the target object, the video data and audio data corresponding to the target object in the multimedia data are reported.

[0270] In some embodiments, the data acquisition unit is used to:

[0271] Separating the video and audio in the multimedia data to obtain video data and audio data;

[0272] Based on the time information in the multimedia data, the pictures in the video data and the audio segments in the audio data are aligned.

[0273] In some embodiments, the region where the plurality of objects are located includes a first region and a second region;

[0274] The data acquisition unit is used to:

[0275] Acquire video data based on a camera set in the first area;

[0276] Audio data is acquired based on the microphone set in the second area.

[0277] In some embodiments, the video recognition unit is used to:

[0278] Dividing video data in multimedia data into continuous picture sets;

[0279] Slice each picture in the picture set, determine the slice corresponding to each picture and the time information and position coding information corresponding to the slice;

[0280] The slices corresponding to each image and the time information and position encoding information corresponding to the slices are input into the action recognition model.

[0281] In some embodiments, the training methods of the action recognition model, the timbre recognition model and the speech recognition model include independent training and joint training.

[0282] In some embodiments, the action recognition model, the timbre recognition model, and the speech recognition model include multiple cascaded feature extraction modules.

[0283] In some embodiments, the feature extraction module includes a first layer normalization submodule, an attention submodule, a second layer normalization submodule, and a fully connected layer submodule connected in sequence;

[0284] The attention submodule includes multiple attention feature extraction units arranged in parallel.

[0285] Figure 8 is a schematic diagram of the structure of the electronic device provided by this application, such as Figure 8 As shown, the electronic device may include: a processor (Processor) 810, a communication interface (Communications Interface) 820, a memory (Memory) 830 and a communication bus (Communications Bus) 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic command in the memory 830 to execute the method described in the above embodiment, for example:

[0286] Acquire multimedia data corresponding to multiple objects; input video data in the multimedia data into an action recognition model to obtain object identification and action recognition results of each object; input audio data in the multimedia data into a timbre recognition model to obtain object identification and audio segmentation of each object, and input the audio segmentation of each object into a speech recognition model to obtain speech recognition results of each object; determine abnormal behavior recognition results of the multimedia data based on the object identification, action recognition results and speech recognition results of each object.

[0287] In addition, the logical commands in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several commands to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0288] The processor in the electronic device provided in the embodiment of the present application can call the logic instructions in the memory to implement the above method. Its specific implementation method is consistent with the implementation method of the aforementioned method and can achieve the same beneficial effects, which will not be repeated here.

[0289] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the methods provided in the above embodiments are implemented.

[0290] Its specific implementation is consistent with the aforementioned method implementation and can achieve the same beneficial effects, so it will not be repeated here.

[0291] An embodiment of the present application provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0292] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0293] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0294] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for identifying abnormal behavior, characterized in that: include: Obtain multimedia data corresponding to multiple objects; Inputting the video data in the multimedia data into the action recognition model to obtain the object identification and action recognition results of each object; Inputting the audio data in the multimedia data into a timbre recognition model to obtain an object identifier and an audio segment of each object, and inputting the audio segment of each object into a speech recognition model to obtain a speech recognition result of each object; Determining abnormal behavior recognition results of the multimedia data based on object identifications, action recognition results, and voice recognition results of each object; After determining the abnormal behavior recognition result of the multimedia data based on the object identification, action recognition result and voice recognition result of each object, the method further includes: Determining an object identifier of the target object based on the identity information of the target object; Determining a motion recognition result and a speech recognition result of the target object based on the object identifier of the target object and the abnormal behavior recognition result of the multimedia data; Based on the action recognition result of the target object, determining the video data corresponding to the target object in the multimedia data; Based on the speech recognition result of the target object, determining the audio data corresponding to the target object in the multimedia data; Determining a trajectory curve of the target object based on the video data and audio data corresponding to the target object in the multimedia data; Based on the trajectory curve and time information of the target object, the video data and audio data corresponding to the target object in the multimedia data are reported.

2. The abnormal behavior identification method according to claim 1, characterized in that: The step of inputting the audio data in the multimedia data into a timbre recognition model to obtain an object identifier and an audio segment of each object, and inputting the audio segment of each object into a speech recognition model to obtain a speech recognition result of each object includes: Performing time-frequency conversion on the audio data in the multimedia data to obtain a frequency domain conversion result corresponding to the audio data; Inputting the frequency domain conversion result into the timbre recognition model to obtain object identification of each object, and separating the audio data based on the object identification to obtain audio segments of each object; The audio segments of each object are input into the speech recognition model to obtain the speech recognition results of each object.

3. The abnormal behavior identification method according to claim 1, characterized in that: The determining of the abnormal behavior recognition result of the multimedia data based on the object identification, action recognition result and voice recognition result of each object includes: When the action recognition result of any object is an abnormal action and the speech recognition result of any object is an abnormal speech, determining that any object has abnormal behavior; Based on the object identification of any one of the objects, first warning information is generated.

4. The abnormal behavior identification method according to claim 1, characterized in that: The determining of the abnormal behavior recognition result of the multimedia data based on the object identification, action recognition result and voice recognition result of each object includes: When the action recognition result of any object is an abnormal action or the speech recognition result of any object is an abnormal speech, obtaining the confidence of the action recognition result of any object output by the action recognition model and the confidence of the speech recognition result of any object output by the speech recognition model; Determining an abnormal behavior determination score of the any object based on the confidence level of the action recognition result of the any object and the confidence level of the speech recognition result of the any object; When the abnormal behavior determination score is greater than a preset score threshold, determining that any of the objects has abnormal behavior; Based on the object identification of any one of the objects, second warning information is generated.

5. The abnormal behavior identification method according to claim 1, characterized in that: The determining of the abnormal behavior recognition result of the multimedia data based on the object identification, action recognition result and voice recognition result of each object includes: When the action recognition result of any object is a normal action and the voice recognition result of any object is a normal voice, it is determined that any object does not have abnormal behavior.

6. The abnormal behavior identification method according to claim 1, characterized in that: The acquiring of multimedia data corresponding to the multiple objects includes: Separating the video and audio in the multimedia data to obtain video data and audio data; Based on the time information in the multimedia data, the pictures in the video data and the audio segments in the audio data are aligned.

7. The abnormal behavior identification method according to claim 1, characterized in that: The areas where the multiple objects are located include a first area and a second area; The acquiring of multimedia data corresponding to the multiple objects includes: Acquire the video data based on a camera set in the first area; The audio data is acquired based on the microphone arranged in the second area.

8. The abnormal behavior identification method according to claim 1, characterized in that: The step of inputting the video data in the multimedia data into the action recognition model comprises: Dividing the video data in the multimedia data into continuous picture sets; Slicing each picture in the picture set, determining a slice corresponding to each picture and time information and position coding information corresponding to the slice; The slices corresponding to each image and the time information and position coding information corresponding to the slices are input into the action recognition model.

9. The abnormal behavior identification method according to claim 1, characterized in that: The training methods of the action recognition model, the timbre recognition model and the speech recognition model include independent training and joint training.

10. The abnormal behavior identification method according to any one of claims 1 to 9, characterized in that: The action recognition model, the timbre recognition model and the speech recognition model include a plurality of cascaded feature extraction modules.

11. The abnormal behavior identification method according to claim 10, characterized in that: The feature extraction module includes a first layer normalization submodule, an attention submodule, a second layer normalization submodule and a fully connected layer submodule connected in sequence; The attention submodule includes a plurality of attention feature extraction units arranged in parallel.

12. An abnormal behavior recognition device, characterized in that: include: A data acquisition unit, used to acquire multimedia data corresponding to a plurality of objects; A video recognition unit, used to input the video data in the multimedia data into an action recognition model to obtain an object identification and action recognition result of each object; An audio recognition unit, used for inputting the audio data in the multimedia data into a timbre recognition model to obtain an object identifier and an audio segment of each object, and inputting the audio segment of each object into a speech recognition model to obtain a speech recognition result of each object; A behavior recognition unit, configured to determine an abnormal behavior recognition result of the multimedia data based on an object identifier, an action recognition result, and a voice recognition result of each object; The behavior recognition unit is also used for: Determining an object identifier of the target object based on the identity information of the target object; Determining a motion recognition result and a speech recognition result of the target object based on the object identifier of the target object and the abnormal behavior recognition result of the multimedia data; Based on the action recognition result of the target object, determining the video data corresponding to the target object in the multimedia data; Based on the speech recognition result of the target object, determining the audio data corresponding to the target object in the multimedia data; Determining a trajectory curve of the target object based on the video data and audio data corresponding to the target object in the multimedia data; Based on the trajectory curve and time information of the target object, the video data and audio data corresponding to the target object in the multimedia data are reported.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the abnormal behavior identification method according to any one of claims 1 to 11 is implemented.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the abnormal behavior identification method according to any one of claims 1 to 11 is implemented.