Audio and video stream data processing method, device, electronic device and storage medium

Abstract

The present application provides an audio and video stream data processing method, device, electronic device and storage medium. The electronic device obtains audio and video stream data including audio data and video data, obtains target audio feature information using the audio data, obtains target video feature information using the video data, and then obtains the target information density of the audio and video stream data based on the target audio feature information and the target video feature information, so that the electronic device can timely and comprehensively grasp the security-related situations in the audio and video stream data based on the target information density, so as to ensure the accuracy of the security operations performed by the electronic device based on the target information density.

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a method, device, electronic device and storage medium for processing audio and video stream data. Background Art

[0002] The home security alarm system uses sensors installed in the house to monitor the security status of the living environment and perform corresponding alarm operations when the security status is abnormal. However, most domestic households currently only collect environmental information through cameras installed in fixed locations, and use security equipment to process the environmental information to obtain corresponding security information and perform security operations accordingly.

[0003] In the prior art, there are two situations in which security information is obtained. In one situation, the security equipment uses a single recognition model to synchronously process the environmental information collected by the camera, and after obtaining the corresponding recognition result, it plays it synchronously with the environmental information, resulting in insufficient analysis of the environmental information, and easy neglect of important information, resulting in incorrect corresponding security operations, causing loss of personal and property safety; in the other situation, the security equipment uses different analysis methods to analyze the environmental information stored locally to obtain multi-angle analysis results, but cannot achieve continuous monitoring of the environmental information collected by the camera, resulting in untimely execution of security operations. Summary of the invention

[0004] The present application provides an audio and video stream data processing method, device, electronic device and storage medium to solve the technical problems of untimely execution and low accuracy of security operations.

[0005] In a first aspect, the present application provides a method for processing audio and video stream data, the method comprising:

[0006] Obtaining audio and video stream data; the audio and video stream data includes audio data and video data;

[0007] Using the audio data to obtain target audio feature information, and using the video data to obtain target video feature information;

[0008] Obtaining target information density of audio and video stream data according to target audio feature information and target video feature information;

[0009] Perform corresponding security operations based on the target information density.

[0010] In the above technical solution, the electronic device extracts features of the audio data and video data in the audio and video stream data obtained in real time, respectively, and performs a comprehensive analysis on the above-obtained features to obtain the target information density related to security in the audio and video stream data. This not only enables the electronic device to fully understand the security-related risks in the audio and video stream data through the target information density, but also enables the electronic device to perform corresponding security operations in a timely manner according to the target information density.

[0011] Optionally, obtaining target audio feature information using audio data specifically includes:

[0012] Analyze the sound wave change state of the audio data to determine the abnormal sound state information; the abnormal sound state information includes the abnormal level, abnormal duration and abnormal number;

[0013] Perform voice recognition on the audio data to determine risk voice information; the risk voice information includes risk keywords and / or risk timbre information;

[0014] The target audio feature information includes abnormal sound state information and / or risky voice information.

[0015] Optionally, analyzing the sound wave changes of the audio data to determine the abnormal sound state information specifically includes:

[0016] Calculating the frequency of sound wave amplitude change of the audio data to determine an abnormal sound wave signal whose degree of change of the sound wave amplitude and the frequency of change meet the conditions of abnormal sound state;

[0017] Counting the sound wave signals, determining the abnormal level, abnormal duration and abnormal number of abnormal sound wave signals in the audio and video stream data;

[0018] The abnormal state condition is that the degree of change of the sound wave amplitude is within the preset sound wave amplitude change range, and the change frequency is within the preset frequency change range.

[0019] Optionally, voice recognition is performed on the audio data to determine risky voice information, specifically including:

[0020] Use the speech recognition model to process the audio data and obtain the text information corresponding to the audio data;

[0021] Use preset keywords to perform keyword matching on text information and identify risk keywords.

[0022] Optionally, voice recognition is performed on the audio data to determine risky voice information, specifically including:

[0023] The audio data is processed using a voiceprint model to obtain the user identity information corresponding to the audio data; the voiceprint model is a model trained using the audio data of the target user;

[0024] When the user identity information is not the target user, the audio data is determined as risky timbre information.

[0025] In the above technical solution, the electronic device realizes multi-angle analysis of audio data by analyzing the sound wave changes of audio data, extracting risk keywords, and determining the user identity through voiceprint, thereby ensuring that the electronic device misses important feature information in the process of processing audio and video stream data, and ensuring the accuracy of security operations made by the electronic device based on the information obtained.

[0026] Optionally, obtaining target video feature information using video data specifically includes:

[0027] Performing principal component analysis on the target image in the video data to obtain analysis results, and determining abnormal images in the video data based on the analysis results;

[0028] Use the behavior recognition model to process video data and determine abnormal behavior information in the video data;

[0029] The target video feature information includes abnormal image and / or abnormal behavior information.

[0030] Optionally, performing principal component analysis on the target image in the video data to obtain analysis results, and determining abnormal images in the video data according to the analysis results, specifically includes:

[0031] According to a preset frame number interval, multiple frames of target images are selected from the video data;

[0032] Perform principal component analysis on each target image to obtain the image mean change state of the target images in adjacent frames;

[0033] When the image mean change state meets the preset risk image condition, the target image is determined as an abnormal image.

[0034] Optionally, obtaining a target information density of the audio and video stream data according to the target audio feature information and the target video feature information specifically includes:

[0035] The target audio feature information and the target video feature information are processed using the data information density fitting model to obtain the target information density of the audio and video stream data.

[0036] In the above technical solution, the electronic device analyzes the two types of data, audio data and video data, which are split from the audio and video stream data separately, to prevent security-related features from appearing only in a single type of data, thereby ensuring the comprehensiveness of the electronic device's analysis of the audio and video stream data, thereby improving the accuracy of its related security operations.

[0037] In a second aspect, the present application provides an audio and video stream data processing device, comprising:

[0038] The acquisition module is used to obtain audio and video stream data; the audio and video stream data includes audio data and video data;

[0039] A processing module, used to obtain target audio feature information using audio data, and to obtain target video feature information using video data;

[0040] The processing module is also used to obtain the target information density of the audio and video stream data according to the target audio feature information and the target video feature information;

[0041] The processing module is also used to perform corresponding security operations according to the target information density.

[0042] In a third aspect, the present application provides an electronic device, including: a processor and a memory communicatively connected to the processor;

[0043] Memory stores computer-executable instructions;

[0044] The processor is used to implement the audio and video stream data processing method involved in the first aspect when executing the computer execution instruction.

[0045] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed by a processor, they are used to implement the audio and video stream data processing method involved in the first aspect.

[0046] The present application provides an audio and video stream data processing method, device, electronic device and storage medium. The electronic device obtains audio and video stream data including audio data and video data, obtains target audio feature information using the audio data, obtains target video feature information using the video data, and then obtains the target information density of the audio and video stream data based on the target audio feature information and the target video feature information, so that the electronic device can timely and comprehensively grasp the security-related situations in the audio and video stream data based on the target information density, so as to ensure the accuracy of the security operations performed by the electronic device based on the target information density. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0048] Figure 1 A flowchart of a method for processing audio and video stream data provided by the present application according to an exemplary embodiment;

[0049] Figure 2A flowchart of a method for processing audio and video stream data provided by the present application according to another exemplary embodiment;

[0050] Figure 3 A schematic diagram of an audio and video stream data processing method provided by the present application according to another exemplary embodiment;

[0051] Figure 4 This is a structural diagram of an audio and video stream data processing device provided by the present application according to an embodiment;

[0052] Figure 5 A schematic structural diagram of an electronic device is provided according to an embodiment of the present application.

[0053] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0054] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0055] The present application provides an audio and video stream data processing method, device, electronic device and storage medium, aiming to solve the technical problems of untimely execution and low accuracy of security operations. The technical concept of the present application is: after obtaining the audio and video stream data, the electronic device processes the audio data and video data in the audio and video stream data separately, obtains the characteristic information corresponding to the audio and video stream data from multiple angles, ensures the electronic device's comprehensive analysis of the audio and video stream data, and determines the target information density in the audio and video stream data based on the characteristic information, so that the electronic device can perform security operations in a targeted manner according to the above target information density, and ensures that the electronic device can perform the above security operations in a timely manner when the security status is detected to be abnormal.

[0056] The audio and video stream data processing method provided by the present application can be applied to the scenario of real-time monitoring and analysis of audio and video stream data. In this scenario, the electronic device is the execution subject, and the electronic device can have a built-in audio and video stream data acquisition unit, or can be connected to an audio and video stream data acquisition device, so that the electronic device can obtain and process the original audio and video stream data in real time. Among them, the audio and video stream data acquisition unit or the audio and video stream data acquisition device is used to collect the environmental information of the environment in which it is located in real time, and generate the corresponding original audio and video stream data.

[0057] After obtaining the original audio and video stream data, the electronic device segments the original audio and video stream data at preset time intervals to obtain segmented data, namely, audio and video stream data. Each audio and video stream data contains video data of the same number of frames and audio data corresponding to the video data. The electronic device splits the above audio and video stream data to obtain corresponding audio data and video data. The electronic device analyzes the audio data and video data respectively to obtain corresponding feature information. The electronic device processes the above feature information to obtain the target information density of the audio and video stream data, and performs corresponding security operations according to the target information density, which includes: saving the audio and video stream data whose target information density in the original audio and video stream data is within a preset range, generating corresponding alarm information according to the target information density, and sending the alarm information to the corresponding functional department.

[0058] In one embodiment, the electronic device includes a main processing unit, an audio processing unit and a video processing unit. The main processing unit is connected to the audio processing unit and the video processing unit respectively. After obtaining the original audio and video stream data, the electronic device uses the main processing unit to perform a slice operation on the original audio and video stream data to obtain slice data, that is, audio and video stream data. The main processing unit splits the above data according to the data type to obtain audio data and video data. The main processing unit sends the audio data to the audio processing unit so that the audio processing unit analyzes the sound wave information and audio content in the audio data to determine the security-related information contained in the audio data; the main processing unit also sends the video data to the video processing unit so that the video processing unit performs motion analysis on each frame image in the video data, as well as pixel change analysis between adjacent frames to determine the security-related information in the video data. The main processing unit determines the target information density in the audio and video stream data based on the information obtained by the audio processing unit and the video processing unit, and then performs corresponding security operations based on the target information density.

[0059] Figure 1 FIG. 1 is a flow chart of a method for processing audio and video stream data provided by the present application according to an exemplary embodiment. Figure 1 As shown, the audio and video stream data processing method includes:

[0060] S101. An electronic device obtains audio and video stream data.

[0061] The audio and video stream data is data collected in real time by an audio and video stream data collection unit or an audio and video stream data collection device for the environment information in which it is located. The audio and video stream data collection unit is a unit installed inside an electronic device, and the audio and video stream data collection device is a device external to the electronic device. After collecting the audio and video stream data, the device transmits it to the electronic device in real time according to a preset communication method, so that the electronic device obtains the audio and video stream data in real time.

[0062] The audio and video stream data includes audio data and video data.

[0063] More specifically, the audio data in the audio and video stream data is synchronized with the video data.

[0064] S102: The electronic device obtains target audio feature information using the audio data, and obtains target video feature information using the video data.

[0065] The target audio feature information represents feature information related to the abnormal security state in the audio data, and the target video feature information represents feature information related to the abnormal security state in the video data.

[0066] The security abnormal state is an environmental state that includes security abnormal behavior, for example: an environmental state that includes dangerous actions and / or dangerous words, and an environmental state that includes production safety accidents.

[0067] The electronic device processes the audio data and the video data respectively to obtain corresponding target audio feature information and target video feature information.

[0068] More specifically, the target audio feature information includes abnormal sound state information and / or risky voice information, wherein the abnormal sound state information is information that the waveform of the sound wave signal is abnormal, and the risky voice information is preset sensitive information in the semantic information in the audio data.

[0069] The target video feature information includes abnormal image and / or abnormal behavior information.

[0070] Among them, the abnormal image is one or more frames of images existing in the target video and having pixel features associated with the environmental state of the above-mentioned production safety accident. For example, when the pixel change state of each frame image when a production safety accident occurs is consistent with the preset pixel change state, the image is determined to be an abnormal image.

[0071] S103: The electronic device obtains target information density of the audio and video stream data according to the target audio feature information and the target video feature information.

[0072] The target information density refers to the number of features related to security abnormal events in the audio and video stream data.

[0073] The electronic device processes and counts the target audio feature information and the target video feature information obtained in the audio and video stream data to obtain the target information density in the audio and video stream data.

[0074] S104: The electronic device performs corresponding security operations according to the target information density.

[0075] The electronic device compares the target information density with a preset information density range, determines the security risk level of the event described by the audio and video stream data, and performs corresponding security operations according to the security risk level.

[0076] The security operation includes but is not limited to saving the segmented data in the audio and video stream data whose target information density is within a preset range, generating corresponding alarm information according to the target information density, and sending the alarm information to the corresponding functional department.

[0077] In the above technical solution, the electronic device extracts features of the audio data and video data in the audio and video stream data obtained in real time, respectively, and performs a comprehensive analysis on the above-obtained features to obtain the target information density related to security in the audio and video stream data. This not only enables the electronic device to fully understand the security-related risks in the audio and video stream data through the target information density, but also enables the electronic device to perform corresponding security operations in a timely manner according to the target information density.

[0078] Figure 2 FIG. 1 is a flow chart of a method for processing audio and video stream data according to another exemplary embodiment of the present application. Figure 2 As shown, the method includes:

[0079] S201. An electronic device obtains audio and video stream data.

[0080] The electronic device obtains original audio and video stream data, and performs segmentation processing on the data according to a preset segmentation time interval to obtain multiple audio and video stream data.

[0081] The audio and video stream data have the same time length, including video data with the same number of frames and audio data corresponding to the video data. For example, the duration of multiple audio and video stream data obtained by the electronic device is 1 minute, and each audio and video stream data includes 100 frames of video data.

[0082] After obtaining the audio and video stream data, the electronic device extracts the attribute information of the audio and video stream data, including the start time and end time of the audio data stream data, the storage type of the audio and video stream data, the size of the audio and video stream data, the file offset position of the audio and video stream data in the original audio and video stream data, the marking information of the audio and video stream data, and the marking of key packages and non-key packages.

[0083] In addition, the electronic device splits the audio and video stream data into audio data and video data.

[0084] S202: The electronic device analyzes the sound wave change state of the audio data to determine abnormal sound state information.

[0085] The audio data includes a sound wave signal, and parameters of the sound wave signal include loudness and timbre.

[0086] The sound abnormal state information is information describing the abnormal situation in the sound wave signal, and the sound abnormal state information includes the abnormal level, abnormal duration and abnormal number.

[0087] The electronic device determines the abnormal state of the sound by analyzing the waveform change state of the sound wave signal, and updates the attribute information of the audio and video stream data according to the abnormal state. More specifically, the marking information of the audio and video stream data, as well as the marking of key packages and non-key packages are updated. For example, "sound abnormality" is added to the marking information of the audio and video stream data, and the audio and video stream data is marked as a key package.

[0088] In one embodiment, the electronic device analyzes the change state of the loudness amplitude of the sound wave signal, that is, analyzes the decibel data of the sound wave signal that continuously changes.

[0089] When the decibel data of the sound wave signal is within the preset decibel range, the time period, duration and abnormality level of the sound wave signal that is continuously within the preset decibel range are determined. The electronic device counts the number of occurrences of the above-mentioned duration and determines the number as the number of abnormal sound occurrences in the audio data. More specifically, there is more than one preset decibel range, and the corresponding abnormality level is also more than one.

[0090] For example: when the decibel data of the sound wave signal is greater than 120 decibels between 1ms and 3ms, the sound wave signal in this time period is determined to be the first abnormal sound wave signal, the abnormal level of the sound wave signal is the "dangerous" level, and the abnormal duration is 2ms; when the decibel data of the sound wave signal is greater than 60 decibels and less than or equal to 120 decibels between 7ms and 12ms, the sound wave signal in this time period is determined to be the second abnormal signal, the abnormal level of the sound wave signal is the "abnormality exists" level, and the abnormal duration is 5ms.

[0091] In another embodiment, the electronic device calculates the frequency of change of the sound wave amplitude of the audio data, determines the abnormal sound wave signal whose degree of change of the sound wave amplitude and the change frequency meet the abnormal state condition of the sound, and then counts the sound wave signal to determine the abnormal duration and abnormal number of abnormal sound wave signals in the audio and video stream data. Among them, the abnormal state condition is that the degree of change of the sound wave amplitude is within the preset sound wave amplitude change range, and the change frequency is within the preset frequency change range. For example, when the sound is very small, its sound wave amplitude is not within the preset sound wave amplitude change range, but its change frequency is within the preset change frequency, the electronic device will determine that the event corresponding to the audio data is not dangerous, that is, it is not related to security anomalies, and therefore, the audio signal is not determined as an abnormal sound wave signal.

[0092] S203: The electronic device performs voice recognition on the audio data to determine risky voice information.

[0093] The risk voice information includes risk keywords and / or risk timbre information.

[0094] In one embodiment, the electronic device may determine the risky voice information in the audio data only by the presence of risky keywords in the audio data.

[0095] In the process of determining risk keywords, the electronic device uses the speech recognition model to process the audio data, obtains the text information corresponding to the audio data, and uses the preset keywords to match the text information to determine the risk keywords. The electronic device determines the sentence containing the risk keyword as risk voice information. Among them, the speech recognition model is a trained model stored locally in the electronic device. The risk keywords involved in this embodiment are preset keywords when training the speech recognition model.

[0096] In another embodiment, the electronic device may determine the risky voice information in the audio data only by the presence state of the risky timbre information in the audio data.

[0097] When determining risky voice information, the electronic device uses a voiceprint model to process audio data to obtain user identity information corresponding to the audio data. The voiceprint model is a model trained using the audio data of the target user. In one embodiment, when there is voice information in the audio data that does not correspond to the preset user identity information, the sentence containing the voice information is determined as risky voice information. For example: the target place only allows user A to enter, and the voice information of user A is the voice information corresponding to the preset user identity information. When the electronic device monitors the voice information of other users, it indicates that the target place is in a security abnormality, and the electronic device determines the sentence containing the voice information of other users as risky voice information.

[0098] In another embodiment, the electronic device may simultaneously utilize the existence status of the risk keywords and the risk timbre information in the audio data to jointly determine the risk voice information in the audio data.

[0099] The electronic device uses a speech recognition model to determine the risk keywords present in the audio data, and determines the identity of the user who makes the sound of the risk keywords in the audio data based on the voiceprint model to determine the security risk level of the user, and determines the sentence containing the risk keyword as risk voice information.

[0100] After determining the risky voice information, the electronic device also updates the attribute information of the audio and video stream data according to the risky voice information.

[0101] S204: The electronic device performs principal component analysis on the target image in the video data to obtain an analysis result, and determines an abnormal image in the video data based on the analysis result.

[0102] More specifically, the electronic device selects multiple frames of target images from the video data at preset frame intervals to prevent the influence of single frame image abnormality on the principal component analysis result. The electronic device performs principal component analysis on each target image to obtain the image mean change state of adjacent frame target images. When the image mean change state meets the preset risk image condition, the target image is determined as an abnormal image and the attribute information of the audio and video stream data is updated accordingly.

[0103] For example: when an electronic device performs principal component analysis on multiple frames of target images and determines that the mean value of the red channel image of each target image exceeds a preset number of means, it is determined that there is a risk of fire in the environment in which the audio and video stream data acquisition unit or the audio and video stream data acquisition device is located, and the above target image is determined as an abnormal image.

[0104] S205: The electronic device processes the video data using the behavior recognition model to determine abnormal behavior information in the video data.

[0105] The behavior recognition model is a trained model stored locally in the electronic device and is used to recognize the actions performed by the user in the video.

[0106] More specifically, the electronic device uses the target recognition model to determine the identity information of the user in the video data, and then uses the behavior recognition model to determine the behavior information of each user. The electronic device queries each identity information and the behavior information corresponding to the identity information in the abnormal behavior information mapping table, determines whether the behavior information is abnormal behavior information, and updates the attribute information of the audio and video stream data according to the abnormal behavior information. Among them, the abnormal behavior information mapping table represents the mapping relationship between the identity information and its corresponding preset abnormal behavior information. Different identity information corresponds to different preset abnormal behavior information.

[0107] For example: user A is an elderly person. In the abnormal behavior information mapping table, the preset abnormal behavior information corresponding to user A is wrestling, convulsions, etc.; user B is a young person. In the abnormal behavior information mapping table, the preset abnormal behavior information corresponding to user B is pushing, shoving, convulsions, etc.

[0108] S206: The electronic device processes the target audio feature information and the target video feature information using the data information density fitting model to obtain the target information density of the audio and video stream data.

[0109] The target audio feature information includes the sound abnormal state information obtained in step S202 and / or the risky voice information obtained in step S203. The target video feature information includes the abnormal image obtained in step S204 and / or the abnormal behavior information obtained in step S205.

[0110] The input data of the data information density fitting model is the target audio feature information and the target video feature information. The output data of the model is the target information density, which represents the number of features related to security abnormal events in the audio and video stream data determined by the electronic device based on the above analysis results.

[0111] The data information density fitting model may be a mathematical modeling model or a model trained using a neural network, which is not limited here.

[0112] S207: The electronic device performs corresponding security operations according to the target information density.

[0113] The security operation has been explained in detail in step S104 and will not be repeated here.

[0114] In the above technical solution, the electronic device analyzes the two types of data, audio data and video data, which are split from the audio and video stream data separately, to prevent security-related features from appearing only in a single type of data, thereby ensuring the comprehensiveness of the electronic device's analysis of the audio and video stream data, thereby improving the accuracy of its related security operations.

[0115] The following will be Figure 3 The corresponding embodiments explain the relationship between the above processing steps in detail, wherein: Figure 3 The relationship between the electronic device and the camera shown is only one scenario applicable to the solution proposed in this application.

[0116] After the camera collects the original audio and video stream data containing the information of its environment, it transmits the data to the electronic device. The main processor in the electronic device intercepts the audio and video stream data from it in the order of receiving the original video stream and processes the audio and video stream data. The main processor intercepts the audio and video stream data through a slicing operation. The time length corresponding to each audio and video stream data intercepted by the electronic device is the same, and the number of frames of the video data contained is also the same.

[0117] The main processor divides the obtained audio and video stream data into audio data and video data, and transmits the audio data to the audio processor and transmits the video data to the video processor. The audio processor and the video processor process the obtained data at the same time.

[0118] The audio processor processes audio data including: sound wave analysis, speech recognition processing and voiceprint recognition. The execution order of the above three processing methods can be any order, that is, the audio processor can complete the above three processing processes in sequence or in parallel. The order of the processing processes is not specifically limited here.

[0119] The video processor processes the video data including principal component analysis and behavior recognition. The execution order of the two processing methods can also be any order.

[0120] Among them, the processing of audio data and video data has been Figure 2 The corresponding embodiments are explained in detail and will not be repeated here.

[0121] The audio processor sends the processing results to the main processor, and the video processor sends the processing results to the main processor. The main processor calls the data information density fitting model to analyze all the processing results obtained, obtains the information density of the audio and video stream data, and performs corresponding security operations when the information density is within the preset information density range. The security operations include: storing the current audio and video stream data locally or in a designated storage location, generating alarm information to remind surrounding users, sending alarm information to corresponding functional departments, and sending alarm information to user terminals associated with electronic devices.

[0122] In the above technical scheme, the electronic device obtains audio and video stream data including audio data and video data, obtains target audio feature information using the audio data, obtains target video feature information using the video data, and then obtains the target information density of the audio and video stream data based on the target audio feature information and the target video feature information, so that the electronic device can timely and comprehensively grasp the security-related situations in the audio and video stream data based on the target information density, so as to ensure the accuracy of the security operations performed by the electronic device based on the target information density.

[0123] Figure 4300 is a schematic diagram of the structure of an audio and video stream data processing device according to an embodiment of the present application. The audio and video stream data processing device 300 includes an acquisition module 301 and a processing module 302, wherein:

[0124] The acquisition module 301 is used to obtain audio and video stream data; the audio and video stream data includes audio data and video data.

[0125] The processing module 302 is used to obtain target audio feature information using audio data, and to obtain target video feature information using video data.

[0126] The processing module 302 is further configured to obtain a target information density of the audio and video stream data according to the target audio feature information and the target video feature information.

[0127] The processing module 302 is further configured to execute corresponding security operations according to the target information density.

[0128] In one embodiment, the processing module 302 is specifically used for:

[0129] Analyze the sound wave change state of the audio data to determine the abnormal sound state information; the abnormal sound state information includes the abnormal level, abnormal duration and abnormal number;

[0130] Perform voice recognition on the audio data to determine risk voice information; the risk voice information includes risk keywords and / or risk timbre information;

[0131] The target audio feature information includes abnormal sound state information and / or risky voice information.

[0132] In one embodiment, the processing module 302 is specifically used for:

[0133] Calculating the frequency of sound wave amplitude change of the audio data to determine an abnormal sound wave signal whose degree of change of the sound wave amplitude and the frequency of change meet the conditions of abnormal sound state;

[0134] Counting the sound wave signals, determining the abnormal level, abnormal duration and abnormal number of abnormal sound wave signals in the audio and video stream data;

[0135] The abnormal state condition is that the degree of change of the sound wave amplitude is within the preset sound wave amplitude change range, and the change frequency is within the preset frequency change range.

[0136] In one embodiment, the processing module 302 is specifically used for:

[0137] Use the speech recognition model to process the audio data and obtain the text information corresponding to the audio data;

[0138] Use preset keywords to perform keyword matching on text information and identify risk keywords.

[0139] In one embodiment, the processing module 302 is specifically used for:

[0140] The audio data is processed using a voiceprint model to obtain the user identity information corresponding to the audio data; the voiceprint model is a model trained using the audio data of the target user;

[0141] When the user identity information is not the target user, the audio data is determined as risky timbre information.

[0142] In one embodiment, the processing module 302 is specifically used for:

[0143] Performing principal component analysis on the target image in the video data to obtain analysis results, and determining abnormal images in the video data based on the analysis results;

[0144] Use the behavior recognition model to process video data and determine abnormal behavior information in the video data;

[0145] The target video feature information includes abnormal image and / or abnormal behavior information.

[0146] In one embodiment, the processing module 302 is specifically used for:

[0147] According to a preset frame number interval, multiple frames of target images are selected from the video data;

[0148] Perform principal component analysis on each target image to obtain the image mean change state of the target images in adjacent frames;

[0149] When the image mean change state meets the preset risk image condition, the target image is determined as an abnormal image.

[0150] In one embodiment, the processing module 302 is specifically used for:

[0151] The target audio feature information and the target video feature information are processed using the data information density fitting model to obtain the target information density of the audio and video stream data.

[0152] Figure 5 The present application provides a structural diagram of an electronic device according to an embodiment. Among them, the electronic device 400 includes a memory 401 and a processor 402, and the memory 401 is used to store computer instructions executable by the processor. The memory 401 may include a high-speed random access memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk memory, and can also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0153] When executing computer instructions, the processor 402 implements the various steps in the audio and video stream data processing method with the electronic device as the execution subject in the above-mentioned embodiment. For details, please refer to the relevant description in the above-mentioned method embodiment. The processor 402 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the invention can be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0154] Optionally, the memory 401 can be independent or integrated with the processor 402. When the memory 401 is independently provided, the electronic device 400 further includes a bus for connecting the memory 401 and the processor 402.

[0155] The embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored. When a processor executes the computer instructions, each step of the audio and video stream data processing method in the above embodiment is implemented.

[0156] The embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the various steps in the audio and video stream data processing method in the above embodiment.

[0157] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0158] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for processing audio and video stream data, characterized in that: The method comprises: Obtaining audio and video stream data; the audio and video stream data includes audio data and video data; Using the audio data to obtain target audio feature information, and using the video data to obtain target video feature information; According to the data information density fitting model, the target audio feature information and the target video feature information, the target information density of the audio and video stream data is obtained; wherein the input data of the data information density fitting model is the target audio feature information and the target video feature information, and the output data of the data information density fitting model is the target information density; the target audio feature information includes abnormal sound state information and / or risky voice information; the target video feature information includes abnormal image and / or abnormal behavior information; the target information density represents the number of features related to abnormal security events existing in the audio and video stream data determined by the electronic device according to the analysis result; the analysis result is obtained by performing principal component analysis on the target image in the video data; According to the target information density, corresponding security operations are performed.

2. The method according to claim 1, characterized in that: Obtaining target audio feature information using the audio data specifically includes: Analyze the sound wave change state of the audio data to determine the sound abnormal state information; the sound abnormal state information includes the abnormal level, abnormal duration and abnormal number; Perform voice recognition on the audio data to determine risk voice information; the risk voice information includes risk keywords and / or risk timbre information.

3. The method according to claim 2, characterized in that Analyzing the sound wave changes of the audio data to determine the abnormal sound state information specifically includes: Calculating the frequency of sound wave amplitude change of the audio data to determine an abnormal sound wave signal whose degree of change and frequency of change of the sound wave amplitude meet the condition of abnormal sound state; Counting the sound wave signals to determine the abnormal level, abnormal duration and abnormal number of the abnormal sound wave signals in the audio and video stream data; The abnormal state condition is that the degree of change of the sound wave amplitude is within a preset sound wave amplitude change range, and the change frequency is within a preset frequency change range.

4. The method according to claim 2, characterized in that: Performing voice recognition on the audio data to determine risky voice information specifically includes: Processing the audio data using a speech recognition model to obtain text information corresponding to the audio data; Using preset keywords, keyword matching is performed on the text information to determine risk keywords.

5. The method according to claim 2, characterized in that: Performing voice recognition on the audio data to determine risky voice information specifically includes: Processing the audio data using a voiceprint model to obtain user identity information corresponding to the audio data; the voiceprint model is a model trained using the audio data of the target user; When the user identity information is not the target user, the audio data is determined as risky timbre information.

6. The method according to claim 1, characterized in that Obtaining target video feature information using the video data specifically includes: Performing principal component analysis on the target image in the video data to obtain an analysis result, and determining an abnormal image in the video data based on the analysis result; The video data is processed using a behavior recognition model to determine abnormal behavior information in the video data.

7. The method according to claim 6, characterized in that Performing principal component analysis on the target image in the video data to obtain an analysis result, and determining the abnormal image in the video data according to the analysis result, specifically includes: Selecting a plurality of frames of target images from the video data at a preset frame interval; Performing principal component analysis on each of the target images to obtain image mean value change states of target images of adjacent frames; When the image mean value change state meets the preset risk image condition, the target image is determined as an abnormal image.

8. An audio and video stream data processing device, characterized in that: include: An acquisition module, used to obtain audio and video stream data; the audio and video stream data includes audio data and video data; A processing module, used to obtain target audio feature information using the audio data, and to obtain target video feature information using the video data; The processing module is also used to obtain the target information density of the audio and video stream data according to the data information density fitting model, the target audio feature information and the target video feature information; wherein the input data of the data information density fitting model is the target audio feature information and the target video feature information, and the output data of the data information density fitting model is the target information density; the target audio feature information includes abnormal sound state information and / or risky voice information; the target video feature information includes abnormal image and / or abnormal behavior information; the target information density indicates the number of features related to security abnormal events existing in the audio and video stream data determined by the electronic device according to the analysis result; the analysis result is obtained by performing principal component analysis on the target image in the video data; The processing module is also used to perform corresponding security operations according to the target information density.

9. An electronic device, characterized in that: include: A processor and a memory communicatively connected to the processor; The memory stores computer instructions; When executing the computer instructions, the processor is used to implement the audio and video stream data processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which, when executed by a processor, are used to implement the audio and video stream data processing method as described in any one of claims 1 to 7.

Images