Video recognition method and device, storage medium and electronic device

Abstract

The application discloses a video recognition method and device, a storage medium and an electronic device. The method comprises the following steps: acquiring an object voice feature of an audio frame included in a to-be-recognized video frame; acquiring an object behavior feature of an image frame corresponding to the audio frame; fusing the object voice feature and the object behavior feature to obtain a fusion feature; and identifying whether the to-be-recognized video frame is a target video frame according to the fusion feature, and obtaining an identification result. The application can recognize a video frame.

Description

Technical Field

[0001] This application belongs to the field of electronic technology, and in particular relates to a video recognition method, device, storage medium and electronic device. Background Technology

[0002] With the continuous development of multimedia technology, more and more users are using various recording devices to record videos to document all aspects of their lives or work. For example, teachers use recording devices to record classroom activities, creating corresponding classroom videos. However, taking classroom videos as an example, these videos include key video frames, such as question-and-answer frames and other video frames. These key video frames are more important for teaching research; therefore, it is necessary to identify these video frames. Summary of the Invention

[0003] This application provides a video recognition method, apparatus, storage medium, and electronic device that can recognize video frames.

[0004] In a first aspect, embodiments of this application provide a video recognition method, including:

[0005] Obtain the speech features of the audio frames included in the video frame to be identified;

[0006] Obtain the object behavior features of the image frame corresponding to the audio frame;

[0007] The object's speech features and the object's behavioral features are fused to obtain fused features;

[0008] Based on the fusion features, it is determined whether the video frame to be identified is the target video frame, and the identification result is obtained.

[0009] Secondly, embodiments of this application provide a video recognition device, including:

[0010] The speech feature acquisition module is used to acquire the speech features of the audio frames included in the video frame to be identified.

[0011] The behavior feature acquisition module is used to acquire the object behavior features of the image frame corresponding to the audio frame;

[0012] The feature fusion module is used to fuse the object's speech features and the object's behavior features to obtain fused features;

[0013] The video frame recognition module is used to identify whether the video frame to be identified is the target video frame based on the fusion features, and to obtain the recognition result.

[0014] Thirdly, embodiments of this application provide a storage medium storing a computer program thereon, which, when executed on a computer, causes the computer to perform the video recognition method provided in embodiments of this application.

[0015] Fourthly, embodiments of this application also provide an electronic device, including a memory and a processor, wherein the processor executes the video recognition method provided in embodiments of this application by calling a computer program stored in the memory.

[0016] In this embodiment of the application, by obtaining the object behavior features of the image frames included in the video frame to be identified; obtaining the object speech features of the audio frames corresponding to the image frames; fusing the object behavior features and the object speech features to obtain fused features; and identifying whether the video frame to be identified is a target video frame based on the fused features to obtain the identification result, the identification of video frames can be achieved. Attached Figure Description

[0017] The technical solution and its beneficial effects will become apparent from the following detailed description of specific embodiments of this application, in conjunction with the accompanying drawings.

[0018] Figure 1 This is a flowchart illustrating the video recognition method provided in the embodiments of this application.

[0019] Figure 2 This is a schematic diagram of a scene for the video recognition method provided in the embodiments of this application.

[0020] Figure 3 This is a schematic diagram of the structure of the video recognition device provided in the embodiments of this application.

[0021] Figure 4 This is a schematic diagram of the first structure of the electronic device provided in the embodiments of this application.

[0022] Figure 5 This is a schematic diagram of a second structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0023] It should be noted that the terms "first" and "second," etc., used in this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not limited to the listed steps or modules, but some embodiments also include steps or modules not listed, or some embodiments also include other steps or modules inherent to these processes, methods, products, or devices.

[0024] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0025] This application provides a video recognition method, a video recognition device, a storage medium, and an electronic device. The execution entity of the video recognition method can be the video recognition device provided in this application, or an electronic device integrating the video recognition device. The video recognition device can be implemented in hardware or software. The electronic device can be a smartphone, tablet computer, PDA, laptop computer, or other device equipped with a processor and possessing data processing capabilities.

[0026] Please see Figure 1 , Figure 1 This is a flowchart illustrating the video recognition method provided in an embodiment of this application. The process may include:

[0027] In step 101, the speech features of the audio frames included in the video frame to be identified are obtained.

[0028] The video frame to be identified can be a video frame within a video. The video to be identified can be a video whose included video frames are the target video frames. Target video frames can include key video frames, such as question-and-answer video frames or video frames explaining key content. Different videos may correspond to the same or different target video frames, depending on the specific requirements.

[0029] For example, assuming the video to be identified is a live classroom video, and the actual requirement is to identify the question-and-answer video frames (including student question frames and teacher answer frames) within the live classroom video, then the target video frames can be these question-and-answer video frames. This live classroom video can be obtained by filming the teacher and students' behavior in the classroom using cameras equipped in the classroom, and by recording the teacher and students' voices using recording equipment equipped in the classroom. Alternatively, this live classroom video can be obtained by recording the teacher and students in the classroom using video recording equipment equipped in the classroom.

[0030] The video frame to be identified includes audio frames and image frames. The audio frames included in the same video frame to be identified correspond to the image frames included in it. Audio frames are used to record sound information, and image frames are used to record image information. An audio frame and its corresponding image frame are combined to form a video frame.

[0031] In this embodiment, the speech features of the audio frames included in the video frame to be identified are obtained. These speech features can describe the speaking object in the audio frame, such as the speaker, i.e., who is speaking.

[0032] In step 102, the object behavior characteristics of the image frame corresponding to the audio frame are obtained.

[0033] In this embodiment, the object behavior features of the image frame corresponding to the audio frame are obtained. These object behavior features can describe the objects in the image frame, such as human behavior, such as raising a hand, opening a mouth, looking up, turning a head, standing, etc.

[0034] In 103, the speech features and behavioral features of the object are fused to obtain the fused features.

[0035] In this embodiment, after obtaining the object speech features of the audio frame and the object behavior features of the image frame corresponding to the audio frame in the video frame to be identified, the object speech features and object behavior features are fused to obtain the fused features.

[0036] For example, object speech features and object behavior features can be concatenated to obtain fused features. Alternatively, object speech features and object behavior features can be concatenated end-to-end, such as with the object speech features first and the object behavior features second, or vice versa, to obtain fused features.

[0037] In step 104, based on the fusion features, it is determined whether the video frame to be identified is the target video frame, and the identification result is obtained.

[0038] In this embodiment, after obtaining the fusion feature, the system identifies whether the video frame to be identified is the target video frame based on the fusion feature, thus obtaining an identification result. The identification result includes whether the video frame to be identified is the target video frame or not. When the fusion feature identifies the video frame to be identified as the target video frame, the identification result is "The video frame to be identified is the target video frame." When the fusion feature identifies the video frame to be identified as not the target video frame, the identification result is "The video frame to be identified is not the target video frame."

[0039] Since the speech features describe the speaking object in the video to be identified, and the behavior features describe the behavior of the object in the video to be identified, the fused features obtained by fusing the speech features and behavior features describe the speaking object and the behavior of the object in the video to be identified. By using the information included in the fused features, it is possible to identify whether the video frame to be identified is the target video frame.

[0040] For example, object behavior features describe objects standing in the video to be identified, while object speech features describe the presence of a speaker in the video to be identified. This speaker is not a teacher, and students typically stand when asking questions. Therefore, the video frame to be identified can be determined as a student asking a question. As another example, object behavior features describe objects standing in the video to be identified, while object speech features describe the presence of a speaker in the video to be identified. This speaker is a teacher, and students typically stand when asking questions, and also typically stand when listening to the teacher's response. Therefore, the video frame to be identified can be determined as a teacher responding to questions.

[0041] In this embodiment, by acquiring the object behavior features of the image frames included in the video frame to be identified; acquiring the object speech features of the audio frames corresponding to the image frames; fusing the object behavior features and the object speech features to obtain fused features; and identifying whether the video frame to be identified is a target video frame based on the fused features to obtain the identification result, video frame identification can be achieved.

[0042] In an optional embodiment, based on the fusion features, it is determined whether the video frame to be identified is the target video frame, and an identification result is obtained, including:

[0043] (1) Input the fused features into the video frame classification model to obtain the classification result;

[0044] (2) Based on the classification results, identify whether the video frame to be identified is the target video frame and obtain the identification result.

[0045] In this embodiment, the sample fusion vector of the sample video frames can be used to train the corresponding model to obtain a video frame classification model. After obtaining the fusion features of the video frame to be identified, the fusion features are input into the video frame classification model to obtain the classification result. The classification result includes 0 or 1. Wherein, 0 indicates that the video frame to be identified is not the target video frame, and 1 indicates that the video frame to be identified is the target video frame. Therefore, when the classification result is 0, it can be determined that the identification result is that the video frame to be identified is not the target video frame. When the classification result is 1, it can be determined that the identification result is that the video frame to be identified is the target video frame.

[0046] It should be noted that the video frame classification model can be trained by those skilled in the art based on the actual situation.

[0047] In an optional embodiment, before obtaining the object speech features of the audio frames included in the video frame to be identified, the method further includes:

[0048] (1) Obtain the video to be identified;

[0049] (2) Extract the video frames to be identified from the video to be identified.

[0050] For example, the video to be identified can be divided into multiple video frames according to a preset sampling rate. For example, the preset sampling rate can be 5 frames per second or 10 frames per second, and so on.

[0051] In an optional embodiment, the video frame classification model includes a long short-term memory network model. The fused features are input into the video frame classification model to obtain a classification result, including:

[0052] The fusion features of each video frame to be identified are sequentially input into the Long Short-Term Memory network model to obtain the classification results of each video frame to be identified.

[0053] Based on the classification results, it is determined whether the video frame to be identified is the target video frame, and the identification results are obtained, including:

[0054] Based on the classification results of each video frame to be identified, we determine whether each video frame is the target video frame and obtain the identification result of each video frame to be identified.

[0055] Long Short-Term Memory (LSTM) is a type of recurrent neural network specifically designed to address the long-term dependency problem inherent in general Recurrent Neural Networks (RNNs). An LSTM is a neural network containing LSTM blocks or other similar units, and may also be described as an intelligent network unit because it can remember values ​​of indefinite duration. Each block contains a gate that determines whether the input is important enough to be remembered and whether it can be output.

[0056] For example, assuming 1000 video frames are identified from the video to be identified, the object speech features of the audio frames included in each of the 1000 video frames can be obtained, thus obtaining the object speech features of each of the 1000 video frames to be identified. Simultaneously, the object behavior features of the image frames corresponding to the audio frames included in each of the 1000 video frames to be identified can be obtained, thus obtaining the object behavior features of each of the 1000 video frames to be identified. Then, the object speech features of each of the 1000 video frames to be identified are fused with... The object's behavioral characteristics are used to obtain the fused features of each of the 1000 video frames to be identified. Then, the fused features of each of the 1000 video frames to be identified are sequentially input into a pre-trained long short-term memory network model to obtain the classification results of each of the 1000 video frames to be identified. Based on the classification results of each of the 1000 video frames to be identified, it is determined whether each of the 1000 video frames to be identified is a target video frame, thereby identifying the target video frame from the 1000 video frames to be identified.

[0057] For example, if the video to be identified is 3 seconds long and the preset sampling rate is 5 frames per second, then video frames M1 to M5 can be extracted sequentially from the first second of the video segment (where video frames M1 to M5 are ordered by time as: video frames M1, M2, M3, M4, M5; for example, the video frame at 200 milliseconds is extracted as video frame M1, the video frame at 400 milliseconds as video frame M2, the video frame at 600 milliseconds as video frame M3, the video frame at 800 milliseconds as video frame M4, and the video frame at 1000 milliseconds as video frame M5). Similarly, video frames M6 to M10 can be extracted sequentially from the second second of the video segment (video frames M6 to M10 are ordered by time as: video frames M6, M7, M8, M9, M10; for example, the video frame at 1200 milliseconds is extracted as video frame M5). As video frame M6, the video frame at 1400 milliseconds is extracted as video frame M7, the video frame at 1600 milliseconds is extracted as video frame M8, the video frame at 1800 milliseconds is extracted as video frame M9, and the video frame at 2000 milliseconds is extracted as video frame M10. From the 3-second video segment of the video to be identified, video frames M11 to M15 are extracted sequentially (video frames M11 to M15 are ordered by time as: video frames M11, M12, M13, M14, M15, for example, the video frame at 2200 milliseconds is extracted as video frame M11, the video frame at 2400 milliseconds is extracted as video frame M12, the video frame at 2600 milliseconds is extracted as video frame M13, the video frame at 2800 milliseconds is extracted as video frame M14, and the video frame at 3000 milliseconds is extracted as video frame M15).

[0058] like Figure 2 As shown, the fusion features C1 of video frame M1, C2 of video frame M2, C3 of video frame M3, C4 of video frame M4, C5 of video frame M5, C6 of video frame M6, C7 of video frame M7, C8 of video frame M8, C9 of video frame M9, C10 of video frame M10, C11 of video frame M11, C12 of video frame M12, C13 of video frame M13, C14 of video frame M14, and C15 of video frame M15 can be sequentially input into the Long Short-Term Memory network model to obtain the classification results of video frames M11 to M15. Based on the classification results of video frames M11 to M15, it is determined whether video frames M11 to M15 are target video frames, thus obtaining the recognition results of video frames M11 to M15.

[0059] It should be noted that the long short-term memory network model can be trained by those skilled in the art based on the actual situation.

[0060] In an optional embodiment, after identifying whether each video frame to be identified is a target video frame based on the classification results of each video frame to be identified, and obtaining the identification results of each video frame to be identified, the method further includes:

[0061] The target video is obtained based on the target video frames.

[0062] For example, suppose 1000 video frames to be identified are determined from the video to be identified, and 300 target video frames are determined from these 1000 video frames to be identified. These 300 target video frames can be directly combined into a target video.

[0063] For example, considering that the human eye can perceive a frame rate of at least 24 frames per second without any stuttering, when the preset sampling rate is lower than this frame rate, the corresponding video segments can be edited from the target video frames to obtain the target video. When the preset sampling rate is greater than or equal to this frame rate, the electronic device can directly generate the target video based on the target video frames.

[0064] For example, assuming the preset sampling rate is 5 frames per second, 100 temporally consecutive target video frames are obtained. Assuming that these 100 temporally consecutive target video frames are sampled from the video segment from 10 minutes 20 seconds to 10 minutes 40 seconds in the video to be identified, then the target video can be the video segment from 10 minutes 20 seconds to 10 minutes 40 seconds in the video to be identified.

[0065] For example, assuming the preset sampling rate is 30 frames per second, resulting in 600 temporally consecutive target video frames, these 600 temporally consecutive target video frames can be combined into a target video, thus obtaining a 20-second target video.

[0066] In an optional embodiment, obtaining the object speech features of the audio frames included in the video frame to be identified includes:

[0067] (1) Extract the audio features of the audio frames corresponding to the image frames;

[0068] (2) Perform speaker clustering on the audio features to obtain the clustering results;

[0069] (3) Obtain the object speech features of the audio frame based on the clustering results.

[0070] For example, assuming 1000 video frames are identified from a video to be identified, audio features, such as FBank vectors, can be extracted from the audio frames included in each of these 1000 video frames. Since the number of speakers in the video is uncertain, a spectral clustering algorithm is used to cluster the audio features, such as the FBank vectors, into speakers. Based on the clustering results, the speech features of the audio frames included in each video frame to be identified are obtained, such as the speech vectors of the speakers.

[0071] Specifically, (1) select multiple initial speaking objects (such as 3 speaking objects) as the initial cluster center vector.

[0072] (2) Calculate the distance between the FBank vector of the audio frame included in each video frame to be identified and the initial cluster center vector. For example, calculate the cosine similarity between the FBank vector of the audio frame included in each video frame to be identified and the initial cluster center vector, and classify the FBank vector of the audio frame included in each video frame to be identified into the category corresponding to the cluster center vector with the smallest distance.

[0073] (3) Recalculate the cluster center vector for each category according to the following formula;

[0074]

[0075] Where x represents the FBank vector of the audio frames included in each video frame to be identified, +c i + indicates the total number of FBank vectors for category j, a j This represents the cluster center vector of category j.

[0076] (4) Repeat steps (2) and (3) above until the termination condition is met. The termination condition includes the number of iterations reaching a preset number or the distance between the cluster center vectors of each category obtained in the previous round and the cluster center vectors of each category obtained in the new round being less than a preset distance, etc. The preset number of iterations and the preset distance can be set by those skilled in the art according to actual needs.

[0077] (5) The cluster center vectors of the corresponding categories can be obtained from the above steps. The FBank vector of the audio frame included in each video frame to be identified is assigned to a category, and the cluster center vector of that category is the object speech vector of the audio frame included in the corresponding video frame to be identified. For example, suppose the FBank vector of the audio frame included in a video frame to be identified is assigned to category 1, and the cluster center vector of category 1 is a1. Then the object speech vector of the audio frame included in the video frame to be identified is the cluster center vector a1.

[0078] It is understandable that when the video to be identified is a live video of a class, since the teacher speaks the most during the class, the category with the most FBank vectors will be the one whose audio frames contain the teacher.

[0079] For example, suppose 1000 video frames are identified from the video to be identified, and the FBank vectors of the audio frames included in 800 of these video frames are all classified into category 1. Then, it can be determined that the speaker in the audio frames included in these 800 video frames includes the teacher.

[0080] In an optional embodiment, obtaining the object behavior features of the image frame corresponding to the audio frame includes:

[0081] (1) Input the image frame into the behavior detection model to obtain the detection result;

[0082] (1) Based on the detection results, the object behavior features of the image frame are obtained.

[0083] In this embodiment, an image frame is input into a pre-trained behavior detection model to obtain detection results. Based on the detection results, object behavior features of the image frame, such as object behavior vectors, are obtained. The detection results include multiple key points of each object in the image frame; therefore, the object behavior features of the image frame are vectors composed of multiple key points of each object in the image frame.

[0084] It should be noted that the behavior detection model can be trained by those skilled in the art based on the actual situation, as long as it can detect multiple key points of each object in the image frame.

[0085] In an optional embodiment, the video frame classification model includes a long short-term memory network model. The fused features are input into the video frame classification model to obtain a classification result, including:

[0086] (1) Determine the target object behavior features that meet the preset conditions from the object behavior features of each video frame to be identified in the video to be identified;

[0087] (2) Input the target fusion feature corresponding to the target object behavior feature and the fusion feature after the target fusion feature into the long short-term memory network model in sequence to obtain the classification results of the target video frame to be identified and the target video frame to be identified after the target video frame;

[0088] Based on the classification results, it is determined whether the video frame to be identified is the target video frame, and the identification results are obtained, including:

[0089] Based on the classification results of the target video frame to be identified and the subsequent video frames to be identified, the system identifies whether the target video frame to be identified and the subsequent video frames to be identified are the target video frame, and obtains the identification result.

[0090] For example, taking a classroom video of a lesson as the target video frame and a question-and-answer video frame as the target video frame, considering that students usually make specific actions when asking questions, such as raising their hands, and that question-and-answer video frames must be a series of consecutive video frames in the classroom video, we can start identifying the target video frame from the video frame with the specific behavior. That is to say, the preset conditions include the corresponding specific behavior, such as raising hands. Suppose 1000 video frames to be identified are determined from a classroom video. The 450th frame contains multiple key points corresponding to the action of raising a hand, which are key points for the object's behavior. Based on these key points, the action can be identified as raising a hand. Starting from the 450th frame, the fused vectors of the 450th frame and subsequent frames are input into a Long Short-Term Memory (LSTM) network model to obtain classification results. Based on these classification results, the 450th frame and subsequent frames are then identified as target frames, yielding the identification result.

[0091] Please see Figure 3 , Figure 3This is a schematic diagram of the structure of a video recognition device provided in an embodiment of this application. The video recognition device 200 includes a voice feature acquisition module 201, a behavior feature acquisition module 202, a feature fusion module 203, and a video frame recognition module 204.

[0092] The speech feature acquisition module 201 is used to acquire the speech features of the audio frames included in the video frame to be identified.

[0093] The behavior feature acquisition module 202 is used to acquire the object behavior features of the image frame corresponding to the audio frame.

[0094] The feature fusion module 203 is used to fuse the speech features of the object and the behavior features of the object to obtain fused features.

[0095] The video frame recognition module 204 is used to identify whether the video frame to be identified is the target video frame based on the fusion features, and to obtain the recognition result.

[0096] In an optional embodiment, the video frame recognition module 204 can be used to: input the fused features into a video frame classification model to obtain a classification result; and, based on the classification result, identify whether the video frame to be identified is a target video frame to obtain a recognition result.

[0097] In an optional embodiment, the video recognition device 200 may further include a video frame extraction module, which can be used to: acquire a video to be recognized; and extract video frames to be recognized from the video to be recognized.

[0098] In an optional embodiment, the video frame classification model includes a long short-term memory network model and a video frame recognition module 204, which can be used to: sequentially input the fusion features of each video frame to be recognized determined from the video to be recognized into the long short-term memory network model to obtain the classification result of each video frame to be recognized; and, based on the classification result of each video frame to be recognized, identify whether each video frame to be recognized is a target video frame to obtain the recognition result of each video frame to be recognized.

[0099] In an optional embodiment, the video recognition device 200 may further include a video acquisition module, which can be used to: acquire a target video based on a target video frame.

[0100] In an optional embodiment, the speech feature acquisition module 201 can be used to: extract the audio features of the audio frame; perform speaker clustering processing on the audio features to obtain clustering results; and obtain the object speech features of the audio frame based on the clustering results.

[0101] In an optional embodiment, the behavior feature acquisition module 202 can be used to: input the image frame into a behavior detection model to obtain a detection result; and obtain the object behavior features of the image frame based on the detection result.

[0102] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed on a computer, causes the computer to perform the video recognition method provided in this embodiment.

[0103] This application also provides an electronic device, including a memory and a processor, wherein the processor executes the video recognition method provided in this embodiment by calling a computer program stored in the memory.

[0104] For example, the aforementioned electronic devices can be mobile terminals such as tablets or smartphones. See also Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0105] The electronic device 300 may include components such as a processor 301 and a memory 302. Those skilled in the art will understand that... Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0106] The processor 301 is the control center of the electronic device. It connects various parts of the electronic device through various interfaces and lines. By running or executing the application program stored in the memory 302 and calling the data stored in the memory 302, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole.

[0107] Memory 302 can be used to store applications and data. The applications stored in memory 302 contain executable code. Applications can be composed of various functional modules. Processor 301 executes various functional applications and data processing by running the applications stored in memory 302.

[0108] In this embodiment, the processor 301 in the electronic device loads the executable code corresponding to the processes of one or more applications into the memory 302 according to the following instructions, and the processor 301 runs the applications stored in the memory 302, thereby realizing the process:

[0109] Obtain the speech features of the audio frames included in the video frame to be identified;

[0110] Obtain the object behavior features of the image frame corresponding to the audio frame;

[0111] The object's speech features and the object's behavioral features are fused to obtain fused features;

[0112] Based on the fusion features, it is determined whether the video frame to be identified is the target video frame, and the identification result is obtained.

[0113] Please see Figure 5 The electronic device 300 may include components such as a processor 301, a memory 302, an input unit 303, and an output unit 304.

[0114] The processor 301 is the control center of the electronic device. It connects various parts of the electronic device through various interfaces and lines. By running or executing the application program stored in the memory 302 and calling the data stored in the memory 302, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole.

[0115] Memory 302 can be used to store applications and data. The applications stored in memory 302 contain executable code. Applications can be composed of various functional modules. Processor 301 executes various functional applications and data processing by running the applications stored in memory 302.

[0116] The input unit 303 can be used to receive input numbers, characters, or user characteristic information (such as fingerprints), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0117] The output unit 304 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of electronic devices. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. The output unit may include a display screen, which may include a display area.

[0118] In this embodiment, the processor 301 in the electronic device loads the executable code corresponding to the processes of one or more applications into the memory 302 according to the following instructions, and the processor 301 runs the applications stored in the memory 302, thereby realizing the process:

[0119] Obtain the speech features of the audio frames included in the video frame to be identified;

[0120] Obtain the object behavior features of the image frame corresponding to the audio frame;

[0121] The object's speech features and the object's behavioral features are fused to obtain fused features;

[0122] Based on the fusion features, it is determined whether the video frame to be identified is the target video frame, and the identification result is obtained.

[0123] In an optional embodiment, when the processor 301 executes the step of identifying whether the video frame to be identified is a target video frame based on the fusion features and obtains the identification result, it may perform the following steps: inputting the fusion features into a video frame classification model to obtain a classification result; and identifying whether the video frame to be identified is a target video frame based on the classification result to obtain the identification result.

[0124] In an optional embodiment, before the processor 301 performs the acquisition of the object speech features of the audio frames included in the video frame to be identified, it may also perform: acquiring the video to be identified; and extracting the video frame to be identified from the video to be identified.

[0125] In an optional embodiment, the video frame classification model includes a long short-term memory network model. When the processor 301 executes the step of inputting the fused features into the video frame classification model to obtain classification results, it can perform the following: sequentially inputting the fused features of each video frame to be identified from the video to be identified into the long short-term memory network model to obtain classification results for each video frame to be identified; when the processor 301 executes the step of identifying whether the video frame to be identified is a target video frame based on the classification results to obtain identification results, it can perform the following: identifying whether each video frame to be identified is a target video frame based on the classification results of each video frame to be identified to obtain identification results for each video frame to be identified.

[0126] In an optional embodiment, after the processor 301 executes the step of identifying whether each video frame to be identified is a target video frame based on the classification results of each video frame to be identified, and obtains the identification results of each video frame to be identified, it may also execute the step of obtaining the target video based on the target video frame.

[0127] In an optional embodiment, when the processor 301 executes the process of obtaining the object speech features of the audio frames included in the video frame to be identified, it may perform the following: extracting the audio features of the audio frames; performing speaker clustering processing on the audio features to obtain clustering results; and obtaining the object speech features of the audio frames based on the clustering results.

[0128] In an optional embodiment, when the processor 301 executes the step of obtaining the object behavior features of the image frame corresponding to the audio frame, it may perform the following: inputting the image frame into the behavior detection model to obtain the detection result; and obtaining the object behavior features of the image frame based on the detection result.

[0129] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed description of the video recognition method above, which will not be repeated here.

[0130] The video recognition device provided in this application embodiment belongs to the same concept as the video recognition method in the above embodiment. Any of the methods provided in the video recognition method embodiment can be run on the video recognition device. For details of its implementation process, please refer to the video recognition method embodiment, which will not be repeated here.

[0131] It should be noted that, for the video recognition method of this application embodiments, those skilled in the art will understand that all or part of the process of implementing the video recognition method of this application embodiments can be accomplished by a computer program controlling related hardware. The computer program can be stored in a computer-readable storage medium, such as a memory, and executed by at least one processor. During execution, it can include the process of the video recognition method embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), etc.

[0132] For the video recognition device of this application embodiment, its functional modules can be integrated into a processing chip, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0133] The above provides a detailed description of a video recognition method, apparatus, storage medium, and electronic device provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A video recognition method, characterized in that, include: Obtain the speech features of the audio frames included in the video frame to be identified, wherein the video frame to be identified is a video frame in a classroom live video, and the speech features describe the speaking object in the audio frame; Obtain the object behavior features of the image frame corresponding to the audio frame, and the object behavior features describe the behavior of the object in the image frame in the classroom; The object's speech features and the object's behavioral features are fused to obtain fused features; Based on the fusion features, it is determined whether the video frame to be identified is a target video frame, and an identification result is obtained. The target video frame includes key video frames in the classroom live video. The acquisition of the object speech features of the audio frames included in the video frame to be identified includes: Extract the audio features of the audio frame; Speaker clustering is performed on the audio features to obtain the clustering results; The object speech features of the audio frame are obtained based on the clustering results.

2. The video recognition method according to claim 1, characterized in that, The step of identifying whether the video frame to be identified is the target video frame based on the fusion features, and obtaining the identification result, includes: The fused features are input into the video frame classification model to obtain the classification result; Based on the classification results, it is determined whether the video frame to be identified is the target video frame, and the identification result is obtained.

3. The video recognition method according to claim 2, characterized in that, Before obtaining the object speech features of the audio frames included in the video frame to be identified, the method further includes: Obtain the video to be recognized; Extract the video frames to be identified from the video to be identified.

4. The video recognition method according to claim 3, characterized in that, The video frame classification model includes a long short-term memory network model. The step of inputting the fused features into the video frame classification model to obtain the classification result includes: The fusion features of each video frame to be identified, determined from the video to be identified, are sequentially input into the long short-term memory network model to obtain the classification results of each video frame to be identified. The step of identifying whether the video frame to be identified is the target video frame based on the classification result, and obtaining the identification result, includes: Based on the classification results of each video frame to be identified, it is determined whether each video frame to be identified is a target video frame, and the identification result of each video frame to be identified is obtained.

5. The video recognition method according to claim 4, characterized in that, After identifying whether each video frame to be identified is a target video frame based on the classification results of each video frame to be identified, and obtaining the identification results of each video frame to be identified, the method further includes: The target video is obtained based on the target video frames.

6. The video recognition method according to any one of claims 1 to 4, characterized in that, The process of obtaining the object behavior features of the image frame corresponding to the audio frame includes: The image frame is input into the behavior detection model to obtain the detection result; Based on the detection results, the object behavior features of the image frame are obtained.

7. A video recognition device, characterized in that, include: The speech feature acquisition module is used to extract the audio features of the audio frames included in the video frame to be identified, perform speaker clustering processing on the audio features to obtain the clustering results, and obtain the object speech features of the audio frame based on the clustering results. The video frame to be identified is a video frame in a classroom live video, and the object speech features describe the speaking object in the audio frame. The behavior feature acquisition module is used to acquire the object behavior features of the image frame corresponding to the audio frame, wherein the object behavior features describe the behavior of the object in the image frame in the classroom. The feature fusion module is used to fuse the object's speech features and the object's behavior features to obtain fused features; The video frame recognition module is used to identify whether the video frame to be identified is a target video frame based on the fusion features, and to obtain the recognition result. The target video frame includes key video frames in the classroom live video.

8. A storage medium, characterized in that, The storage medium stores a computer program that, when run on a computer, causes the computer to perform the video recognition method according to any one of claims 1 to 6.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing a computer program, and the processor executing the video recognition method according to any one of claims 1 to 6 by calling the computer program stored in the memory.

Images