Key frame determination method and device, electronic equipment, medium and program product

Abstract

The invention provides a key frame determination method and device, electronic equipment, a medium and a program product, which can be used in the field of image processing. The method comprises the following steps: determining a target event area of a first image of a current frame in a video; obtaining an event segment of the target event area in the video; acquiring image features and audio features of the event segments, and reasoning the event segments according to the image features and the audio features to determine key frames in the video; and carrying out risk detection on an image corresponding to the key frame. In the application, through the image and the audio of the event segment, the image area representing the end of the behavior event in the image is reasoned, reasoning of the whole image is not needed, the time length of risk auditing of the image is reduced, that is to say, key frame analysis is carried out in combination with two modalities of the video and the audio under relatively small resource overhead, and the accuracy of risk auditing of the image is improved. The capturing accuracy and recall rate of the key frame are remarkably improved, and the video risk control effect is remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a key frame determination method, device, electronic device, medium and program product. Background Art

[0002] With the rapid development of the Internet, the output of videos has exploded. The rise of social media, short video platforms, live broadcasts, etc. has led to a large number of generated videos. As the number of videos increases, the risks they bring are becoming increasingly prominent. For example, the spread of illegal and other harmful content on online platforms not only threatens the safety of users, but also poses a challenge to the compliance and reputation of the platform. In order to cope with this complex situation, it is necessary to monitor the risks of video content in videos in order to immediately ban harmful content in videos.

[0003] The existing risk monitoring of video content is to extract key frame images from the video and then perform risk detection on the key frames. However, the risk review of images generally calls dozens of deep learning models and combines hundreds of rules for risk analysis, which takes a long time. Therefore, the frame extraction frequency is not too high, and key frames containing illegal information may not be collected, which means that illegal information in images is difficult to capture. Summary of the invention

[0004] The present application provides a key frame determination method, device, electronic device, medium and program product, which solves the problem that illegal information in an image is difficult to capture.

[0005] In a first aspect, the present application provides a key frame determination method, comprising:

[0006] Determine a target event region of a first image of a current frame in a video, wherein the target event region is used to represent an image region where a behavior event in the video ends, and the image of the video includes multiple image regions;

[0007] Acquire an event segment of the target event area in the video, wherein the event segment is composed of a plurality of first event areas, each of the first event areas contains the same behavior event, and each of the first event areas contains the target event area;

[0008] Acquire image features and audio features of the event segment, and infer the event segment based on the image features and the audio features to determine key frames in the video;

[0009] Perform risk detection on the image corresponding to the key frame.

[0010] In some embodiments, determining the target event area of ​​the first image of the current frame in the video includes:

[0011] Acquire each second event region inherited by the first image from the second image of the previous frame, and determine each third event region newly added to the first image;

[0012] A target event area of ​​the first image is determined according to each of the second event areas and each of the third event areas.

[0013] In some embodiments, acquiring each second event region inherited by the first image from the second image of the previous frame includes:

[0014] Determine the inter-frame mean square error and inter-frame structural similarity corresponding to the pixel points at the same position in the first image and the second image;

[0015] Determine a first matrix and a second matrix for each sub-region in the first image according to the inter-frame mean square error and the inter-frame structural similarity of each pixel point, each sub-region includes a plurality of pixels, the first matrix is ​​composed of the inter-frame mean square error of each pixel point in the corresponding sub-region, and the second matrix is ​​composed of the inter-frame structural similarity of each pixel point in the corresponding sub-region;

[0016] In response to the first matrix and the second matrix of the sub-region both indicating that a behavior event occurs in the sub-region, determining the sub-region as an event sub-region, and determining a first ratio between the number of the event sub-regions and the number of sub-regions in the first image;

[0017] In response to the first ratio being less than or equal to a first preset ratio, in the ongoing event region of the second image, each second event region inherited by the first image from the second image is determined.

[0018] In some embodiments, determining, in the continued event area of ​​the second image, each second event area inherited by the first image from the second image includes:

[0019] Determining a second ratio between the number of event sub-regions included in the ongoing event region and the number of sub-regions included in the ongoing event region;

[0020] In response to the second ratio being greater than or equal to a second preset ratio, the ongoing event region is determined as a second event region inherited by the first image from the second image.

[0021] In some embodiments, after determining the second ratio between the number of event sub-regions included in the ongoing event region and the number of sub-regions included in the ongoing event region, the method further includes:

[0022] In response to the second ratio being less than a second preset ratio, it is determined that the ongoing event region has ended, and the ongoing event region is determined as a target event region of the first image.

[0023] In some embodiments, after determining a first ratio between the first number and a second number of sub-regions in the first image, the method further includes:

[0024] In response to the first ratio being greater than a first preset ratio, the ongoing event region of the second image is determined as the target event region of the first image.

[0025] In some embodiments, determining each third event area newly added to the first image includes:

[0026] According to different clustering strategies, clustering is performed multiple times on each sub-region in the first image to obtain each initial event region corresponding to each clustering strategy, wherein the number of clustering centers of each clustering strategy is different, and the clustering center in each initial event region is the event sub-region;

[0027] Determine, according to the sub-regions in each initial event region corresponding to the clustering strategy, a first target sub-region corresponding to the clustering strategy, wherein the first target sub-region is used to indicate a non-overlapping sub-region between each initial event region corresponding to the clustering strategy;

[0028] Obtaining the number of targets in the first target sub-region corresponding to each of the clustering strategies;

[0029] The initial event regions of the clustering strategy corresponding to the largest number of targets are determined as the third event regions newly added to the first image.

[0030] In some embodiments, determining the target event area of ​​the first image according to each of the second event areas and each of the third event areas includes:

[0031] According to different event area merging strategies, multiple event areas are selected in sequence from each of the second event areas and each of the third event areas to obtain a set corresponding to each of the event area merging strategies, wherein each of the sets includes event areas selected each time, and the number of event areas selected each time is different;

[0032] Determining, according to the event areas selected in each of the sets, a merging effect parameter of the event area merging strategy corresponding to the set;

[0033] Each selected event region in the set corresponding to the maximum combined effect parameter is used as an intermediate event region, and a target event region is determined according to each of the second event regions, each of the third event regions and the intermediate event region.

[0034] In some embodiments, determining the merging effect parameters of the event area merging strategy corresponding to each set according to the event area selected from the set includes:

[0035] Determining, according to each selected event region in the set, a second target sub-region corresponding to the set, wherein the second target sub-region is used to indicate a sub-region that does not overlap between each selected event region in the set, wherein the first image includes a plurality of sub-regions, and the sub-region of the first image includes a plurality of pixel points;

[0036] According to the number of second target sub-areas corresponding to the set, a merging effect parameter corresponding to the event area merging strategy corresponding to the set is determined.

[0037] In some embodiments, determining the target event area according to each of the second event areas, each of the third event areas, and the intermediate event area includes:

[0038] According to each of the intermediate event areas, determining a fourth event area in each of the second event areas and each of the third event areas, wherein the fourth event area is an event area other than the intermediate event area among the second event area and the third event area;

[0039] The fourth event area is determined as a target event area of ​​the first image.

[0040] In some embodiments, the method further includes: after taking each selected event region in the set corresponding to the maximum combined effect parameter as the intermediate event region, the method further includes:

[0041] In each of the intermediate event regions, a plurality of event region groups are determined, wherein the intermediate event regions included in each of the event region groups have overlapping sub-regions, the first image includes the plurality of sub-regions, and the sub-regions of the first image include a plurality of pixel points;

[0042] In response to the number of overlapping sub-regions between the intermediate event regions included in the event region group being greater than or equal to a preset number, determining the intermediate event region in the event region group belonging to the second image as a continuing event region of the first image;

[0043] In response to the number of overlapping sub-regions between the intermediate event regions included in the event region group being less than a preset number, each intermediate event region of the event region group is determined as a continuing event region of the first image.

[0044] In some embodiments, the inferring the event segment according to the image feature and the audio feature includes:

[0045] Determine image features and audio features of each first event region in the event segment;

[0046] splicing the image features and the audio features of the first event area to obtain splicing features of the first event area;

[0047] The splicing features of each of the first event regions are input into the inference model.

[0048] In a second aspect, the present application provides a key frame determination device, comprising:

[0049] A determination module, used to determine a target event region of a first image of a current frame in a video, wherein the target event region is used to represent an image region where a behavior event in the video ends, and the image of the video includes multiple image regions;

[0050] A first acquisition module, configured to acquire an event segment of the target event region in the video, wherein the event segment is composed of a plurality of first event regions, each of which contains a same behavior event, and each of which contains the target event region;

[0051] A second acquisition module is used to acquire image features and audio features of the event segment, and infer the event segment according to the image features and the audio features to determine the key frames in the video;

[0052] The detection module is used to perform risk detection on the image corresponding to the key frame.

[0053] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory and a communication interface communicatively connected to the processor;

[0054] The communication interface is used to communicate with other communication devices;

[0055] The memory is used to store computer-executable instructions;

[0056] The processor is used to execute the computer-executable instructions stored in the memory to implement the key frame determination method provided in the first aspect.

[0057] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the key frame determination method provided in the first aspect is implemented.

[0058] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the key frame determination method provided in the first aspect.

[0059] The key frame determination method, device, electronic device, medium and program product provided in the present application obtain a target event area representing the end of a behavioral event from an image in a video, thereby obtaining a fragment representing an event that has ended based on the behavioral event that ended in the target event area, and the event fragment contains multiple event areas, and the event area is an image area in an image in a video frame. Therefore, the image area in the image representing the end of the behavioral event is inferred through the image and audio of the event fragment, without the need to infer the entire image, thereby reducing the time for risk review of the image. That is, with a relatively small resource overhead, key frame analysis is performed in combination with the two modes of video and audio, which significantly improves the key frame capture accuracy and recall rate, and significantly improves the video risk control effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0061] Figure 1 The following is a schematic diagram of the steps of the key frame determination method according to the embodiment of the present application: Figure 1 ;

[0062] Figure 2 The video division level involved in the embodiment of the present application;

[0063] Figure 3 A schematic diagram of the structure of the reasoning model involved in the embodiment of the present application;

[0064] Figure 4 The following is a schematic diagram of the steps of the key frame determination method according to the embodiment of the present application: Figure 2 ;

[0065] Figure 5 The following is a schematic diagram of the steps of the key frame determination method according to the embodiment of the present application: Figure 3 ;

[0066] Figure 6 Schematic diagram of the division of sub-regions of an image in an embodiment of the present application;

[0067] Figure 7 The following is a schematic diagram of the steps of the key frame determination method according to the embodiment of the present application: Figure 4 ;

[0068] Figure 8 The following is a schematic diagram of the steps of the key frame determination method according to the embodiment of the present application: Figure 5 ;

[0069] Fig. 9 The following is a schematic diagram of the steps of the key frame determination method according to the embodiment of the present application: Figure 6 ;

[0070] Fig.10 A schematic diagram of a program module of a key frame determination device provided in an embodiment of the present application;

[0071] Fig.11 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0072] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application. In addition, although the disclosure in the present application is introduced according to one or several exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete implementation method separately.

[0074] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0075] In addition, the terms "include" and "have" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such products or devices.

[0076] The term "module" used in the embodiments of the present application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or a combination of hardware and / or software code that can perform the functions associated with the element.

[0077] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operating

[0078] With the rapid development of the Internet, the output of videos has exploded. The rise of social media, short video platforms, live broadcasts, etc. has led to a large number of generated videos. As the number of videos increases, the risks they bring are becoming increasingly prominent. For example, the spread of illegal and other harmful content on online platforms not only threatens the safety of users, but also poses a challenge to the compliance and reputation of the platform. In order to cope with this complex situation, it is necessary to monitor the risks of video content in videos in order to immediately ban harmful content in videos.

[0079] The existing risk monitoring of video content is to extract key frame images from the video and then perform risk detection on the key frame images.

[0080] The inventors of this application have found that the risk review of images generally calls upon dozens of deep learning models and combines hundreds of rules for risk analysis, which is time-consuming. Therefore, the frame extraction frequency will not be too high, and key frames containing violation information may not be collected, which means that it is difficult to capture violation information in the image.

[0081] The inventor of the present application therefore conceived of obtaining a target event region representing the end of a behavioral event from an image in a video, thereby obtaining a fragment representing an event that has ended based on the behavioral event that ended in the target event region. An event fragment includes multiple event regions, and an event region is an image region in an image in a video frame. Therefore, the image region in the image representing the end of the behavioral event is inferred through the image and audio of the event fragment, without the need to infer the entire image, thereby reducing the duration of the risk review of the image. That is, with a relatively small resource overhead, key frame analysis is performed by combining the two modalities of video and audio, which significantly improves the key frame capture accuracy and recall rate, and significantly improves the video risk control effect.

[0082] The technical solution shown in the present application is described in detail through the following embodiments. It should be noted that the following embodiments can exist independently or in combination with each other, and the same or similar contents will not be described repeatedly in different embodiments.

[0083] Reference Figure 1 , Figure 1 The following is a flow chart of the key frame determination method provided in the embodiment of the present application. Figure 1 , the key frame determination method includes:

[0084] Step S101, determining a target event region of a first image of a current frame in a video, wherein the target event region is used to represent an image region where a behavior event in the video ends, and the image of the video includes multiple image regions.

[0085] In this embodiment, the execution subject is a key frame determination device. For the convenience of description, the key frame determination device is referred to as a device below. The device can be a server or a terminal device with a key frame determination function.

[0086] In this embodiment, event segments are extracted from the video from which key frames need to be extracted. An event segment refers to a continuous behavioral event recorded in a video. For example, a behavioral event is a person A waving his hand. There are multiple images recorded in the video. The segment video consisting of the image area of ​​the person A's hand in each image is the event segment. Figure 2 For a video, it can be divided into the following levels: scene, shot, behavioral event, frame. There can be multiple behavioral events in a video at the same time. For example, in a video, character A waves and character B laughs. Character A waving is event A, and character B laughing is event B.

[0087] The device analyzes each frame of the image to filter out unchanged behavioral events and obtain ended behavioral events. Ended behavioral events represent shot switching, important events, the start of an action, or the end of an action, etc. Therefore, it is necessary to determine the image area in the current frame that represents the shot switching, the end of an important event, the start of an action, and the end of an action.

[0088] The device uses the current image in the video as the first image and determines the target event area of ​​the first image, and the target event area is used to represent the image area where the behavior event in the video ends. Exemplarily, each frame image in the video is divided into a fixed number of sub-areas, for example, divided into 10×10 sub-areas, each sub-area contains multiple pixel points of the image, and the image area is composed of a fixed number of sub-areas, for example, composed of 3×3 sub-areas, that is, the image in the video contains multiple image areas.

[0089] Exemplarily, the device first identifies the image area where the behavioral event occurs in the first image as the target image area, and then identifies whether there is a moving object in the target image area. For example, if there are people and movable objects in the target image area, there is a moving object in the target image area, and the device regards the target image area as the target event area where the behavioral event ends.

[0090] Step S102: obtaining an event segment of a target event region in a video, wherein the event segment is composed of a plurality of first event regions, each of which contains a same behavior event, and each of which contains a target event region.

[0091] After determining the target event region, the device obtains an event segment of the target region in the video, wherein the event frequency band is composed of a plurality of first event regions, each of which contains the same behavior event, and each of which contains the target event region.

[0092] Exemplarily, the behavioral event in the target event area is person A waving, and the first image is the 101st frame in the video, that is, person A starts waving in the 101st frame of the video and stops waving in the 110th frame. Therefore, the event segment of person A waving includes the image area of ​​waving in the 10 frames from the 101st frame to the 110th frame.

[0093] Step S103: acquiring image features and audio features of the event segment, and inferring the event segment based on the image features and the audio features to determine key frames in the video.

[0094] After obtaining the event fragment, the event fragment is split into video frames and audio, and the video frame is input into the image feature extraction model to obtain the image features of the event fragment; the network of the image feature extraction model is a lightweight convolutional neural network that can be set up on mobile devices with limited computing resources. It aims to achieve efficient image classification, target detection and semantic segmentation tasks under limited computing resources, and has low resource overhead in high-traffic scenarios of key frame screening. The audio of the event fragment is input into the audio feature extraction model to obtain the audio features of the event fragment.

[0095] In one example, the device inputs the audio features and the image features into the inference model, and the inference model outputs a key frame. For example, the event segment includes images of frames 101 to 110, and the 101st frame image is detected as a key frame image.

[0096] In another example, the device determines the image features and audio features of each first event area in the event clip, splices the image features and audio features of the first event area to obtain the spliced ​​features of the first event area, and then inputs the spliced ​​features of each first event area into the inference model.

[0097] The audio modality is characterized by extracting features through the ConvNet (convolutional neural network) model. The audio segment corresponding to a single frame is short, so only a simple network is required. The input of the ConvNet model is the Log Mel spectrogram feature.

[0098] After the features of the video frame mode (image features) and the audio mode are merged, they are input into the ConvGRU (convolutional gated recurrent unit) for inference, that is, ConvGRU is the inference model to determine whether it is a key frame. ConvGRU is a model that combines convolutional neural networks and gated recurrent units. It is often used to process spatiotemporal data, such as images in videos or dynamic sequences. It models the dependencies of sequence data in the time dimension and uses convolution operations to capture spatial features. It is suitable for classification tasks of continuous video clips. The main structure of ConvGRU is as follows:

[0099] Reset the gate:

[0100]

[0101] Update Gate:

[0102]

[0103] Candidate status:

[0104]

[0105] Final state:

[0106]

[0107] The video side uses a pre-trained lightweight network image feature extraction model to solve the video frame extraction task through transfer learning. The audio side uses a ConvNet model to extract audio clip features. The one pooling layer and two convolutional layers at the bottom of the image feature extraction model are removed, concatenated with the output of the ConvNet model, and then merged and output to the ConvGRU layer to capture the spatiotemporal characteristics of the event clip. At the same time, the top-level structure of MoblieNet is fine-tuned to reduce the input image size to match the characteristics of small-size local screenshots of event clips. In addition, the image feature extraction model can be used as the image feature extraction layer, the ConvNet model as the audio feature extraction layer, and the ConvGRU layer, pooling layer, and fully connected layer to form an inference model. The structure of the inference model is specifically referred to. Figure 3 , where the last fully connected layer contains the activation function Sigmod.

[0108] In this embodiment, the event fragment is analyzed by the ConvGRU model to obtain key frames. The ConvGRU model retains the structure of the input image or feature map in space, and combines the audio timing features to more effectively model the spatiotemporal correlation; the deep separable convolution of the image feature extraction model and the dual-gate design of GRU significantly reduce the inference model parameters and computational complexity to cope with the high-traffic scenario of video review; fine-tuning is performed on the basis of the image feature extraction model to reduce the low input image size, which not only matches the characteristics of the small-size local screenshots of the event fragments, but also further reduces the model parameter volume.

[0109] Step S104: Perform risk detection on the image corresponding to the key frame.

[0110] After determining the key frame, the device can perform risk detection on the image corresponding to the key frame, that is, detect violation information on the image as the key frame to determine whether the image contains violation information. Specifically, the device recognizes the first text information displayed in the image in the key frame through OCR (Optical Character Recognition), and obtains the second text information through ASR (Automatic Speech Recognition), and inputs the first text information and the second text information into the text model to identify whether the image contains violation information through the text model.

[0111] In this embodiment, a target event area representing the end of a behavioral event is obtained from an image in a video, thereby obtaining a fragment representing the ended event based on the behavioral event that ended in the target event area, and the event fragment contains multiple event areas, and the event area is an image area in an image in a video frame. Therefore, the image area in the image representing the end of the behavioral event is inferred through the image and audio of the event fragment, without the need to infer the entire image, thereby reducing the time for risk review of the image. That is, with a relatively small resource overhead, key frame analysis is performed by combining the two modalities of video and audio, which significantly improves the key frame capture accuracy and recall rate, and significantly improves the video risk control effect.

[0112] Reference Figure 4 , Figure 4 The process diagram of the key frame determination method of this application is shown as follows: Figure 2 ,based on Figure 1 In the illustrated embodiment, step S101 includes:

[0113] Step S401 : acquiring each second event region inherited by the first image from the second image of the previous frame, and determining each third event region newly added to the first image.

[0114] In this embodiment, the device obtains each second event area inherited by the first image from the second image of the previous frame. Specifically, the behavior event in the event area existing in the second image of the previous frame will continue, so the identified event area of ​​the second image can be inherited to the second event area of ​​the first image.

[0115] In addition, the first image will have a newly added event area, which is defined as a third event area. That is, the device obtains the third event area by identifying the first image.

[0116] Step S402: determining a target event region of the first image according to each second event region and each third event region.

[0117] After the device obtains the second event area and the third event area, the device calculates the first similarity between the third event area and each second event area. If there is a first similarity greater than or equal to a preset threshold, then there is a second event area in the second image that is the same as the behavioral event in the third event area, and this behavioral event has not ended; if all the first similarities are less than the preset threshold, it can be determined that the behavioral event in the third event area does not exist in the second image and is a new behavioral event, so the third event area can be used as the target event area.

[0118] In addition, the device also needs to calculate the second similarity between the second event area and each third event area. If there is a second similarity greater than or equal to a preset threshold, then there is a third event area in the first image that is the same behavior event as the behavior event in the second event area, and this behavior event has not ended; if all the second similarities are less than the preset threshold, it can be determined that the behavior event in the second event area does not exist in the second image, the behavior event in the second event area is a ended behavior event, and thus the second event area can be used as the target event area.

[0119] In this embodiment, the device obtains each second event area inherited by the first image from the second image of the previous frame, and determines each third event area newly added to the first image, so as to quickly identify the target event area of ​​the first image based on each second event area and each third event area.

[0120] Reference Figure 5 , Figure 5 The process diagram of the key frame determination method of this application is shown as follows: Figure 3 ,based on Figure 4 In the illustrated embodiment, step S401 includes:

[0121] Step S501, determining the inter-frame mean square error and inter-frame structural similarity corresponding to the pixel points at the same position in the first image and the second image.

[0122] In this embodiment, the image in the video includes a plurality of pixels, and the device calculates the corresponding inter-frame mean square error and inter-frame structural similarity of the pixels at the same position in the first image and the second image.

[0123] Exemplarily, the inter-frame mean square error is:

[0124]

[0125] Among them, MSE(X,Y) is the mean square error between frames, m and n indicate the width and height of the image, i and j represent the pixel coordinates of the pixel point, Y represents the pixel matrix of the current frame image, and X represents the pixel matrix of the previous frame image.

[0126] The inter-frame structural similarity is:

[0127]

[0128] Among them, SSIM(X,Y) is the inter-frame structural similarity, μ X is the mean pixel value of the previous frame, μ Y is the mean pixel value of the current frame image, C1 and C2 are constants, σ XY is the pixel value covariance of the two frames of images, is the pixel value variance of the pixel in the previous frame image, is the pixel value variance of the pixel points in the current frame image.

[0129] Step S502, based on the inter-frame mean square error and inter-frame structural similarity of each pixel point, determine the first matrix and the second matrix of each sub-region in the first image, each sub-region includes multiple pixels, the first matrix is ​​composed of the inter-frame mean square error of each pixel point in the corresponding sub-region, and the second matrix is ​​composed of the inter-frame structural similarity of each pixel point in the corresponding sub-region.

[0130] After determining the inter-frame mean square error and inter-frame structural similarity of each pixel point in the first image, the device determines a first matrix and a second matrix for each sub-region in the first image.

[0131] Exemplarily, the device divides each frame image in the video into multiple sub-areas, referring to Figure 6 Each frame image is divided into 10×10 sub-regions, each sub-region contains multiple pixels, the inter-frame mean square error of each pixel in the sub-region constitutes a first matrix, and the inter-frame structural similarity of each pixel in the sub-region constitutes a second matrix.

[0132] Step S503, in response to the first matrix and the second matrix of the sub-region both indicating that a behavior event occurs in the sub-region, determining the sub-region as an event sub-region, and determining a first ratio between the number of event sub-regions and the number of sub-regions in the first image.

[0133] After determining the first matrix and the second matrix of each sub-region, the device determines whether the sub-region is an event sub-region based on the first matrix and the second matrix of the sub-region. The event sub-region refers to a part of the displayed behavior event. For example, if a sub-region displays a hand, then the sub-region is an event sub-region.

[0134] Exemplarily, the device determines a first comparison result between the first matrix and the first unit matrix, and determines a first comparison result between the second matrix and the second unit matrix, and determines whether the sub-region is an event sub-region based on the first comparison result and the second comparison result. Specifically, the judgment method is as follows:

[0135] F e =(F MSE >t1)∧(F SSIM <t2)

[0136] Among them, t1 is the first unit matrix, t2 is the second unit matrix, F MSE is the first matrix, F SSIM is the second matrix, “∧” is the AND operation, F e is the event sub-area.

[0137] From the above, it can be seen that when the first matrix is ​​greater than the first unit matrix and the second matrix is ​​less than the second unit matrix, it can be determined that both the first matrix and the second matrix indicate that a behavior event occurs in the sub-region, and the sub-region is determined as an event sub-region.

[0138] The device determines each event sub-region in the above manner, and determines a first ratio between the number of event sub-regions and the number of sub-regions in the first image. The first ratio is:

[0139]

[0140] Among them, p e is the first ratio, count(F e ) is the number of event sub-regions, and s×s is the number of sub-regions in the image.

[0141] Step S504 , in response to the first ratio being less than or equal to the first preset ratio, determining, in the ongoing event region of the second image, each second event region inherited by the first image from the second image.

[0142] After determining the first ratio, the device compares the first ratio p e and the first preset threshold value t3. When the first ratio pe If it is less than or equal to the first preset ratio t3, it can be determined that the ongoing event area of ​​the second image has not ended. In the ongoing event area of ​​the image, each second event area inherited by the first image from the second image is determined, and the ongoing event area is the event area of ​​the second image recognized by the device.

[0143] When the first ratio p e If it is greater than the first preset ratio t3, it is determined that a shot switch has occurred, and all currently existing event areas are terminated, that is, the existing event areas of the second image are determined as the target event areas of the first image, and the event fragments of each event area that is about to end are sent to the key frame extraction filter in the device. The key frame filter extractor is the component for setting the inference model.

[0144] In this embodiment, by calculating the mean square error between frames and structural similarity, local event regions are captured in the video, so that the resource overhead required for the device to determine the key frame is small and the key frame can be determined quickly. The device preliminarily screens out event sub-regions based on the mean square error between frames and structural similarity, reduces the amount of calculation of the subsequent key frame extraction process, filters out the unchanged area, and reduces the input noise of the subsequent process.

[0145] In this embodiment, the device divides the image into multiple sub-areas, and there is no need to perform calculations based on single pixels, thereby reducing the workload of the device. That is, based on the first matrix and the second matrix of each sub-area, the device quickly determines the second event areas inherited by the first image from the second image.

[0146] Reference Figure 7 , Figure 7 The process diagram of the key frame determination method of this application is shown as follows: Figure 4 ,based on Figure 5 In any of the embodiments shown in , step S503 includes:

[0147] Step S701, determining a second ratio between the number of event sub-regions included in the ongoing event region and the number of sub-regions included in the ongoing event region.

[0148] In this embodiment, the device determines the number of event sub-regions included in the ongoing event region, and the event sub-regions are determined in the same manner as above. The device then determines the number of sub-regions included in the ongoing event region, and calculates the ratio between the number of event sub-regions included in the ongoing event region and the number of sub-regions included in the ongoing event region, and the ratio is defined as a second ratio. Exemplarily, the second ratio is:

[0149]

[0150] in, is the second ratio, d×d is the number of sub-regions in the ongoing event region, N is the number of event sub-regions in the ongoing event region, em Characterizes the number of regions with ongoing events. Figure 6 , the sub-region in the image is 10×10, then d×d can be 3×3; 10×10 and 3×3 are just examples, and do not limit the number of sub-regions of the image and the number of sub-regions in the event region.

[0151] Step S702: in response to the second ratio being greater than or equal to the second preset ratio, determining the ongoing event region as a second event region inherited by the first image from the second image.

[0152] Device comparison second ratio and the second preset threshold t4, when the second ratio If the value is greater than or equal to the second preset threshold value t4, the ongoing event region has not ended, and the ongoing event region is determined as a second event region inherited by the first image from the second image.

[0153] When the second ratio If the value is less than the second preset threshold value t4, it is determined that the ongoing event region has ended, and the ongoing event region is determined as the target event region of the first image, that is, the event segment of the ongoing event region is generated to the key frame filter.

[0154] In this embodiment, the device quickly and accurately determines the second event region inherited by the first image from the second image based on the ratio between the number of event sub-regions included in the ongoing event region and the number of sub-regions included in the ongoing event region.

[0155] Reference Figure 8 , Figure 8 The process diagram of the key frame determination method of this application is shown as follows: Figure 5 ,based on Figure 5 In the illustrated embodiment, step S501 includes:

[0156] Step S801, clustering each sub-region in the first image multiple times according to different clustering strategies to obtain each initial event region corresponding to each clustering strategy, wherein the number of cluster centers of each clustering strategy is different, and the cluster center in each initial event region is an event sub-region.

[0157] In this embodiment, the device performs multiple clustering on each sub-region in the first image, that is, the device obtains multiple clustering strategies. Each clustering strategy has a different number of categories, and when clustering, the event sub-region is used as the cluster center. For example, the number of categories is 1, 2...N respectively. em , and get N em A clustering strategy.

[0158] After each clustering, multiple event regions are obtained. Each event region is defined as an initial event region. Each initial event region contains d×d sub-regions.

[0159] Step S802: determining a first target sub-region corresponding to the clustering strategy according to sub-regions in each initial event region corresponding to the clustering strategy, wherein the first target sub-region is used to indicate a non-overlapping sub-region between each initial event region corresponding to the clustering strategy.

[0160] The device clusters each sub-region according to each clustering strategy to obtain each initial event region corresponding to each clustering strategy. The device determines the first target sub-region corresponding to the clustering strategy based on the sub-regions of each initial event region corresponding to the clustering strategy. The first target sub-region refers to the non-overlapping sub-region of each initial event region corresponding to the clustering strategy. Exemplarily, the clustering strategy corresponds to initial event region 1, initial event region 2, and initial event region 3. Initial event region 1 is composed of sub-region 12, sub-region 13, sub-region 22, and sub-region 23. Sub-region 12 refers to the sub-region of the first row and second column in the image. Similarly, sub-region 13 refers to the sub-region of the first row and third column in the image. Initial event region 2 is composed of sub-region 22, sub-region 23, sub-region 32, and sub-region 33. Initial event region 3 is composed of sub-region 23, sub-region 22, and sub-region 23. Therefore, the overlapping sub-areas between the initial event area 1 and the initial event area 2 are sub-areas 22 and sub-area 23, and the overlapping sub-areas between the initial event area 2 and the initial event area 3 are sub-areas 23 and sub-area 33. Therefore, sub-areas 12, sub-area 13, sub-area 24, sub-area 32 and sub-area 34 are non-overlapping sub-areas, that is, sub-areas 12, sub-area 13, sub-area 24, sub-area 32 and sub-area 34 are all used as first target sub-areas.

[0161] Step S803: Obtain the number of targets in the first target sub-region corresponding to each clustering strategy.

[0162] Each clustering strategy has a corresponding first target sub-region, and the device determines the number of first target sub-regions of each clustering strategy as the target number, which refers to the number of sub-blocks in the clustering strategy that only fall within a single event area.

[0163] In addition, the number of targets can be expressed by the formula: when the number of initial event regions N in the clustering strategy is em =2:

[0164]

[0165] Among them, DN1 , D N2 is the set of event sub-regions within the two initial event regions.

[0166] Step S804: determine each initial event region of the clustering strategy corresponding to the maximum number of targets as each third event region newly added to the first image.

[0167] The clustering effect of the clustering strategy corresponding to the largest number of targets is the best, and the device uses the initial event regions of the clustering strategy corresponding to the largest target data as the third event regions newly added to the first image.

[0168] In this embodiment, the device clusters each sub-region in the first image multiple times based on each clustering strategy, and obtains the event region clustered by the clustering strategy with better clustering effect as each third event region newly added to the first image, thereby accurately determining the event region newly added to the first image.

[0169] Reference Fig. 9 , Fig. 9 The process diagram of the key frame determination method of this application is shown as follows: Figure 6 ,based on Figures 4 to 8 In any of the embodiments shown in , step S402 includes:

[0170] Step S901, according to different event area merging strategies, in each second event area and each third event area, multiple event areas are selected in turn to obtain a set corresponding to each event area merging strategy, wherein each set includes event areas selected each time, and the number of event areas selected each time is different.

[0171] In this embodiment, the device merges the newly added third event area of ​​the first image with the inherited second event area. The newly added third event area refers to the event area of ​​the first image identified by the device. The third event area and the behavior event in a second event area may be the same behavior event. The third event area and the second event area with the same behavior event need to be merged.

[0172] The device takes 1,2...N in turn em event areas as a strategy for merging multiple event areas, that is, the device selects multiple event areas in each second event area and the third event area in turn according to different event area strategies. In order to reduce the workload, the device can select 1, 2...N in turn. emThe third event areas are merged with each second event area. It can be understood that the event area merging strategy refers to merging a set number of third event areas with each second event area. Each event area selected according to each event area merging strategy is used as a set corresponding to the event area merging strategy, that is, each set contains the event areas selected each time, and the number of event areas selected each time is different.

[0173] Step S902: Determine the merging effect parameters of the event area merging strategy corresponding to the set according to the event areas selected in each set.

[0174] After obtaining the set corresponding to each event area merging strategy, the device determines the merging effect parameters of the event area merging strategy corresponding to the set based on the event areas selected in each set.

[0175] In one example, the device determines the second target sub-region corresponding to the set based on each selected event region in the set, the second target sub-region refers to a sub-region that does not overlap between each selected event region in the set, the first image includes multiple sub-regions, and the sub-region of the first image includes multiple pixels. The set includes a subset D of the third event region newly added to the first image N2 and the subset D of the surviving event regions (second event regions) inherited by the first image from the second image O1 , the device determines the subset D N2 With subset D O1 The intersection between them is represented by D O1 ∩D N2 , so the second target sub-region corresponding to the set is expressed as: (|D O1 |-|D O1 ∩D N2 |)+(|D N2 |-|D O1 ∩D N2 |).

[0176] The device determines the merging effect parameter corresponding to the set by the number of the second target sub-areas in the set. The merging effect parameter is expressed as:

[0177]

[0178] in, is the combined effect parameter, α is the compensation coefficient of the event area to extend the life cycle of the event area; N ris the longest life cycle of the event region; r is the life cycle of the current third event region. For example, the behavior events in the third event region appear in the first frame to the tenth frame, and the frame rate of the video is 30 frames / second, then r = 9 / 30 = 0.3 seconds. The above is an event region merging strategy that merges a third event region. When multiple third event regions are selected in the event region merging strategy to merge with each second event region, each third event region corresponds to a Each third event area The average value of is the combined effect parameter.

[0179] Step S903 : taking each selected event region in the set corresponding to the maximum combined effect parameter as an intermediate event region, and determining a target event region according to each second event region, each third event region and the intermediate event region.

[0180] Each event area merging strategy corresponds to a merging effect parameter. The device determines the maximum merging effect parameter among the various merging effect parameters. The event areas of each option in the set corresponding to the maximum merging effect parameter are used as intermediate event areas, thereby determining the target event area based on each second event area, each third event area and each intermediate event area.

[0181] Exemplarily, the device determines the fourth event area in each second event area and each third event area based on each intermediate event area. The fourth event area is the event area in the second event area and the third event area, excluding the intermediate event area. It can be understood that the fourth event area is the second event area or the third event area that is not determined as the intermediate event area, that is, the fourth event area is the eliminated event area, and the behavior event in the eliminated event area is ended. The device determines the fourth event area as the target event area, that is, determines the eliminated event area as the target event area.

[0182] Further, after determining each intermediate event area, the device needs to merge each intermediate event area. Specifically, the device determines multiple event area groups in each intermediate event area, each event area group contains overlapping sub-areas, the first image contains multiple sub-areas, and the sub-areas of the first image contain multiple pixels, and the event area group refers to the newly added third event area and the continued second event area with the same sub-areas; when the number of overlapping sub-areas between the intermediate event areas contained in the event area group is greater than or equal to the preset number, the intermediate event area belonging to the second image in the event area group is determined as the continued event area of ​​the first image, that is, the newly added third event area in the event area group is discarded; when the number of overlapping sub-areas between the intermediate event areas contained in the event area group is less than the preset number, each intermediate event area of ​​the event area group is determined as the continued event area of ​​the first image, that is, the newly added third event area and the continued second event area are retained, so that when the device determines the target event area of ​​the next frame image, it can determine the event area inherited from the first image by the multiple continued event areas associated with the first image.

[0183] In this embodiment, the device accurately determines the target event region of the first image based on the optimal event region merging strategy.

[0184] Based on the contents described in the above embodiments, a key frame determination device is also provided in the embodiments of the present application. Fig.10 , Fig.10 Schematic diagram of a program module of a key frame determination device provided in an embodiment of the present application. In some implementations, the key frame determination device 1000 includes:

[0185] A determination module, used to determine a target event region of a first image of a current frame in a video, wherein the target event region is used to represent an image region where a behavior event ends in the video, and the image of the video includes multiple image regions;

[0186] A first acquisition module 1020 is used to acquire an event segment of a target event region in a video, wherein the event segment is composed of a plurality of first event regions, each of which contains the same behavior event, and each of which contains the target event region;

[0187] The second acquisition module 1030 is used to acquire image features and audio features of the event segment, and infer the event segment based on the image features and the audio features to determine the key frames in the video;

[0188] The detection module 1040 is used to perform risk detection on the image corresponding to the key frame.

[0189] In some embodiments, the key frame determination device 1000 is specifically used for:

[0190] Acquire each second event region inherited by the first image from the second image of the previous frame, and determine each third event region newly added to the first image;

[0191] A target event region of the first image is determined according to each second event region and each third event region.

[0192] In some embodiments, the key frame determination device 1000 is specifically used for:

[0193] Determine the inter-frame mean square error and inter-frame structural similarity corresponding to the pixel points at the same position in the first image and the second image;

[0194] Determine a first matrix and a second matrix for each sub-region in the first image according to the inter-frame mean square error and the inter-frame structural similarity of each pixel point, each sub-region includes a plurality of pixels, the first matrix is ​​composed of the inter-frame mean square error of each pixel point in the corresponding sub-region, and the second matrix is ​​composed of the inter-frame structural similarity of each pixel point in the corresponding sub-region;

[0195] In response to the first matrix and the second matrix of the sub-region both indicating that a behavior event occurs in the sub-region, determining the sub-region as an event sub-region, and determining a first ratio between the number of event sub-regions and the number of sub-regions in the first image;

[0196] In response to the first ratio being less than or equal to the first preset ratio, in the continuing event region of the second image, each second event region inherited by the first image from the second image is determined.

[0197] In some embodiments, the key frame determination device 1000 is specifically used for:

[0198] Determine a second ratio between the number of event sub-regions included in the ongoing event region and the number of sub-regions included in the ongoing event region;

[0199] In response to the second ratio being greater than or equal to the second preset ratio, the ongoing event region is determined as a second event region inherited by the first image from the second image.

[0200] In some embodiments, the key frame determination device 1000 is specifically used for:

[0201] In response to the second ratio being less than the second preset ratio, it is determined that the ongoing event region has ended, and the ongoing event region is determined as the target event region of the first image.

[0202] In some embodiments, the key frame determination device 1000 is specifically used for:

[0203] In response to the first ratio being greater than the first preset ratio, the ongoing event region of the second image is determined as the target event region of the first image.

[0204] In some embodiments, the key frame determination device 1000 is specifically used for:

[0205] According to different clustering strategies, clustering is performed multiple times on each sub-region in the first image to obtain each initial event region corresponding to each clustering strategy, wherein the number of clustering centers of each clustering strategy is different, and the clustering center in each initial event region is an event sub-region;

[0206] Determine a first target sub-region corresponding to the clustering strategy according to the sub-regions in each initial event region corresponding to the clustering strategy, wherein the first target sub-region is used to indicate a non-overlapping sub-region between each initial event region corresponding to the clustering strategy;

[0207] Obtain the number of targets in the first target sub-region corresponding to each clustering strategy;

[0208] The initial event regions of the clustering strategy corresponding to the largest number of targets are determined as the third event regions newly added to the first image.

[0209] In some embodiments, the key frame determination device 1000 is specifically used for:

[0210] According to different event area merging strategies, multiple event areas are selected in turn in each second event area and each third event area to obtain a set corresponding to each event area merging strategy, wherein each set includes event areas selected each time, and the number of event areas selected each time is different;

[0211] According to the event areas selected in each set, determining the merging effect parameters of the event area merging strategy corresponding to the set;

[0212] Each selected event region in the set corresponding to the maximum combined effect parameter is used as an intermediate event region, and a target event region is determined according to each second event region, each third event region and the intermediate event region.

[0213] In some embodiments, the key frame determination device 1000 is specifically used for:

[0214] Determine, according to each selected event region in the set, a second target sub-region corresponding to the set, where the second target sub-region is used to indicate a sub-region that does not overlap between each selected event region in the set, the first image includes a plurality of sub-regions, and the sub-region of the first image includes a plurality of pixel points;

[0215] According to the number of second target sub-regions corresponding to the set, a merging effect parameter corresponding to the event region merging strategy corresponding to the set is determined.

[0216] In some embodiments, the key frame determination device 1000 is specifically used for:

[0217] According to each intermediate event area, determining a fourth event area in each second event area and each third event area, wherein the fourth event area is an event area other than the intermediate event area among the second event area and the third event area;

[0218] The fourth event region is determined as the target event region of the first image.

[0219] In some embodiments, the key frame determination device 1000 is specifically used for:

[0220] In each intermediate event region, a plurality of event region groups are determined, wherein the intermediate event regions included in each event region group have overlapping sub-regions, the first image includes the plurality of sub-regions, and the sub-regions of the first image include a plurality of pixel points;

[0221] In response to the number of overlapping sub-regions between the intermediate event regions included in the event region group being greater than or equal to a preset number, determining the intermediate event region in the event region group belonging to the second image as the ongoing event region of the first image;

[0222] In response to the number of overlapping sub-regions between the intermediate event regions included in the event region group being less than a preset number, each intermediate event region of the event region group is determined as a continuing event region of the first image.

[0223] In some embodiments, the key frame determination device 1000 is specifically used for:

[0224] Determine image features and audio features of each first event region in the event segment;

[0225] splicing the image features and the audio features of the first event region to obtain splicing features of the first event region;

[0226] The concatenated features of each first event region are input into the inference model.

[0227] It should be noted that the various steps in the key frame determination method executed by the key frame determination device refer to the above embodiments for details, and will not be described in detail here.

[0228] Furthermore, based on the contents described in the above embodiments, an electronic device is also provided in an embodiment of the present application, which includes at least one processor, and a communication interface and a memory connected to the processor; wherein the communication interface is used to communicate with other communication devices, and the memory stores computer-executable instructions; the above at least one processor executes the computer-executable instructions stored in the memory to implement each step in the key frame determination method described in the above embodiments.

[0229] In order to better understand the embodiments of the present application, refer to Fig.11 , Fig.11 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0230] like Fig.11 As shown, the electronic device 1100 of this embodiment includes: a processor 1101, a memory 1102, and a communication interface 1104; wherein:

[0231] Memory 1102, used to store computer-executable instructions;

[0232] The communication interface 1104 is used to communicate with other communication devices;

[0233] The processor 1101 is used to execute the computer-executable instructions stored in the memory to implement the various steps in the query optimization method described in the above embodiment.

[0234] Optionally, the memory 1102 may be independent or integrated with the processor 1101 .

[0235] When the memory 1102 is independently provided, the device further includes a bus 1103 for connecting the memory 1102 , the communication interface 1104 and the processor 1101 .

[0236] An embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, each step in the key frame determination method described in the above embodiment is implemented.

[0237] An embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, the computer program implements the various steps in the key frame determination method described in the above embodiment.

[0238] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0239] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0240] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The above-mentioned module-composed unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0241] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0242] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the application may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0243] The memory may include high-speed memory, and may also include non-volatile storage, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0244] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0245] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0246] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A key frame determination method, characterized in that: include: Determine a target event region of a first image of a current frame in a video, wherein the target event region is used to represent an image region where a behavior event in the video ends, and the image of the video includes multiple image regions; Acquire an event segment of the target event area in the video, wherein the event segment is composed of a plurality of first event areas, each of the first event areas contains the same behavior event, and each of the first event areas contains the target event area; Acquire image features and audio features of the event segment, and infer the event segment based on the image features and the audio features to determine key frames in the video; Perform risk detection on the image corresponding to the key frame.

2. The method according to claim 1, characterized in that The step of determining a target event region of a first image of a current frame in a video includes: Acquire each second event region inherited by the first image from the second image of the previous frame, and determine each third event region newly added to the first image; A target event area of ​​the first image is determined according to each of the second event areas and each of the third event areas.

3. The method according to claim 2, characterized in that The acquiring of each second event region inherited by the first image from the second image of the previous frame includes: Determine the inter-frame mean square error and inter-frame structural similarity corresponding to the pixel points at the same position in the first image and the second image; Determine a first matrix and a second matrix for each sub-region in the first image according to the inter-frame mean square error and the inter-frame structural similarity of each pixel point, each sub-region includes a plurality of pixels, the first matrix is ​​composed of the inter-frame mean square error of each pixel point in the corresponding sub-region, and the second matrix is ​​composed of the inter-frame structural similarity of each pixel point in the corresponding sub-region; In response to the first matrix and the second matrix of the sub-region both indicating that a behavior event occurs in the sub-region, determining the sub-region as an event sub-region, and determining a first ratio between the number of the event sub-regions and the number of sub-regions in the first image; In response to the first ratio being less than or equal to a first preset ratio, in the ongoing event region of the second image, each second event region inherited by the first image from the second image is determined.

4. The method according to claim 3, characterized in that The step of determining, in the continued event area of ​​the second image, each second event area inherited by the first image from the second image comprises: Determining a second ratio between the number of event sub-regions included in the ongoing event region and the number of sub-regions included in the ongoing event region; In response to the second ratio being greater than or equal to a second preset ratio, the ongoing event region is determined as a second event region inherited by the first image from the second image.

5. The method according to claim 4, characterized in that After determining the second ratio between the number of event sub-regions included in the ongoing event region and the number of sub-regions included in the ongoing event region, the method further includes: In response to the second ratio being less than a second preset ratio, it is determined that the ongoing event region has ended, and the ongoing event region is determined as a target event region of the first image.

6. The method according to claim 3, characterized in that After determining a first ratio between the first number and a second number of sub-regions in the first image, the method further includes: In response to the first ratio being greater than a first preset ratio, the ongoing event region of the second image is determined as the target event region of the first image.

7. The method according to claim 3, characterized in that The determining each third event area newly added to the first image includes: According to different clustering strategies, clustering is performed multiple times on each sub-region in the first image to obtain each initial event region corresponding to each clustering strategy, wherein the number of clustering centers of each clustering strategy is different, and the clustering center in each initial event region is the event sub-region; Determine, according to the sub-regions in each initial event region corresponding to the clustering strategy, a first target sub-region corresponding to the clustering strategy, wherein the first target sub-region is used to indicate a non-overlapping sub-region between each initial event region corresponding to the clustering strategy; Obtaining the number of targets in the first target sub-region corresponding to each of the clustering strategies; The initial event regions of the clustering strategy corresponding to the largest number of targets are determined as the third event regions newly added to the first image.

8. The method according to claim 2, characterized in that: The step of determining the target event area of ​​the first image according to each of the second event areas and each of the third event areas comprises: According to different event area merging strategies, multiple event areas are selected in sequence from each of the second event areas and each of the third event areas to obtain a set corresponding to each of the event area merging strategies, wherein each of the sets includes event areas selected each time, and the number of event areas selected each time is different; Determining, according to the event areas selected in each of the sets, a merging effect parameter of the event area merging strategy corresponding to the set; Each selected event region in the set corresponding to the maximum combined effect parameter is used as an intermediate event region, and a target event region is determined according to each of the second event regions, each of the third event regions and the intermediate event region.

9. The method according to claim 8, characterized in that The step of determining the merging effect parameters of the event area merging strategy corresponding to each of the sets according to the event areas selected in the sets includes: Determining, according to each selected event region in the set, a second target sub-region corresponding to the set, wherein the second target sub-region is used to indicate a sub-region that does not overlap between each selected event region in the set, wherein the first image includes a plurality of sub-regions, and the sub-region of the first image includes a plurality of pixel points; According to the number of second target sub-areas corresponding to the set, a merging effect parameter corresponding to the event area merging strategy corresponding to the set is determined.

10. The method according to claim 8, characterized in that The determining of the target event area according to each of the second event areas, each of the third event areas, and the intermediate event area comprises: According to each of the intermediate event areas, determining a fourth event area in each of the second event areas and each of the third event areas, wherein the fourth event area is an event area other than the intermediate event area among the second event area and the third event area; The fourth event area is determined as a target event area of ​​the first image.

11. The method according to claim 8, characterized in that Also includes: After taking each selected event region in the set corresponding to the maximum combined effect parameter as the intermediate event region, the method further includes: In each of the intermediate event regions, a plurality of event region groups are determined, wherein the intermediate event regions included in each of the event region groups have overlapping sub-regions, the first image includes the plurality of sub-regions, and the sub-regions of the first image include a plurality of pixel points; In response to the number of overlapping sub-regions between the intermediate event regions included in the event region group being greater than or equal to a preset number, determining the intermediate event region in the event region group belonging to the second image as a continuing event region of the first image; In response to the number of overlapping sub-regions between the intermediate event regions included in the event region group being less than a preset number, each intermediate event region of the event region group is determined as a continuing event region of the first image.

12. The method according to any one of claims 1 to 11, characterized in that The reasoning about the event segment according to the image feature and the audio feature includes: Determine image features and audio features of each first event region in the event segment; splicing the image features and the audio features of the first event area to obtain splicing features of the first event area; The splicing features of each of the first event regions are input into the inference model.

13. A key frame determination device, characterized in that: include: A determination module, used to determine a target event region of a first image of a current frame in a video, wherein the target event region is used to represent an image region where a behavior event in the video ends, and the image of the video includes multiple image regions; A first acquisition module, configured to acquire an event segment of the target event region in the video, wherein the event segment is composed of a plurality of first event regions, each of which contains a same behavior event, and each of which contains the target event region; A second acquisition module is used to acquire image features and audio features of the event segment, and infer the event segment according to the image features and the audio features to determine the key frames in the video; The detection module is used to perform risk detection on the image corresponding to the key frame.

14. An electronic device, characterized in that: include: A processor, and a memory and a communication interface communicatively connected to the processor; The communication interface is used to communicate with other communication devices; The memory is used to store computer-executable instructions; The processor is used to execute the computer-executable instructions stored in the memory to implement the key frame determination method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the key frame determination method according to any one of claims 1 to 12 is implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the key frame determination method as described in any one of claims 1 to 12 is implemented.

Images