Event identification method, device, electronic device and storage medium

Abstract

The present disclosure relates to an event recognition method, device, electronic device and storage medium, and relates to the field of Internet technology. The method includes: obtaining multiple frames of images and audio in a video, wherein the video is a video showing a target scene; based on the multiple frames of images, image event recognition is performed to obtain at least one first event information, each first event information is used to indicate an image event and the time point of occurrence of the image event; based on the audio, audio event recognition is performed to obtain at least one second event information, each second event information is used to indicate an audio event and the time period of occurrence of the audio event; at least one first event information and at least one second event information are fused to obtain at least one target event information, each target event information is used to indicate a target event and the time period of occurrence of the target event. The target event information obtained by this method has high accuracy while ensuring comprehensiveness.

Description

Technical Field

[0001] The present disclosure relates to the field of Internet technology, and in particular to an event recognition method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of the video industry, more and more videos are shared on various new media platforms for others to watch. However, for some long videos, there are many dull moments in the video, which are not suitable for sharing. Therefore, there is an urgent need for an event recognition method to identify the highlights in the video so as to collect the highlights. Summary of the invention

[0003] The present disclosure provides an event recognition method, device, electronic device and storage medium, wherein the target event information obtained by the method has high accuracy while ensuring comprehensiveness. The technical solution of the present disclosure is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an event recognition method, the method comprising:

[0005] Acquire multiple frames of images and audio in a video, where the video is a video showing a target scene;

[0006] Based on the multiple frames of images, image event recognition is performed to obtain at least one first event information, each first event information is used to indicate an image event and the occurrence time point of the image event;

[0007] Based on the audio, audio event recognition is performed to obtain at least one second event information, each second event information is used to indicate an audio event and an occurrence time period of the audio event;

[0008] The at least one first event information and the at least one second event information are fused to obtain at least one target event information, where each target event information is used to indicate a target event and an occurrence time period of the target event.

[0009] In some embodiments, the fusing the at least one first event information and the at least one second event information to obtain at least one target event information includes:

[0010] For each first event information, in the case where at least one audio event includes a target audio event that matches the image event indicated by the first event information, the first event information is processed to obtain the target event information, and the audio event matches the image event, which means that the audio event and the image event belong to the same event category, and the occurrence time period of the audio event matches the occurrence time point of the image event; or

[0011] For each second event information, when at least one image event includes a target image event matching the audio event indicated by the second event information, the first event information corresponding to the target image event is processed to obtain the target event information.

[0012] In some embodiments, the processing the first event information to obtain target event information includes:

[0013] Determine a target time period based on the occurrence time point of the image event indicated by the first event information, wherein the occurrence time point of the image event indicated by the first event information is a time point in the target time period;

[0014] The image event indicated by the first event information is used as the target event indicated by the target event information, and the target time period is used as the occurrence time period of the target event indicated by the target event information.

[0015] In some embodiments, the multiple frames of images are obtained from a terminal, and the image event recognition is performed based on the multiple frames of images to obtain at least one first event information, including:

[0016] In response to receiving first N frames of images among the multiple frames of images sent by the terminal, determining a scene type of the target scene, where N is an integer greater than or equal to 1;

[0017] Image event recognition is performed based on the multiple frames of images and the scene type to obtain the at least one first event information.

[0018] In some embodiments, performing image event recognition based on the multiple frames of images to obtain at least one first event information includes:

[0019] For any two adjacent frames of the multiple frames of images, the two frames of images are compared, and when the event scores of the two frames of images change, a central image of the two frames of images is obtained from the video, the timestamp of the central image of the two frames of images is the average of the timestamps of the two frames of images, and the event score is used to indicate the number of occurrences of an image event;

[0020] The central image of the two frames of images is respectively combined with any frame image of the two frames of images to form any two adjacent frames of images, and the above steps are repeated to obtain the central image until the event score of the any two adjacent frames of images formed based on the obtained central image changes, and the difference between the timestamps of the any two adjacent frames of images formed based on the obtained central image is less than the preset difference, the image event corresponding to the changed event score is used as the image event indicated by the first event information, and the average of the timestamps of the any two adjacent frames of images formed based on the obtained central image is used as the occurrence time point of the image event indicated by the first event information.

[0021] In some embodiments, the process of determining whether the event scores of the two frames of images have changed includes:

[0022] Extracting features from the two frames of images respectively to obtain first event scores and second event scores of the two frames of images;

[0023] When the second event score is different from the first event score, it is determined that the event scores of the two frames of images are changed.

[0024] In some embodiments, performing image event recognition based on the multiple frames of images to obtain at least one first event information includes:

[0025] Performing text recognition on the multiple frames of images respectively;

[0026] When the text recognition result of any frame image matches the target text, the image event corresponding to the target text is used as the image event indicated by the first event information, and the timestamp of the image is used as the occurrence time point of the image event indicated by the first event information.

[0027] In some embodiments, the video is a recorded video or a live video, the frame rate of multiple frames of images in the live video is greater than the frame rate of multiple frames of images in the recorded video, and the sampling rate of audio in the live video is greater than the sampling rate of audio in the recorded video.

[0028] In some embodiments, the event identification method further includes:

[0029] Based on the at least one target event information, at least one target sub-segment in the video is edited to obtain an edited video, wherein the start and end times of the at least one target sub-segment respectively match the occurrence time period of the target event indicated by the at least one target event information.

[0030] In some embodiments, the event identification method further includes:

[0031] The at least one target event information is sent to a terminal, the terminal is used to receive the at least one target event information, and based on the at least one target event information, at least one target sub-segment in the video is clipped to obtain a clipped video, wherein the start and end times of the at least one target sub-segment respectively match the occurrence time period of the target virtual event indicated by the at least one target event information.

[0032] According to a second aspect of an embodiment of the present disclosure, there is provided an event recognition method, the method comprising:

[0033] Extracting multiple frames of images and audio from a video, wherein the video is a video showing a target scene;

[0034] Sending the multiple frames of images and the audio to a server, the server is used to receive the multiple frames of images and the audio, perform image event recognition based on the multiple frames of images to obtain at least one first event information, perform audio event recognition based on the audio to obtain at least one second event information, and fuse the at least one first event information and the at least one second event information to obtain at least one target event information;

[0035] Each first event information is used to indicate an image event and the occurrence time point of the image event, each second event information is used to indicate an audio event and the occurrence time period of the audio event, and each target event information is used to indicate a target event and the occurrence time period of the target event.

[0036] In some embodiments, extracting multiple frames of images from a video includes:

[0037] Extracting images from the video based on a preset time interval to obtain multiple frames of original images;

[0038] The multiple frames of original images are cropped respectively to obtain the multiple frames of images, wherein the multiple frames of images are images of target detection areas of the multiple frames of original images respectively, and the target detection areas are used to determine whether an image event occurs.

[0039] In some embodiments, the event identification method further includes:

[0040] receiving the at least one target event information sent by the server;

[0041] Based on the at least one target event information, at least one target sub-segment in the video is edited to obtain an edited video, wherein the start and end times of the at least one target sub-segment respectively match the occurrence time period of the target virtual event indicated by the at least one target event information.

[0042] In some embodiments, the process of determining the at least one target sub-segment includes:

[0043] Based on the at least one target event information, determining at least one original sub-segment from the video, wherein the start and end times of the at least one original sub-segment are respectively the start and end times of the occurrence time period of the target virtual event indicated by the at least one target event information;

[0044] When the number of original sub-segments belonging to the same event category is greater than a preset number corresponding to the event category, screening out the preset number of original sub-segments from the original sub-segments belonging to the event category, and using the screened out original sub-segments as target sub-segments;

[0045] When the number of original sub-segments belonging to the same event category is not greater than a preset number corresponding to the event category, the original sub-segments of the event category are used as target sub-segments.

[0046] In some embodiments, the step of editing at least one target sub-segment in the video based on the at least one target event information includes:

[0047] Acquire a preset video template, where the preset video template is used to indicate a clipping method for the at least one target sub-segment;

[0048] Based on the preset video template, the at least one target sub-segment is mixed, rendered and provided with music to obtain the edited video.

[0049] According to a third aspect of an embodiment of the present disclosure, there is provided an event recognition device, the device comprising:

[0050] An acquisition unit is configured to acquire multiple frames of images and audio in a video, wherein the video is a video showing a target scene;

[0051] An image event recognition unit is configured to perform image event recognition based on the multiple frames of images to obtain at least one first event information, each first event information is used to indicate an image event and a time point of occurrence of the image event;

[0052] an audio event recognition unit, configured to perform audio event recognition based on the audio to obtain at least one second event information, each second event information is used to indicate an audio event and an occurrence time period of the audio event;

[0053] The information fusion unit is configured to fuse the at least one first event information and the at least one second event information to obtain at least one target event information, each target event information is used to indicate a target event and an occurrence time period of the target event.

[0054] In some embodiments, the information fusion unit is configured to, for each first event information, process the first event information to obtain target event information when multiple audio events include a target audio event that matches the image event indicated by the first event information, wherein the audio event and the image event match each other, meaning that the audio event and the image event belong to the same event category, and the time period of occurrence of the audio event matches the time point of occurrence of the image event; or, for each second event information, process the first event information corresponding to the target image event to obtain target event information when at least one image event includes a target image event that matches the audio event indicated by the second event information.

[0055] In some embodiments, the information fusion unit is configured to determine a target time period based on the occurrence time point of the image event indicated by the first event information, and the occurrence time point of the image event indicated by the first event information is a time point in the target time period;

[0056] The image event indicated by the first event information is used as the target event indicated by the target event information, and the target time period is used as the occurrence time period of the target event indicated by the target event information.

[0057] In some embodiments, the multiple frames of images are acquired from a terminal, and the image event recognition unit is configured to determine the scene type of the target scene in response to receiving the first N frames of images among the multiple frames of images sent by the terminal, where N is an integer greater than or equal to 1;

[0058] Image event recognition is performed based on the multiple frames of images and the scene type to obtain the at least one first event information.

[0059] In some embodiments, the image event recognition unit is configured to compare any two adjacent frames of the multiple frames of images, and when the event scores of the two frames of images change, obtain the central image of the two frames of images from the video, the timestamp of the central image of the two frames of images is the average of the timestamps of the two frames of images, and the event score is used to indicate the number of occurrences of the image event;

[0060] The central image of the two frames of images is respectively combined with any frame image of the two frames of images to form any two adjacent frames of images, and the above steps are repeated to obtain the central image until the event score of the any two adjacent frames of images formed based on the obtained central image changes, and the difference between the timestamps of the any two adjacent frames of images formed based on the obtained central image is less than the preset difference, the image event corresponding to the changed event score is used as the image event indicated by the first event information, and the average of the timestamps of the any two adjacent frames of images formed based on the obtained central image is used as the occurrence time point of the image event indicated by the first event information.

[0061] In some embodiments, the image event recognition unit is configured to perform feature extraction on the two frames of images respectively to obtain a first event score and a second event score for each of the two frames of images; when the second event score is different from the first event score, it is determined that the event scores of the two frames of images have changed.

[0062] In some embodiments, the image event recognition unit is configured to perform text recognition on the multiple frames of images respectively;

[0063] When the text recognition result of any frame image matches the target text, the image event corresponding to the target text is used as the image event indicated by the first event information, and the timestamp of the image is used as the occurrence time point of the image event indicated by the first event information.

[0064] In some embodiments, the video is a recorded video or a live video, the frame rate of multiple frames of images in the live video is greater than the frame rate of multiple frames of images in the recorded video, and the sampling rate of audio in the live video is greater than the sampling rate of audio in the recorded video.

[0065] In some embodiments, the event recognition device further comprises:

[0066] The first editing unit is configured to edit at least one target sub-segment in the video based on the at least one target event information to obtain an edited video, wherein the start and end times of the at least one target sub-segment respectively match the occurrence time period of the target event indicated by the at least one target event information.

[0067] In some embodiments, the event recognition device further comprises:

[0068] The second editing unit is configured to send the at least one target event information to a terminal, the terminal is used to receive the at least one target event information, and based on the at least one target event information, edit at least one target sub-segment in the video to obtain an edited video, wherein the start and end times of the at least one target sub-segment respectively match the occurrence time period of the target event indicated by the at least one target event information.

[0069] According to a fourth aspect of an embodiment of the present disclosure, there is provided an event recognition device, the device comprising:

[0070] An extraction unit is configured to extract multiple frames of images and audio from a video, wherein the video is a video showing a target scene;

[0071] a sending unit configured to send the multiple frames of images and the audio to a server, the server being used to receive the multiple frames of images and the audio, perform image event recognition based on the multiple frames of images to obtain at least one first event information, perform audio event recognition based on the audio to obtain at least one second event information, and fuse the at least one first event information and the at least one second event information to obtain at least one target event information;

[0072] Each first event information is used to indicate an image event and the occurrence time point of the image event, each second event information is used to indicate an audio event and the occurrence time period of the audio event, and each target event information is used to indicate a target event and the occurrence time period of the target event.

[0073] In some embodiments, the extraction unit is configured to extract images from the video based on a preset time interval to obtain multiple frames of original images; and to crop the multiple frames of original images respectively to obtain the multiple frames of images, wherein the multiple frames of images are respectively images of a target detection area of ​​the multiple frames of original images, and the target detection area is used to determine whether an image event occurs.

[0074] In some embodiments, the event recognition device further comprises:

[0075] an information receiving unit, configured to receive the at least one target event information sent by the server;

[0076] The editing unit is configured to edit at least one target sub-segment in the video based on the at least one target event information to obtain an edited video, wherein the start and end times of the at least one target sub-segment respectively match the occurrence time period of the target event indicated by the at least one target event information.

[0077] In some embodiments, the editing unit is further configured to determine at least one original sub-segment from the video based on the at least one target event information, the start and end time of the at least one original sub-segment being respectively the start and end time of the occurrence time period of the target event indicated by the at least one target event information; when the number of original sub-segments belonging to the same event category is greater than the preset number corresponding to the event category, the preset number of original sub-segments are filtered out from the original sub-segments belonging to the event category, and the filtered original sub-segments are used as target sub-segments; when the number of original sub-segments belonging to the same event category is not greater than the preset number corresponding to the event category, the original sub-segments of the event category are used as target sub-segments.

[0078] In some embodiments, the editing unit is configured to obtain a preset video template, which is used to indicate an editing method for the at least one target sub-segment; based on the preset video template, the at least one target sub-segment is mixed, rendered and accompanied with music to obtain the edited video.

[0079] According to a fifth aspect of an embodiment of the present disclosure, there is provided an event identification system, the system comprising a terminal and a server;

[0080] The terminal is used to extract multiple frames of images and audio from the video, and send the multiple frames of images and the audio to the server;

[0081] The server is configured to receive the multiple frames of images and the audio, perform image event recognition based on the multiple frames of images to obtain at least one first event information, perform audio event recognition based on the audio to obtain at least one second event information, and fuse the at least one first event information and the at least one second event information to obtain at least one target event information;

[0082] Each first event information is used to indicate an image event and the occurrence time point of the image event, each second event information is used to indicate an audio event and the occurrence time period of the audio event, and each target event information is used to indicate a target event and the occurrence time period of the target event.

[0083] In some embodiments, the server is used to, for each first event information, process the first event information to obtain target event information when at least one audio event includes a target audio event that matches the image event indicated by the first event information, wherein the audio event and the image event match means that the audio event and the image event belong to the same event category, and the occurrence time period of the audio event matches the occurrence time point of the image event; or, for each second event information, process the first event information corresponding to the target image event to obtain target event information when at least one image event includes a target image event that matches the audio event indicated by the second event information.

[0084] In some embodiments, the server is used to determine a target time period based on the occurrence time point of the image event indicated by the first event information, and the occurrence time point of the image event indicated by the first event information is a time point in the target time period; the image event indicated by the first event information is used as the target event indicated by the target event information, and the target time period is used as the occurrence time period of the target event indicated by the target event information.

[0085] In some embodiments, the server is used to determine the scene type of the target scene in response to receiving the first N frames of images among the multiple frames of images sent by the terminal, where N is an integer greater than or equal to 1;

[0086] Image event recognition is performed based on the multiple frames of images and the scene type to obtain the at least one first event information.

[0087] In some embodiments, the server is used to compare any two adjacent frames of the multiple images, and when the event scores of the two frames change, obtain the central image of the two frames from the video, the timestamp of the central image of the two frames is the average of the timestamps of the two frames, and the event score is used to indicate the number of occurrences of the image event; the central images of the two frames are respectively combined with any frame of the two frames to form any two adjacent frames, and the above steps are repeated to obtain the central image until the event score of the any two adjacent frames composed of the obtained central image changes, and the difference between the timestamps of the any two adjacent frames composed of the obtained central image is less than a preset difference, the image event corresponding to the changed event score is used as the image event indicated by the first event information, and the average of the timestamps of the any two adjacent frames composed of the obtained central image is used as the occurrence time point of the image event indicated by the first event information.

[0088] In some embodiments, the server is used to perform feature extraction on the two frames of images respectively to obtain a first event score and a second event score for each of the two frames of images; when the second event score is different from the first event score, it is determined that the event scores of the two frames of images have changed.

[0089] In some embodiments, the terminal is used to extract images from the video based on a preset time interval to obtain multiple frames of original images; and to crop the multiple frames of original images respectively to obtain the multiple frames of images, wherein the multiple frames of images are respectively images of a target detection area of ​​the multiple frames of original images, and the target detection area is used to determine whether an image event occurs.

[0090] In some embodiments, the server is used to perform text recognition on the multiple frames of images respectively; when the text recognition result of any frame of image matches the target text, the image event corresponding to the target text is used as the image event indicated by the first event information, and the timestamp of the image is used as the occurrence time point of the image event indicated by the first event information.

[0091] In some embodiments, the video is a recorded video or a live video, the frame rate of multiple frames of images in the live video is greater than the frame rate of multiple frames of images in the recorded video, and the sampling rate of audio in the live video is greater than the sampling rate of audio in the recorded video.

[0092] In some embodiments, the server is also used to clip at least one target sub-segment in the video based on the at least one target event information to obtain a clipped video, wherein the start and end times of the at least one target sub-segment respectively match the occurrence time period of the target event indicated by the at least one target event information.

[0093] In some embodiments, the server is further used to send the at least one target event information to the terminal; the terminal is further used to receive the at least one target event information, and based on the at least one target event information, to clip at least one target sub-segment in the video to obtain a clipped video, wherein the start and end times of the at least one target sub-segment respectively match the occurrence time period of the target event indicated by the at least one target event information.

[0094] In some embodiments, the terminal is also used to determine at least one original sub-segment from the video based on the at least one target event information, and the start and end time of the at least one original sub-segment are respectively the occurrence time period of the target event indicated by the at least one target event information; when the number of original sub-segments belonging to the same event category is greater than the preset number corresponding to the event category, the preset number of original sub-segments are filtered out from the original sub-segments belonging to the event category, and the filtered original sub-segments are used as target sub-segments; when the number of original sub-segments belonging to the same event category is not greater than the preset number corresponding to the event category, the original sub-segments of the event category are used as target sub-segments.

[0095] In some embodiments, the terminal is further used to obtain a preset video template, which is used to indicate an editing method for the at least one target sub-segment; based on the preset video template, the at least one target sub-segment is mixed, rendered and equipped with music to obtain the edited video.

[0096] According to a sixth aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:

[0097] one or more processors;

[0098] A memory for storing program codes executable by the processor;

[0099] The processor is configured to execute the program code to implement the above event identification method.

[0100] According to a seventh aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When a program code in the computer-readable storage medium is executed by a processor of an electronic device, the electronic device is enabled to perform the above-mentioned event recognition method.

[0101] According to an eighth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which implements the above-mentioned event recognition method when executed by a processor.

[0102] The disclosed embodiments provide an event recognition method, which performs image event recognition based on multiple frames of images in a video to obtain at least one first event information, and performs audio event recognition based on the audio in the video to obtain at least one second event information; since these two types of event information are obtained from different dimensions, at least one first event information and the at least one second event information are further fused to obtain at least one target event information, and the obtained target event information has high accuracy while ensuring comprehensiveness.

[0103] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0104] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0105] Figure 1 The present invention is a flowchart of an event recognition method according to an exemplary embodiment.

[0106] Figure 2 The figure is a flow chart showing another event recognition method according to an exemplary embodiment.

[0107] Figure 3 The figure is a flow chart showing another event recognition method according to an exemplary embodiment.

[0108] Figure 4 The figure is a flow chart of a binary division algorithm according to an exemplary embodiment.

[0109] Figure 5 The figure is a flow chart showing another event recognition method according to an exemplary embodiment.

[0110] Figure 6 The figure is a flow chart showing another event recognition method according to an exemplary embodiment.

[0111] Figure 7 The figure is a flow chart showing another event recognition method according to an exemplary embodiment.

[0112] Figure 8 The figure is a flow chart showing another event recognition method according to an exemplary embodiment.

[0113] Fig. 9 The present invention is a block diagram of an event recognition system according to an exemplary embodiment.

[0114] Fig.10 The present invention is a block diagram of an event recognition device according to an exemplary embodiment.

[0115] Fig.11 It is a block diagram of another event recognition device according to an exemplary embodiment.

[0116] Fig.12 It is a block diagram of a terminal according to an exemplary embodiment.

[0117] Fig.13 It is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0118] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0119] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0120] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the videos involved in this application are all obtained with full authorization.

[0121] Figure 1 is a flow chart of an event recognition method according to an exemplary embodiment. Figure 1 As shown, the method is executed by an electronic device, which is configured as at least one of a terminal and a server, and the method includes the following steps.

[0122] In step S101, multiple frames of images and audio in a video are obtained, and the video is a video showing a target scene.

[0123] In the disclosed embodiments, the video may be a recorded video or a live video. The target scene may be a game scene, a competition scene, or a virtual scene. The competition scene may be a scene of a football match, a basketball match, or other competition. The virtual scene may be a simulation environment of the real world, a semi-simulated and semi-fictional virtual environment, or a purely fictional virtual environment.

[0124] In the disclosed embodiments, the target scene displayed in the video may belong to different scene types; for example, if the target scene is a virtual scene, the scene type includes the scene type of MOBA (Multiplayer Online Battle Arena) games, the scene type of shooting games, the scene type of fighting games, and the scene type of sports games, etc.

[0125] In the embodiment of the present disclosure, the multiple frames of images are images extracted from the video based on a preset time interval, such as the preset time interval is 1 minute, 30 seconds or 10 seconds, etc. The audio is audio extracted from the video.

[0126] In step S102, image event recognition is performed based on the multiple frames of images to obtain at least one first event information, each of which is used to indicate an image event and the time point of occurrence of the image event.

[0127] In the disclosed embodiment, the image event is the performance of the target event in the image, and the target event is the event corresponding to the highlight clip in the video, which can vary according to the scene type of the target scene displayed in the video. For example, if the target scene is a virtual scene, and the scene type is a scene type of a MOBA game, the target event includes various virtual kill events, various virtual kill degree events, and virtual assist death events, etc. Various virtual kill events can be virtual events such as one kill, two kills, three kills, four kills, and five kills, and various virtual kill degree events can be virtual events such as "showing one's strength" and "sweeping the army". If the scene type is a scene type of a gunfight shooting game, the target event can be virtual events such as knockdown, kill, elimination, headshot, continuous shooting, and victory. If the scene type is a scene type of a sports game, the target event can be virtual events such as scoring, scoring, assists, etc.

[0128] In step S103, audio event recognition is performed based on the audio to obtain at least one second event information, where each second event information is used to indicate an audio event and an occurrence time period of the audio event.

[0129] In the disclosed embodiment, the audio event is the audio performance of the target event. For example, if the target event is a virtual continuous shooting event, the audio event may be the occurrence of virtual continuous gunshots.

[0130] In step S104, at least one first event information and at least one second event information are fused to obtain at least one target event information, where each target event information is used to indicate a target event and an occurrence time period of the target event.

[0131] In the disclosed embodiment, the occurrence time point of the image event and the occurrence time period of the audio event are matched, and the event categories to which the image event and the audio event belong are matched to obtain target event information.

[0132] The disclosed embodiments provide an event recognition method, which performs image event recognition based on multiple frames of images in a video to obtain at least one first event information, and performs audio event recognition based on the audio in the video to obtain at least one second event information; since these two types of event information are obtained from different dimensions, the at least one first event information and the at least one second event information are then fused to obtain at least one target event information, and the obtained target event information has high accuracy while ensuring comprehensiveness.

[0133] In some embodiments, fusing at least one first event information and at least one second event information to obtain at least one target event information includes:

[0134] For each first event information, when multiple audio events include a target audio event that matches the image event indicated by the first event information, the first event information is processed to obtain the target event information, and the audio event and image event match means that the audio event and the image event belong to the same event category, and the occurrence time period of the audio event matches the occurrence time point of the image event; or, for each second event information, when at least one image event includes a target image event that matches the audio event indicated by the second event information, the first event information corresponding to the target image event is processed to obtain the target event information.

[0135] In the disclosed embodiment, the target event information is determined only when the target event is represented in both image and audio dimensions, thereby ensuring the accuracy of the target event information.

[0136] In some embodiments, processing the first event information to obtain the target event information includes:

[0137] Determine a target time period based on the occurrence time point of the image event indicated by the first event information, wherein the occurrence time point of the image event indicated by the first event information is a time point in the target time period;

[0138] The image event indicated by the first event information is used as the target event indicated by the target event information, and the target time period is used as the occurrence time period of the target event indicated by the target event information.

[0139] In the disclosed embodiment, a target time period is obtained based on the occurrence time period of the image event, and the video within the target time period can effectively output the overall process of the image event, thereby improving the completeness and accuracy of obtaining the target event.

[0140] In some embodiments, multiple frames of images are acquired from a terminal, and image event recognition is performed based on the multiple frames of images to obtain at least one first event information, including:

[0141] In response to receiving the first N frames of images among the multiple frames of images sent by the terminal, determining the scene type of the target scene, where N is an integer greater than or equal to 1;

[0142] Image event recognition is performed based on multiple frames of images and scene types to obtain at least one first event information.

[0143] In the disclosed embodiment, since the scene types of the target scenes displayed in the video are different, the image events to be identified are also different, and thus image event identification is performed based on the scene type, which can improve the pertinence and accuracy of image event identification.

[0144] In some embodiments, image event recognition is performed based on multiple frames of images to obtain at least one first event information, including:

[0145] For any two adjacent frames of multiple frames, the two frames are compared. When the event scores of the two frames change, the central image of the two frames is obtained from the video. The timestamp of the central image of the two frames is the average of the timestamps of the two frames. The event score is used to indicate the number of occurrences of the image event.

[0146] The central images of the two frames of images are respectively combined with any frame image in the two frames of images to form any two adjacent frames of images, and the above steps are repeated to obtain the central image until the event scores of the any two adjacent frames of images formed based on the obtained central image change, and the difference between the timestamps of the any two adjacent frames of images formed based on the obtained central image is less than the preset difference, the image event corresponding to the changed event score is used as the image event indicated by the first event information, and the average of the timestamps of the any two adjacent frames of images formed based on the obtained central image is used as the occurrence time point of the image event indicated by the first event information.

[0147] In the disclosed embodiment, image event recognition is performed based on a binary algorithm, which enables filtering of images within invalid time periods in the video, thereby reducing the number of images sent to the server. This can effectively reduce the complexity of image event recognition and improve algorithm performance.

[0148] In some embodiments, the process of determining whether the event scores of two frames of images have changed includes:

[0149] Perform feature extraction on the two frames of images respectively to obtain the first event score and the second event score of the two frames of images respectively;

[0150] When the second event score is different from the first event score, it is determined that the event scores of the two frames of images are changed.

[0151] In the disclosed embodiment, event scores are obtained by performing feature extraction on images, thereby improving the efficiency of obtaining event scores.

[0152] In some embodiments, image event recognition is performed based on multiple frames of images to obtain at least one first event information, including:

[0153] Perform text recognition on multiple frames of images respectively;

[0154] When the text recognition result of any frame image matches the target text, the image event corresponding to the target text is used as the image event indicated by the first event information, and the timestamp of the image is used as the occurrence time point of the image event indicated by the first event information.

[0155] In the embodiment of the present disclosure, since the server has a longer time to perform image event recognition when the video is a live video, the image can be recognized in the entire area, thereby improving the accuracy of image event recognition.

[0156] In some embodiments, the video is a recorded video or a live video, the frame rate of multiple frames of images in the live video is greater than the frame rate of multiple frames of images in the recorded video, and the sampling rate of audio in the live video is greater than the sampling rate of audio in the recorded video.

[0157] In the embodiments of the present disclosure, the frame rate and sampling rate of multiple frames of images in a live video are improved, thereby respectively improving the accuracy of image event recognition and audio event recognition in the live video.

[0158] In some embodiments, the event identification method further includes:

[0159] Based on at least one target event information, at least one target sub-segment in the video is edited to obtain an edited video, wherein the start and end times of the at least one target sub-segment respectively match the occurrence time period of the target event indicated by the at least one target event information.

[0160] In the disclosed embodiment, since the target event information has high accuracy, editing based on the at least one target event information can obtain an edited video with high accuracy, thereby improving the accuracy of video editing.

[0161] In some embodiments, the event identification method further includes:

[0162] At least one target event information is sent to a terminal, the terminal is used to receive the at least one target event information, and based on the at least one target event information, at least one target sub-segment in the video is clipped to obtain a clipped video, wherein the start and end times of the at least one target sub-segment respectively match the occurrence time period of the target event indicated by the at least one target event information.

[0163] In the disclosed embodiment, since the target event information has high accuracy, editing based on the at least one target event information can obtain an edited video with high accuracy, thereby improving the accuracy of video editing.

[0164] Figure 2 is a flow chart of another event recognition method according to an exemplary embodiment. Figure 2 As shown, the method is executed by an electronic device and includes the following steps.

[0165] In step S201, multiple frames of images and audio are extracted from a video, where the video is a video showing a target scene.

[0166] In the embodiment of the present disclosure, images are extracted from the video based on a preset time interval to obtain the multiple frames of images; audio is extracted from the video based on a sampling rate to obtain the audio.

[0167] In step S202, the multiple frames of images and the audio are sent to a server, which is used to receive the multiple frames of images and the audio, perform image event recognition based on the multiple frames of images to obtain at least one first event information, perform audio event recognition based on the audio to obtain at least one second event information, and fuse the at least one first event information and the at least one second event information to obtain at least one target event information. Each first event information is used to indicate an image event and the time point of occurrence of the image event, each second event information is used to indicate an audio event and the time period of occurrence of the audio event, and each target event information is used to indicate a target event and the time period of occurrence of the target event.

[0168] Step S202 is similar to steps S102 - S104 and will not be described in detail here.

[0169] The disclosed embodiments provide an event recognition method, which performs image event recognition based on multiple frames of images in a video to obtain at least one first event information, and performs audio event recognition based on the audio in the video to obtain at least one second event information; since these two types of event information are obtained from different dimensions, at least one first event information and the at least one second event information are further fused to obtain at least one target event information, and the obtained target event information has high accuracy while ensuring comprehensiveness.

[0170] In some embodiments, extracting multiple frames of images from a video includes:

[0171] Extracting images from the video based on a preset time interval to obtain multiple frames of original images;

[0172] The multiple frames of original images are cropped respectively to obtain multiple frames of images, where the multiple frames of images are images of target detection areas of the multiple frames of original images, and the target detection areas are used to determine whether an image event occurs.

[0173] In the disclosed embodiment, since image event recognition can be performed based on the target detection area, the target detection area in the original image is cropped out and sent to the server, which effectively reduces the network delay caused by the terminal uploading image data to the server and improves data transmission efficiency.

[0174] In some embodiments, the event identification method further includes:

[0175] Receiving at least one target event information sent by the server;

[0176] Based on at least one target event information, at least one target sub-segment in the video is edited to obtain an edited video, wherein the start and end times of the at least one target sub-segment respectively match the occurrence time period of the target event indicated by the at least one target event information.

[0177] In the disclosed embodiment, since the target event information has high accuracy, editing based on the at least one target event information can obtain an edited video with high accuracy, thereby improving the accuracy of video editing.

[0178] In some embodiments, the process of determining at least one target sub-segment includes:

[0179] Based on at least one target event information, determining at least one original sub-segment from the video, wherein the start and end times of the at least one original sub-segment are respectively the start and end times of the occurrence time period of the target event indicated by the at least one target event information;

[0180] When the number of original sub-segments belonging to the same event category is greater than a preset number corresponding to the event category, a preset number of original sub-segments are screened out from the original sub-segments belonging to the event category, and the screened out original sub-segments are used as target sub-segments;

[0181] When the number of original sub-segments belonging to the same event category is not greater than a preset number corresponding to the event category, the original sub-segments of the event category are used as target sub-segments.

[0182] In the disclosed embodiment, by screening the original sub-segments, it is avoided that the sub-segments of multiple repeated event categories affect the viewing experience of the edited video, thereby improving the quality and accuracy of the edited video.

[0183] In some embodiments, based on at least one target event information, editing at least one target sub-segment in a video includes:

[0184] Acquire a preset video template, where the preset video template is used to indicate a clipping method for at least one target sub-segment;

[0185] Based on a preset video template, at least one target sub-segment is mixed, rendered and matched with music to obtain a clipped video.

[0186] In the disclosed embodiment, since at least one target sub-segment is mixed, rendered and spliced ​​based on a preset video template, secondary processing and creation of the target sub-segment is achieved, making the edited video richer and more exciting, thereby improving the quality of the edited video and the user's experience of watching the edited video.

[0187] Figure 3 is a flow chart of another event recognition method according to an exemplary embodiment. Figure 3 As shown, the method is described by taking the interactive execution of the terminal and the server and the video being a recorded video as an example, and includes the following steps:

[0188] In step S301, the terminal obtains a video.

[0189] In the disclosed embodiment, the video is a recorded video, which is uploaded to the terminal by a user and imported through the terminal, so that the terminal can obtain the complete video at one time.

[0190] In step S302, the terminal extracts multiple frames of images and audio from the video.

[0191] In some embodiments, the multiple frames of images are all original images, and the terminal extracts images from the video based on a preset time interval to obtain the multiple frames of images, that is, the difference between the timestamps of any two adjacent frames of images in the multiple frames of images is the preset time interval.

[0192] In some embodiments, the multiple frames of images are cropped images. Accordingly, the process of extracting multiple frames of images from the video by the terminal includes the following steps: the terminal extracts images from the video based on a preset time interval to obtain multiple frames of original images; the terminal crops the multiple frames of original images respectively to obtain multiple frames of images, the multiple frames of images are respectively images of the target detection area of ​​the multiple frames of original images, and the target detection area is used to determine whether an image event occurs.

[0193] In the disclosed embodiment, since image event recognition can be performed based on the target detection area, the target detection area in the original image is cropped out and sent to the server, which effectively reduces the network delay caused by the terminal uploading image data to the server and improves data transmission efficiency.

[0194] In the disclosed embodiment, the terminal needs to locate the target detection area before cropping multiple frames of original images, and then accurate cropping can be achieved based on the positioning result. In some embodiments, the terminal locates the target detection area based on the target area detection model installed thereon, and the target area detection model is a deep learning model.

[0195] It should be noted that if the scene type of the target scene displayed in the video is different, the position of the target detection area on the original image may be different. Therefore, before the terminal crops the multiple frames of original images, it is necessary to determine the scene type of the target scene, and then determine the target detection area model corresponding to the scene type, so as to locate the target detection area through the determined target detection area model. In the embodiments of the present disclosure, the process of determining the scene type of the target scene includes the following implementation methods:

[0196] In one implementation, the terminal determines the scene type of the target scene based on the first N frames of the multiple frames of original images. In another implementation, the terminal sends the first N frames of the multiple frames of original images to the server, and after the server receives the first N frames of original images, it determines the scene type of the target scene based on the first N frames of original images, and then sends the scene type of the target scene to the terminal. In another implementation, the terminal can also determine the scene type of the target scene based on the first N frames of audio in the audio. In another implementation, the terminal can also send the first N frames of audio in the audio to the server, and after the server receives the first N frames of audio, it determines the scene type of the target scene based on the first N frames of audio, and then sends the scene type of the target scene to the terminal, where N is an integer greater than or equal to 1.

[0197] In some embodiments, during the process of importing a video, the terminal can also extract multiple frames of images and audio from the video that have been imported, so that the importing of the video and the extraction of the images and video can be performed synchronously, thereby improving the efficiency of video processing.

[0198] In step S303, the terminal sends the multiple frames of images and the audio to the server.

[0199] In some embodiments, during the process of importing a video and extracting multiple frames of images and audio from the imported video, the terminal can also send the extracted multiple frames of images and audio to the server, so that multiple processing processes can be carried out simultaneously, thereby further improving the efficiency of video processing.

[0200] In step S304, the server receives the multiple frames of images and the audio.

[0201] In an embodiment of the present disclosure, the terminal can send the multiple frames of images and audio in batches, and accordingly, the server can receive the multiple frames of images and audio in batches. Then, in response to receiving the first N frames of images or the first N frames of audio, the server can determine the scene type of the target scene based on the first N frames of images or the first N frames of audio.

[0202] In step S305, the server performs image event recognition based on the multiple frames of images to obtain at least one first event information, where each first event information is used to indicate an image event and the time point of occurrence of the image event.

[0203] In the embodiment of the present disclosure, the scene types of the target scenes displayed in the video are different, the image events to be identified in the video are different, and thus the methods of image event identification are different. Accordingly, the process in which the server performs image event identification based on the multiple frames of images and obtains at least one first event information includes the following steps: the server determines the scene type of the target scene in response to receiving the first N frames of the multiple frames of images sent by the terminal; the server performs image event identification based on the multiple frames of images and the scene type and obtains at least one first event information.

[0204] In the disclosed embodiment, the server determines the scene type of the target scene based on the first N frames of images. Since the scene types of the target scenes displayed in the video are different, the image events to be identified are also different, and image event identification is performed based on the scene type, which can improve the pertinence and accuracy of image event identification.

[0205] In the disclosed embodiment, the server performs image event recognition through an image event recognition model. Different scene types correspond to different image event recognition models. After the server determines the scene type of the target scene, it performs image event recognition based on the multiple frames of images through the image event recognition model corresponding to the scene type to obtain at least one first event information. The image event recognition model is a deep learning model, and based on the image event recognition model, the event recognition efficiency can be effectively improved.

[0206] It should be noted that if the scene type of the target scene has been determined by the server in step S302, then in step S305, there is no need to execute the step of determining the scene type of the target scene in response to the server receiving the first N frames of images in the multi-frame image sent by the terminal. If the scene type of the target scene has been determined by the terminal in step S302, the server can directly obtain the scene type from the terminal, and there is no need to execute the step of determining the scene type of the target scene in response to the server receiving the first N frames of images in the multi-frame image sent by the terminal in step S305.

[0207] In the embodiment of the present disclosure, the server uses a binary algorithm to perform image event recognition based on multiple frames of images. Accordingly, the process of the server performing image event recognition based on the multiple frames of images to obtain at least one first event information includes the following steps:

[0208] The server compares any two adjacent frames of images in the multiple frames of images, and when the event scores of the two frames of images change, obtains the central image of the two frames of images from the video, the timestamp of the central image of the two frames of images is the average of the timestamps of the two frames of images, and the event score is used to indicate the number of occurrences of the image event; the server combines the central image of the two frames of images with any frame of the two frames of images to form any two adjacent frames of images, and repeats the above steps to obtain the central image until the event score of any two adjacent frames of images formed based on the obtained central image changes, and the difference between the timestamps of any two adjacent frames of images formed based on the obtained central image is less than the preset difference, and the image event corresponding to the changed event score is used as the image event indicated by the first event information, and the average of the timestamps of any two adjacent frames of images formed based on the obtained central image is used as the occurrence time point of the image event indicated by the first event information.

[0209] In the disclosed embodiment, the event score is used to indicate the number of occurrences of an image event, and if the event scores of two frames of images change, it indicates that the image event has occurred. For example, the event score is a KDA (Kill, Death, Assist) score, which can be used to indicate the number of occurrences of virtual kill events, virtual death events, and virtual assist events. For example, for any two adjacent frames in a multi-frame image, if the event score of the virtual kill event of the previous frame is 9, and the event score of the virtual kill event of the next frame is 10, it indicates that a virtual kill event has occurred.

[0210] In the embodiment of the present disclosure, the server extracts features from the image to obtain an event score. Accordingly, the process in which the server determines that the event scores of two frames of images have changed includes the following steps: the server extracts features from the two frames of images respectively to obtain a first event score and a second event score for each of the two frames of images; when the second event score is different from the first event score, the server determines that the event scores of the two frames of images have changed. In the embodiment of the present disclosure, by extracting features from the image to obtain the event score, the efficiency of obtaining the event score is improved.

[0211] In the disclosed embodiment, the event score is a feature of the target detection area. The multiple frames of images may be cropped images, and the process of the server extracting features from the two frames of images respectively is the process of directly extracting features from the target detection area. In this embodiment, feature extraction is directly performed on the cropped images, which improves the efficiency of feature extraction.

[0212] In other embodiments, the multiple frames of images are original images, and the server extracts features from the two frames of images respectively to obtain the first event score and the second event score of the two frames of images respectively, including the following steps: the server determines the target detection area of ​​the two frames of images respectively, extracts features from the target detection area of ​​the two frames of images respectively, and obtains the first event score and the second event score of the two frames of images respectively. In this embodiment, by determining the target area detection by the server, the powerful computing power of the server can be fully utilized, reducing the computing pressure of the terminal.

[0213] In some embodiments, the server can also directly perform text recognition (Optical Character Recognition, OCR) on the two frames of images when the two frames of images are original images or cropped images, to obtain the first event score and the second event score of the two frames of images respectively, thereby improving the convenience of obtaining event scores through text recognition.

[0214] In other embodiments, the server may first determine the target detection areas of the two frames of images respectively, and then perform text recognition on the target detection areas of the two frames of images respectively, when the two frames of images are original images, to obtain the first event score and the second event score of the two frames of images respectively. This further improves the efficiency of obtaining event scores.

[0215] It should be noted that some image events cannot be reflected by event scores, such as result events, which generally occur only once at the end time of the video. Accordingly, the terminal sends the original image at the end time to the server, and the server performs text recognition on the original image after receiving the original image, and obtains the first event information based on the text recognition result; this ensures the comprehensiveness of image event recognition.

[0216] In an embodiment of the present disclosure, if image event recognition is performed based on a binary algorithm, the terminal extracts images from the video at a relatively large preset time interval to obtain multiple frames of images. In one implementation, the terminal divides the video based on the preset time interval to obtain multiple video segments, and then the terminal extracts the starting images of the multiple video segments to obtain the multiple frames of images, that is, any two adjacent frames of images in the multiple frames of images are the start and end images of a video segment. In this way, after the terminal sends the multiple frames of images to the server, the server can identify image events in a video segment based on any two adjacent frames of images, thereby realizing image event recognition of multiple video segments based on the multiple frames of images.

[0217] In the embodiment of the present disclosure, the multiple frames of images sent by the terminal to the server are only images extracted based on a preset time interval. In order for the server to obtain the central image of any two adjacent frames of images in the multiple frames of images, it is also necessary to obtain them from the video at the terminal. Accordingly, the process of the server obtaining the central image of the two frames of images includes the following steps: the server sends a difference comparison result to the terminal, the terminal receives the difference comparison result, obtains the central image of the two frames of images based on the difference comparison result, and sends the central image of the two frames of images to the server. The difference comparison result is used to indicate that the event scores of the two frames of images have changed, and the difference comparison result also carries the timestamps of the two frames of images, thereby facilitating the acquisition of the central image of the two frames of images.

[0218] In the embodiment of the present disclosure, for any two frames of images being compared, if the event scores of the two frames of images do not change, the server will no longer perform image event recognition based on the two frames of images.

[0219] In the disclosed embodiment, image event recognition is performed based on the above-mentioned binary algorithm, which implements the filtering of images within invalid time periods in the video, thereby reducing the number of images sent to the server. This can effectively reduce the complexity of image event recognition and improve algorithm performance.

[0220] See also Figure 4 , Figure 4This is a flowchart of a binary algorithm according to an exemplary embodiment. The terminal divides the video into minute-level segments to obtain multiple minute-level video segments. Then, for each video segment, the terminal extracts two frames of images at the start and end time points of the video segment and sends them to the server; the server compares the two frames of images to determine whether the event scores of the two frames of images have changed. If not, it means that no image event has occurred in the video segment, and the processing is terminated. If so, it means that an image event has occurred in the video segment, and further identification processing is required. Then, it is determined whether the difference between the timestamps of the two frames of images is less than a preset difference. If so, the occurrence time point of the image event is determined based on the timestamps of the two frames of images. If not, the central image of the two frames of images is obtained from the terminal, and the central image is respectively combined with any frame of the two frames of images to form any two adjacent frames of images. The above steps are repeated to obtain the central image until the event score of the two adjacent frames of images formed based on the obtained central image changes, and the occurrence time point of the image event is determined based on the timestamps of the two frames of images, and the processing is terminated.

[0221] In the embodiment of the present disclosure, the first event information includes an image event tag, and the image event tag is used to indicate an image event.

[0222] In step S306, the server performs audio event recognition based on the audio to obtain at least one second event information, where each second event information is used to indicate an audio event and an occurrence time period of the audio event.

[0223] In the disclosed embodiment, the scene types of the target scenes displayed in the videos are different, the audio events to be identified in the videos are different, and thus the methods for identifying the audio events are different. Accordingly, the server performs audio event identification based on the scene types of the target scenes displayed in the audio and video to obtain at least one second event information.

[0224] In the disclosed embodiment, the server performs audio event recognition through an audio event recognition model. Different scene types correspond to different audio event recognition models. After the server determines the scene type of the target scene, it performs audio event recognition based on the audio through the audio event recognition model corresponding to the scene type to obtain at least one second event information. In the disclosed embodiment, the audio event recognition model is a deep learning model. Based on the audio event recognition model, the event recognition efficiency can be effectively improved.

[0225] In an embodiment of the present disclosure, the process of the above-mentioned server performing audio event recognition through an audio event recognition model includes the following steps: the server inputs the audio into the audio event recognition model, the audio event recognition model is used to extract audio features of the audio, and based on the audio features, performs audio event recognition to obtain a target audio event that matches the audio features, and then obtains second event information based on the audio event output by the audio event recognition model and the time period in which the audio event occurs.

[0226] It should be noted that the execution order between the above steps S305-S306 can be changed, and the above step numbers are only for the convenience of description and do not limit the execution order of the steps. Step S305 can be executed before step S306, or after step S306, or step S305 can be executed simultaneously with step S306. In the embodiment of the present disclosure, the simultaneous execution of step S304 and step S305 is used as an example for explanation, so that image event recognition and audio event recognition can be performed simultaneously, thereby improving the efficiency of event recognition.

[0227] It should be noted that if step S305 is performed before step S306, the server can directly obtain the scene type of the target scene determined in step S305 to perform audio event recognition based on the scene type. If step S305 and step S306 are performed simultaneously, and the scene type of the target scene is determined in step S303, the server can directly obtain the scene type of the target scene determined in step S303 to perform audio event recognition based on the scene type. If step S305 and step S306 are performed simultaneously, and the scene type of the target scene is not determined in step S303, then in step S306, the server also needs to determine the scene type of the target scene based on the audio, and then perform audio event recognition based on the scene type.

[0228] In the embodiment of the present disclosure, the second event information includes an audio event tag, and the audio event tag is used to indicate an audio event.

[0229] In step S307, the server fuses at least one first event information and at least one second event information to obtain at least one target event information, where each target event information is used to indicate a target event and an occurrence time period of the target event.

[0230] In the embodiment of the present disclosure, the audio event and the image event belong to the same event category, and the occurrence time period of the audio event matches the occurrence time point of the image event, which indicates that the audio event and the image event correspond to the same target event. Accordingly, the process of the server fusing at least one first event information and at least one second event information to obtain at least one target event information includes the following two implementations: In one implementation, for each first event information, the server processes the first event information to obtain the target event information when multiple audio events include a target audio event that matches the image event indicated by the first event information, and the audio event matches the image event, which means that the audio event and the image event belong to the same event category, and the occurrence time period of the audio event matches the occurrence time point of the image event. In another implementation, for each second event information, the server processes the first event information corresponding to the target image event to obtain the target event information when at least one image event includes a target image event that matches the audio event indicated by the second event information. In the embodiment of the present disclosure, the target event information is determined only when the target event is represented in both the image and audio dimensions, thereby ensuring the accuracy of determining the target event information.

[0231] In the disclosed embodiment, the occurrence of audio events is generally continuous, while the occurrence of image events is generally instantaneous. For example, for a virtual killing event, it takes a period of continuous hitting to complete the virtual killing, which will generate a period of virtual hitting audio, and the event score corresponding to the virtual killing event will only change at the moment the virtual killing is completed, that is, the score change will only occur for a moment on the image. Accordingly, in the disclosed embodiment, the occurrence time period of the audio event matches the occurrence time point of the image event, including the difference between the occurrence time point of the image event being within the occurrence time period of the audio event or exceeding the occurrence time period of the audio event being no greater than a preset threshold, which allows for some inevitable system errors and delay errors to exist, thereby improving the flexibility of event recognition.

[0232] In the disclosed embodiment, since the image event is instantaneous, the image event cannot be effectively reflected only based on the image of the instant. Accordingly, the server processes the first event information to obtain the target event information, including the following steps: the server determines the target time period based on the occurrence time point of the image event indicated by the first event information, and the occurrence time point of the image event indicated by the first event information is a time point in the target time period; the server uses the image event indicated by the first event information as the target event indicated by the target event information, and uses the target time period as the occurrence time period of the target event indicated by the target event information.

[0233] In an embodiment of the present disclosure, the server determines a target time period based on the occurrence time point of the image event indicated by the first event information, including the following steps: the server intercepts a first time period of a first length before the occurrence time point, intercepts a second time period of a second length after the occurrence time point, and determines the target time period based on the first time period and the second time period, the start and end times of the target time period being the start time of the first time period and the end time of the second time period, respectively.

[0234] In the embodiment of the present disclosure, the first duration and the second duration are determined based on the event category of the image event, and different event categories correspond to different first durations and second durations, thereby improving the flexibility of determining the target time period.

[0235] In the disclosed embodiment, a target time period is obtained based on the occurrence time period of the image event, and the video within the target time period can effectively output the overall process of the image event, thereby improving the completeness and accuracy of obtaining the target event.

[0236] In some embodiments, after obtaining at least one target event information, the server will also merge adjacent target events whose time period difference is less than a preset difference, so as to reduce the number of target events and improve the efficiency of subsequent video editing based on the target events.

[0237] In the embodiment of the present disclosure, the target event information includes a target event tag, and the target event tag is used to indicate the target event.

[0238] In step S308, the server sends the at least one target event information to the terminal.

[0239] In the disclosed embodiment, event recognition is performed through a server, and since the server has strong computing power, the efficiency of event recognition can be improved.

[0240] In step 309, the terminal receives the at least one target event information, and based on the at least one target event information, clips at least one target sub-segment in the video to obtain a clipped video, wherein the start and end times of the at least one target sub-segment respectively match the occurrence time period of the target event indicated by the at least one target event information.

[0241] In an embodiment of the present disclosure, the terminal may, based on the time sequence of the at least one target event indicated by the at least one target event information, sequentially edit the segments respectively corresponding to the at least one target event from the video.

[0242] In the disclosed embodiment, the terminal will cut out multiple sub-segments with the same event category from the video based on the at least one target event information, and the sub-segments of the repeated event category will affect the perception of the edited video. Accordingly, the terminal determines at least one target sub-segment including the following steps: the terminal determines at least one original sub-segment from the video based on the at least one target event information, and the start and end time of the at least one original sub-segment are respectively the start and end time of the occurrence time period of the target event indicated by the at least one target event information; when the number of original sub-segments belonging to the same event category is greater than the preset number corresponding to the event category, the terminal selects a preset number of original sub-segments from the original sub-segments belonging to the event category, and uses the selected original sub-segments as the target sub-segments; when the number of original sub-segments belonging to the same event category is not greater than the preset number corresponding to the event category, the terminal uses the original sub-segments of the event category as the target sub-segment.

[0243] In the disclosed embodiment, by screening the original sub-segments, it is avoided that the sub-segments of multiple repeated event categories affect the viewing experience of the edited video, thereby improving the quality and accuracy of the edited video.

[0244] In the embodiment of the present disclosure, the importance of target events of different event categories is different, so the preset numbers corresponding to different event categories are different, that is, the number of original sub-segments allowed to be retained by different event categories is different. Accordingly, after determining at least one original sub-segment, the terminal clusters the at least one original sub-segment to obtain at least one original sub-segment corresponding to multiple event categories. The terminal determines the preset scores of each of the multiple event categories, and determines the preset number corresponding to each event category based on the preset score of each event category. The preset score is used to indicate the importance of the target event of the event category, and the preset number is positively correlated with the preset score.

[0245] In some embodiments, the preset number corresponding to each event category is also positively correlated with the duration of the video, that is, the longer the duration of the video, the greater the preset number corresponding to each event category.

[0246] In the embodiment of the present disclosure, the server performs video editing based on a preset video template. Accordingly, the terminal performs video editing based on the at least one target event information on at least one target sub-segment in the video to obtain a clipped video, including the following steps: the terminal obtains a preset video template, the preset video template is used to indicate the editing method for at least one target sub-segment; the terminal performs remixing, rendering and music matching on the at least one target sub-segment based on the preset video template to obtain a clipped video.

[0247] In the embodiment of the present disclosure, the remixing refers to splicing at least one target sub-segment in a preset order. Different numbers of target sub-segments may correspond to different preset templates, and the terminal obtains the preset video template based on the number of the at least one target sub-segment. Alternatively, different scene types correspond to different preset video templates, and the terminal obtains the preset video template based on the scene type of the target scene displayed by the video.

[0248] In the disclosed embodiment, since at least one target sub-segment is mixed, rendered and spliced ​​based on a preset video template, secondary processing and creation of the target sub-segment is achieved, making the edited video richer and more exciting, thereby improving the quality of the edited video and the user's experience of watching the edited video.

[0249] See also Figure 5 , Figure 5 This is a flow chart of an event recognition method according to an exemplary embodiment. In this embodiment, first, the terminal obtains the video imported by the user, extracts the image and audio in the video, detects the KDA area in the image, and then crops the image. Then the terminal uploads the image and audio to the server through the network terminal, and the server determines the scene type of the target scene displayed by the video, and performs event recognition based on the determined scene type. In the scene type of MOBA games, KDA score recognition is performed based on the image to realize image event recognition, and virtual killing sound effects are detected based on audio to realize audio event recognition. In the scene type of gun battle shooting games, KDA score recognition is performed based on the image to realize image event recognition, and virtual gunshot sound effects are detected based on audio to realize audio event recognition. Then the image event and the audio event are fused to obtain multiple target events and the occurrence time periods of the multiple target events, and sent to the terminal. The terminal cuts out at least one target sub-segment corresponding to the multiple target events from the live video, and then mixes, renders and matches the at least one target sub-segment based on the preset video template to obtain a clipped video.

[0250] The disclosed embodiments provide an event recognition method, which performs image event recognition based on multiple frames of images in a video to obtain at least one first event information, and performs audio event recognition based on the audio in the video to obtain at least one second event information; since these two types of event information are obtained from different dimensions, the at least one first event information and the at least one second event information are then fused to obtain at least one target event information, and the obtained target event information has high accuracy while ensuring comprehensiveness; and editing is then performed based on the at least one target event information to obtain an edited video with high accuracy, thereby improving the accuracy of video editing.

[0251] Figure 6is a flow chart showing another event identification method according to an exemplary embodiment. Figure 6 As shown, the method is executed by a server and is described by taking a live video as an example, and includes the following steps:

[0252] In step S601, a video is obtained, where the video is a live data stream.

[0253] In the embodiment of the present disclosure, the live data stream sent by the receiving terminal is received to obtain the video. In the embodiment of the present disclosure, the live video may be a live video of an event.

[0254] In the embodiment of the present disclosure, the live video is segmented, that is, the live video is periodically obtained, and the live video in a period is used as a video to be edited. The following steps S602-S606 are performed on the live video in each period to obtain the edited video of the live video in each period. In the embodiment of the present disclosure, the duration of a period can be set and changed as needed, such as the duration of a period is 20 minutes.

[0255] In step S602, multiple frames of images and audio are extracted from the video.

[0256] In the embodiment of the present disclosure, an image is extracted from a video based on a preset time interval, and since the video is a live video, a frame of image is extracted at each time point corresponding to the preset time interval. Audio is extracted from a video based on a sampling rate.

[0257] In some embodiments, after acquiring the video, the video is cached to facilitate extraction of images and audio from the cached video, and the extracted multiple frames of images and audio are cached to facilitate event recognition based on the cached multiple frames of images and audio.

[0258] In step S603, image event recognition is performed based on the multiple frames of images to obtain at least one first event information, each of which is used to indicate an image event and the time point of occurrence of the image event.

[0259] In the disclosed embodiment, since the image is extracted in real time, each frame of the extracted image can be processed in real time before reaching the next cycle, so that there is a longer time for image event recognition. Accordingly, the frame rate of multiple frames of images in the live video can be increased to improve the accuracy of image event recognition. In some embodiments, the frame rate of multiple frames of images in the live video is greater than the frame rate of multiple frames of images in the recorded video, and the frame rate is negatively correlated with the preset time interval.

[0260] In the embodiment of the present disclosure, a binary algorithm may be used to perform image event recognition, and its implementation process is the same as step S305, which will not be described in detail here.

[0261] In some embodiments, a text recognition method can also be used for image recognition. Accordingly, the above process of performing image event recognition based on the multiple frames of images to obtain at least one first event information includes the following steps: when the video is a live video, text recognition is performed on the multiple frames of images respectively; when the text recognition result of any frame of image matches the target text, the image event corresponding to the target text is used as the image event indicated by the first event information, and the timestamp of the image is used as the occurrence time point of the image event indicated by the first event information.

[0262] In the disclosed embodiment, the text recognition result is used to indicate text keywords included in the image, and the target text is the text corresponding to the image event, such as kill, elimination, victory, etc.

[0263] In the embodiment of the present disclosure, since there is a longer time for image event recognition when the video is a live video, text recognition can be performed on the entire area of ​​the image, thereby improving the accuracy of image event recognition.

[0264] In step S604, audio event recognition is performed based on the audio to obtain at least one second event information, where each second event information is used to indicate an audio event and an occurrence time period of the audio event.

[0265] In the disclosed embodiments, since the audio is extracted in real time, the extracted audio can be processed in real time before the next cycle is reached, so that there is a longer time for audio event recognition. Accordingly, the sampling rate of the audio in the live video can be increased to improve the accuracy of audio event recognition. In some embodiments, the sampling rate of the audio in the live video is greater than the sampling rate of the audio in the recorded video. In some embodiments, the audio in the live video can also be extracted at a higher bit rate, and the accuracy of audio event recognition can be improved based on the audio.

[0266] It should be noted that it is generally difficult to perform an audio event recognition based on one frame of audio, so an audio event recognition can be performed based on the audio extracted during this period of time after each interval of time, thereby reducing the number of audio event recognitions and improving the efficiency of audio event recognition. In the embodiment of the present disclosure, the process of performing audio event recognition in each time period is the same as step S306, and will not be repeated here.

[0267] In step S605, at least one first event information and at least one second event information are fused to obtain at least one target event information, where each target event information is used to indicate a target event and an occurrence time period of the target event.

[0268] In the embodiment of the present disclosure, since the image and audio are extracted in real time, the first event information and the second event information obtained during each time interval can be fused to obtain the target event information during the time interval. In the embodiment of the present disclosure, the process of fusing the first event information and the second event information in each time period is the same as step S307, and will not be repeated here.

[0269] In step S606, based on the at least one target event information, at least one target sub-segment in the video is clipped to obtain a clipped video, wherein the start and end times of the at least one target sub-segment respectively match the occurrence time period of the target event indicated by the at least one target event information.

[0270] In the embodiment of the present disclosure, step S606 is the same as step S309 and will not be described in detail here.

[0271] It should be noted that since the target event information is obtained in real time, after each target event information is obtained, the original sub-segment corresponding to the target event information is cropped from the video and cached until the current time reaches the next cycle for obtaining the video.

[0272] In some embodiments, after the clip video is generated, the clip video is sent to the terminal, and a prompt message is sent to the terminal, and the prompt message is used to prompt the anchor to view and distribute the clip video, thereby achieving the purpose of facilitating consumption and live broadcast traffic. Based on the method provided in the embodiment of the present disclosure, the clip video is generated, so that while helping users to quickly consume the highlights in the live video, the clip video can be generated without human participation, saving the anchor's creation cost, and the clip video can be continuously distributed during and after the live broadcast, thereby increasing the influence of the live video. In addition, since the clip video of the live video is generated in real time, the clip video generated in this way can become a supplement to the live content in the live video, and the user can selectively switch between the live video and the clip video in the live broadcast room, which strengthens the user's content interaction in the live broadcast room. And the clip video is presented in real time in the live broadcast room, and the clip video output can become an incentive for the anchor's output, thereby promoting the anchor to continuously produce high-quality live content, forming a virtuous circle.

[0273] See also Figure 7 , Figure 7It is a flow chart of an event recognition method according to an exemplary embodiment, and the execution subject is. The method includes 4 process stages, namely, a live video acquisition stage, a target scene classification stage, a highlight recognition stage, and a highlight filming stage. In the live video acquisition stage, a real-time segment of the live video is acquired, and the image and audio are extracted from the real-time segment. In the target scene classification stage, the scene type of the target scene is determined based on the image and audio. In the scene type of MOBA games, in the highlight recognition stage, the KDA area in the image is determined, and the KDA score is identified in the KDA area to realize image event recognition, and the killing sound effect in the audio is detected to realize audio event recognition. In the scene type of gun battle shooting games, in the highlight recognition stage, text recognition is performed on the image to obtain text keywords such as kill, elimination, and victory, and the gunshot sound effect in the audio is detected to identify the audio event. Then, the image event and the audio event are fused to obtain multiple target events and the time periods of occurrence of the multiple target events. In the highlight filming stage, at least one target sub-segment corresponding to the multiple target events is cut out from the live video, and based on a preset video template, the at least one target sub-segment is mixed, rendered and equipped with music to obtain an edited video.

[0274] See also Figure 8 , Figure 8 The present invention is a flowchart of an event recognition method according to an exemplary embodiment. The method includes a data acquisition phase, a highlight moment detection phase and a creation phase. The method includes two implementation modes with different execution subjects. In the implementation mode with a terminal as the execution subject, in the data acquisition phase, the terminal acquires a recorded video imported by a user. In the highlight moment detection phase, after extracting data such as images and audio from the recorded video, the terminal sends the data to the server, obtains an event recognition result through the AI ​​(Artificial Intelligence) recognition model and strategy algorithm of the server, and the server sends the event recognition result to the terminal. The terminal cuts out a highlight segment from the recorded video based on the event recognition result. In the creation phase, the terminal cuts out the highlight segment through AI technology to obtain a cut video. In the implementation mode with a server as the execution subject, in the data acquisition phase, after the user of the terminal starts live broadcasting, the server obtains and caches the live data stream in real time. In the highlight moment detection phase, the server processes the live data stream cached locally, obtains an event recognition result through model recognition and post-processing algorithm, and cuts out a highlight segment from the recorded video based on the event recognition result. During the creation stage, the server uses AI technology to edit the highlight clips to obtain an edited video.

[0275] The method provided by the embodiment of the present disclosure is applicable to various types of videos such as recorded videos and live videos, and can be applied to various scene types, which reduces the difficulty of the algorithm, saves processing resources, and reduces processing delays. The sub-segments corresponding to the target events in the video are edited to obtain edited videos, so that users can quickly consume the wonderful content in the video, avoiding the need for users to concentrate on watching the video for a long time. In addition, the method covers a variety of scene types and can perform targeted processing on videos of various scene types respectively, thereby improving the flexibility and accuracy of video processing.

[0276] The disclosed embodiments provide an event recognition method, which performs image event recognition based on multiple frames of images in a video to obtain at least one first event information, and performs audio event recognition based on the audio in the video to obtain at least one second event information; since these two types of event information are obtained from different dimensions, at least one first event information and at least one second event information are further fused to obtain at least one target event information, and the obtained target event information has high accuracy while ensuring comprehensiveness; and editing is then performed based on the at least one target event information to obtain an edited video with high accuracy, thereby improving the accuracy of video editing.

[0277] The event recognition method provided by the embodiment of the present disclosure can be executed by an electronic device, and the electronic device is configured as at least one of a terminal and a server. Fig. 9 An event recognition system is shown according to an exemplary embodiment. The event recognition system includes a terminal 901 and a server 902. The terminal 901 and the server 902 can be directly or indirectly connected via wired or wireless communication, which is not limited by the present disclosure. In the embodiment of the present disclosure, a target application for video processing is installed on the terminal 901, and the server 902 is used to provide background services for the target application on the terminal.

[0278] Terminal 901 may be at least one of a smart phone, a smart watch, a desktop computer, a laptop, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop, and the like. Terminal 901 may generally refer to one of a plurality of terminals, and this embodiment only takes terminal 901 as an example. Those skilled in the art may appreciate that the number of the above-mentioned terminals may be more or less. For example, the above-mentioned terminals may be only a few, or the above-mentioned terminals may be dozens or hundreds, or a greater number. The embodiments of the present disclosure do not limit the number of terminals and device types.

[0279] The server 902 may be at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. The server 902 may be connected to the terminal 901 and other terminals via a wireless network or a wired network. Optionally, the number of the above servers may be more or less, which is not limited in the embodiments of the present disclosure. Of course, the server 902 may also include other functional servers to provide more comprehensive and diversified services.

[0280] Terminal 901 is used to extract multiple frames of images and audio from the video and send the multiple frames of images and audio to server 902;

[0281] The server 902 is configured to receive multiple frames of images and audio, perform image event recognition based on the multiple frames of images to obtain at least one first event information, perform audio event recognition based on the audio to obtain at least one second event information, and fuse the at least one first event information and the at least one second event information to obtain at least one target event information;

[0282] Each first event information is used to indicate an image event and the time point of occurrence of the image event, each second event information is used to indicate an audio event and the time period of occurrence of the audio event, and each target event information is used to indicate a target event and the time period of occurrence of the target event.

[0283] In some embodiments, the server 902 is used to process the first event information for each first event information, to obtain the target event information, when the multiple audio events include a target audio event that matches the image event indicated by the first event information, wherein the audio event matches the image event, which means that the audio event and the image event belong to the same event category, and the occurrence time period of the audio event matches the occurrence time point of the image event; or

[0284] For each second event information, when at least one image event includes a target image event matching the audio event indicated by the second event information, the first event information corresponding to the target image event is processed to obtain the target event information.

[0285] In some embodiments, the server 902 is used to determine the target time period based on the occurrence time point of the image event indicated by the first event information, and the occurrence time point of the image event indicated by the first event information is a time point in the target time period;

[0286] The image event indicated by the first event information is used as the target event indicated by the target event information, and the target time period is used as the occurrence time period of the target event indicated by the target event information.

[0287] In some embodiments, the server 902 is configured to determine the scene type of the target scene in response to receiving the first N frames of images among the multiple frames of images sent by the terminal 901, where N is an integer greater than or equal to 1;

[0288] Image event recognition is performed based on multiple frames of images and scene types to obtain at least one first event information.

[0289] In some embodiments, the server 902 is used to compare any two adjacent frames of images in the multiple frames of images, and when the event scores of the two frames of images change, obtain the central image of the two frames of images from the video, the timestamp of the central image of the two frames of images is the average of the timestamps of the two frames of images, and the event score is used to indicate the number of occurrences of the image event;

[0290] The central images of the two frames of images are respectively combined with any frame image in the two frames of images to form any two adjacent frames of images, and the above steps are repeated to obtain the central image until the event scores of the any two adjacent frames of images formed based on the obtained central image change, and the difference between the timestamps of the any two adjacent frames of images formed based on the obtained central image is less than the preset difference, the image event corresponding to the changed event score is used as the image event indicated by the first event information, and the average of the timestamps of the any two adjacent frames of images formed based on the obtained central image is used as the occurrence time point of the image event indicated by the first event information.

[0291] In some embodiments, the server 902 is used to extract features from the two frames of images respectively to obtain the first event score and the second event score of the two frames of images respectively;

[0292] When the second event score is different from the first event score, it is determined that the event scores of the two frames of images are changed.

[0293] In some embodiments, the terminal 901 is used to divide the video based on a preset time interval to obtain multiple frames of original images;

[0294] The multiple frames of original images are cropped respectively to obtain multiple frames of images, where the multiple frames of images are images of target detection areas of the multiple frames of original images, and the target detection areas are used to determine whether an image event occurs.

[0295] In some embodiments, the server 902 is configured to perform text recognition on multiple frames of images respectively;

[0296] When the text recognition result of any frame image matches the target text, the image event corresponding to the target text is used as the image event indicated by the first event information, and the timestamp of the image is used as the occurrence time point of the image event indicated by the first event information.

[0297] In some embodiments, the video is a recorded video or a live video, the frame rate of multiple frames of images in the live video is greater than the frame rate of multiple frames of images in the recorded video, and the sampling rate of audio in the live video is greater than the sampling rate of audio in the recorded video.

[0298] In some embodiments, the server 902 is also used to edit at least one target sub-segment in the video based on at least one target event information to obtain an edited video, wherein the start and end times of the at least one target sub-segment respectively match the occurrence time period of the target event indicated by the at least one target event information.

[0299] In some embodiments, the server 902 is further configured to send at least one target event information to the terminal 901;

[0300] Terminal 901 is used to receive at least one target event information, and based on the at least one target event information, to edit at least one target sub-segment in the video to obtain an edited video, wherein the start and end times of the at least one target sub-segment respectively match the occurrence time period of the target event indicated by the at least one target event information.

[0301] In some embodiments, terminal 901 is used to determine at least one original sub-segment from the video based on at least one target event information, and the start and end time of the at least one original sub-segment are respectively the occurrence time period of the target event indicated by the at least one target event information;

[0302] When the number of original sub-segments belonging to the same event category is greater than a preset number corresponding to the event category, a preset number of original sub-segments are screened out from the original sub-segments belonging to the event category, and the screened out original sub-segments are used as target sub-segments;

[0303] When the number of original sub-segments belonging to the same event category is not greater than a preset number corresponding to the event category, the original sub-segments of the event category are used as target sub-segments.

[0304] In some embodiments, terminal 901 is used to obtain a preset video template, where the preset video template is used to indicate a clipping method for at least one target sub-segment;

[0305] Based on a preset video template, at least one target sub-segment is mixed, rendered and matched with music to obtain a clipped video.

[0306] The disclosed embodiments provide an event recognition system. By performing image event recognition based on multiple frames of images in a video, at least one first event information is obtained. By performing audio event recognition based on the audio in the video, at least one second event information is obtained. Since these two types of event information are obtained from different dimensions, at least one first event information and at least one second event information are further fused to obtain at least one target event information. The obtained target event information has high accuracy while ensuring comprehensiveness. Then, based on the at least one target event information, editing is performed to obtain an edited video with high accuracy, thereby improving the accuracy of video editing.

[0307] One point that needs to be explained is that the event recognition system provided in the embodiment of the present application belongs to the same concept as the embodiment of the above-mentioned event recognition method. The specific process is detailed in the embodiment of the event recognition method and will not be repeated here.

[0308] Fig.10 FIG. 1 is a block diagram of an event recognition device according to an exemplary embodiment. Fig.10 , the device comprises:

[0309] The acquisition unit 1001 is configured to acquire multiple frames of images and audio in a video, where the video is a video showing a target scene;

[0310] The image event recognition unit 1002 is configured to perform image event recognition based on multiple frames of images to obtain at least one first event information, each first event information is used to indicate an image event and an occurrence time point of the image event;

[0311] The audio event recognition unit 1003 is configured to perform audio event recognition based on the audio to obtain at least one second event information, each second event information is used to indicate an audio event and an occurrence time period of the audio event;

[0312] The information fusion unit 1004 is configured to fuse at least one first event information and at least one second event information to obtain at least one target event information, where each target event information is used to indicate a target event and an occurrence time period of the target event.

[0313] In some embodiments, the information fusion unit 1004 is configured to, for each first event information, process the first event information to obtain target event information when multiple audio events include a target audio event that matches the image event indicated by the first event information, wherein the audio event and the image event match each other, meaning that the audio event and the image event belong to the same event category, and the time period of occurrence of the audio event matches the time point of occurrence of the image event; or, for each second event information, process the first event information corresponding to the target image event to obtain the target event information when at least one image event includes a target image event that matches the audio event indicated by the second event information.

[0314] In some embodiments, the information fusion unit 1004 is configured to determine the target time period based on the occurrence time point of the image event indicated by the first event information, and the occurrence time point of the image event indicated by the first event information is a time point in the target time period;

[0315] The image event indicated by the first event information is used as the target event indicated by the target event information, and the target time period is used as the occurrence time period of the target event indicated by the target event information.

[0316] In some embodiments, multiple frames of images are acquired from a terminal, and the image event recognition unit 1002 is configured to determine the scene type of the target scene in response to receiving the first N frames of images among the multiple frames of images sent by the terminal, where N is an integer greater than or equal to 1;

[0317] Image event recognition is performed based on multiple frames of images and scene types to obtain at least one first event information.

[0318] In some embodiments, the image event recognition unit 1002 is configured to compare any two adjacent frames of images in the multiple frames of images, and when the event scores of the two frames of images change, obtain the central image of the two frames of images from the video, the timestamp of the central image of the two frames of images is the average of the timestamps of the two frames of images, and the event score is used to indicate the number of occurrences of the image event;

[0319] The central images of the two frames of images are respectively combined with any frame image in the two frames of images to form any two adjacent frames of images, and the above steps are repeated to obtain the central image until the event scores of the any two adjacent frames of images formed based on the obtained central image change, and the difference between the timestamps of the any two adjacent frames of images formed based on the obtained central image is less than the preset difference, the image event corresponding to the changed event score is used as the image event indicated by the first event information, and the average of the timestamps of the any two adjacent frames of images formed based on the obtained central image is used as the occurrence time point of the image event indicated by the first event information.

[0320] In some embodiments, the image event recognition unit 1002 is configured to perform feature extraction on the two frames of images respectively to obtain the first event score and the second event score of the two frames of images respectively; when the second event score is different from the first event score, it is determined that the event scores of the two frames of images have changed.

[0321] In some embodiments, the image event recognition unit 1002 is configured to perform text recognition on multiple frames of images respectively;

[0322] When the text recognition result of any frame image matches the target text, the image event corresponding to the target text is used as the image event indicated by the first event information, and the timestamp of the image is used as the occurrence time point of the image event indicated by the first event information.

[0323] In some embodiments, the video is a recorded video or a live video, the frame rate of multiple frames of images in the live video is greater than the frame rate of multiple frames of images in the recorded video, and the sampling rate of audio in the live video is greater than the sampling rate of audio in the recorded video.

[0324] In some embodiments, the event recognition device further comprises:

[0325] The first editing unit is configured to edit at least one target sub-segment in the video based on at least one target event information to obtain an edited video, wherein the start and end times of the at least one target sub-segment respectively match the occurrence time period of the target event indicated by the at least one target event information.

[0326] In some embodiments, the event recognition device further comprises:

[0327] The second editing unit is configured to send at least one target event information to the terminal, and the terminal is used to receive the at least one target event information, and based on the at least one target event information, edit at least one target sub-segment in the video to obtain an edited video, wherein the start and end times of the at least one target sub-segment respectively match the occurrence time period of the target event indicated by the at least one target event information.

[0328] The disclosed embodiment provides an event recognition device, which obtains at least one first event information by performing image event recognition based on multiple frames of images in a video, and obtains at least one second event information by performing audio event recognition based on the audio in the video. Since these two types of event information are obtained from different dimensions, the at least one first event information and the at least one second event information are further fused to obtain at least one target event information. The obtained target event information has high accuracy while ensuring comprehensiveness, and then editing based on the at least one target event information can obtain an edited video with high accuracy, thereby improving the accuracy of video editing.

[0329] Fig.11 FIG. 1 is a block diagram of an event recognition device according to an exemplary embodiment. Fig.11 , the device comprises:

[0330] The extraction unit 1101 is configured to extract multiple frames of images and audio from a video, where the video is a video showing a target scene;

[0331] The sending unit 1102 is configured to send the multiple frames of images and audio to a server, the server is used to receive the multiple frames of images and audio, perform image event recognition based on the multiple frames of images to obtain at least one first event information, perform audio event recognition based on the audio to obtain at least one second event information, and fuse the at least one first event information and the at least one second event information to obtain at least one target event information;

[0332] Each first event information is used to indicate an image event and the time point of occurrence of the image event, each second event information is used to indicate an audio event and the time period of occurrence of the audio event, and each target event information is used to indicate a target event and the time period of occurrence of the target event.

[0333] In some embodiments, the extraction unit 1101 is configured to extract images from the video based on a preset time interval to obtain multiple frames of original images; crop the multiple frames of original images respectively to obtain multiple frames of images, and the multiple frames of images are respectively images of the target detection area of ​​the multiple frames of original images, and the target detection area is used to determine whether an image event occurs.

[0334] In some embodiments, the event recognition device further comprises:

[0335] An information receiving unit, configured to receive at least one target event information sent by the server;

[0336] The editing unit is configured to edit at least one target sub-segment in the video based on at least one target event information to obtain an edited video, wherein the start and end times of the at least one target sub-segment respectively match the occurrence time period of the target event indicated by the at least one target event information.

[0337] In some embodiments, the editing unit is further configured to determine at least one original sub-segment from the video based on at least one target event information, the start and end time of the at least one original sub-segment being respectively the start and end time of the occurrence time period of the target event indicated by the at least one target event information; when the number of original sub-segments belonging to the same event category is greater than the preset number corresponding to the event category, a preset number of original sub-segments are screened out from the original sub-segments belonging to the event category, and the screened out original sub-segments are used as target sub-segments; when the number of original sub-segments belonging to the same event category is not greater than the preset number corresponding to the event category, the original sub-segments of the event category are used as target sub-segments.

[0338] In some embodiments, the editing unit is configured to obtain a preset video template, which is used to indicate an editing method for at least one target sub-segment; based on the preset video template, at least one target sub-segment is mixed, rendered and accompanied with music to obtain an edited video.

[0339] The disclosed embodiment provides an event recognition device, which obtains at least one first event information by performing image event recognition based on multiple frames of images in a video, and obtains at least one second event information by performing audio event recognition based on the audio in the video. Since these two types of event information are obtained from different dimensions, the at least one first event information and the at least one second event information are further fused to obtain at least one target event information. The obtained target event information has high accuracy while ensuring comprehensiveness, and then editing based on the at least one target event information can obtain an edited video with high accuracy, thereby improving the accuracy of video editing.

[0340] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0341] Fig.12 It is a block diagram of a terminal 1200 according to an exemplary embodiment. Fig.12 The structure block diagram of a terminal 1200 provided by an exemplary embodiment of the present disclosure is shown. Generally, the terminal 1200 includes: a processor 1201 and a memory 1202 .

[0342] The processor 1201 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1201 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1201 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1201 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1201 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0343] The memory 1202 may include one or more computer-readable storage media, which may be non-transitory. The memory 1202 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1202 is used to store at least one program code, which is used to be executed by the processor 1201 to implement the event recognition method provided by the method embodiment of the present disclosure.

[0344] In some embodiments, the terminal 1200 may further optionally include: a peripheral device interface 1203 and at least one peripheral device. The processor 1201, the memory 1202 and the peripheral device interface 1203 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1203 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1204, a display screen 1205, a camera assembly 1206, an audio circuit 1207 and a power supply 1208.

[0345] The peripheral device interface 1203 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1201 and the memory 1202. In some embodiments, the processor 1201, the memory 1202, and the peripheral device interface 1203 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1201, the memory 1202, and the peripheral device interface 1203 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0346] The radio frequency circuit 1204 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1204 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1204 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1204 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1204 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1204 may also include circuits related to NFC (Near Field Communication), which is not limited in the present disclosure.

[0347] The display screen 1205 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1205 is a touch display screen, the display screen 1205 also has the ability to collect touch signals on the surface or above the surface of the display screen 1205. The touch signal can be input to the processor 1201 as a control signal for processing. At this time, the display screen 1205 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1205 can be one, and the front panel of the terminal 1200 is set; in other embodiments, the display screen 1205 can be at least two, which are respectively set on different surfaces of the terminal 1200 or are folded; in some other embodiments, the display screen 1205 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal 1200. Even, the display screen 1205 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1205 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0348] The camera assembly 1206 is used to capture images or videos. Optionally, the camera assembly 1206 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1206 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0349] The audio circuit 1207 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1201 for processing, or input them into the radio frequency circuit 1204 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 1200. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1201 or the radio frequency circuit 1204 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1207 may also include a headphone jack.

[0350] The power supply 1208 is used to power various components in the terminal 1200. The power supply 1208 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1208 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0351] Those skilled in the art will understand that Fig.12 The structure shown in the figure does not constitute a limitation on the terminal 1200, and the terminal 1200 may include more or fewer components than those shown in the figure, or combine certain components, or adopt a different component arrangement.

[0352] Fig.13 It is a structural diagram of a server provided according to an embodiment of the present application. The server 1300 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1301 and one or more memories 1302, wherein the memory 1302 is used to store executable program codes, and the processor 1301 is configured to execute the above executable program codes to implement the event recognition methods provided by the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the server may also include other components for implementing device functions, which will not be repeated here.

[0353] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the instructions can be executed by a processor of a terminal to complete the above event identification method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0354] In an exemplary embodiment, a computer program product is also provided, including a computer program, which implements the above event identification method when executed by a processor.

[0355] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0356] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An event recognition method, characterized in that: The method comprises: Periodically acquiring a video and caching the acquired video, wherein the video includes a live data stream showing a target scene; Extracting multiple frames of images and audio from the video, caching the extracted multiple frames of images and audio, and processing the extracted audio and each frame of the image in real time before reaching the next cycle; Determining a scene type of the target scene based on the multiple frames of images; Based on the multiple frames of images, image event recognition is performed by using an image event recognition model corresponding to the scene type to obtain at least one first event information, each first event information is used to indicate an image event and a time point of occurrence of the image event, the image event is a manifestation of a target event in an image, and varies according to the scene type, and different scene types correspond to different image event recognition models; The audio is input into an audio event recognition model corresponding to the scene type, so that the audio event recognition model performs audio event recognition based on the audio features of the audio, and obtains a target audio event matching the audio features, and different scene types correspond to different audio event recognition models; based on the audio event output by the audio event recognition model and the time period of occurrence of the audio event, at least one second event information is obtained, each second event information is used to indicate an audio event and the time period of occurrence of the audio event, and the audio event is the audio performance of the target event, and the image event recognition and the audio event recognition are performed simultaneously; The at least one first event information and the at least one second event information are merged to obtain at least one target event information, where each target event information is used to indicate a target event and an occurrence time period of the target event; Based on the at least one target event information, determining at least one original sub-segment from the video, wherein the start and end times of the at least one original sub-segment are respectively the start and end times of the occurrence time period of the target event indicated by the at least one target event information; Clustering the at least one original sub-segment to determine at least one original sub-segment corresponding to each of the plurality of event categories; For any event category, when the number of original sub-segments belonging to the event category is greater than the preset number corresponding to the event category, the preset number of original sub-segments are screened out from the original sub-segments belonging to the event category, and the screened preset number of original sub-segments are used as target sub-segments; when the number of original sub-segments belonging to the event category is not greater than the preset number corresponding to the event category, the original sub-segments belonging to the event category are used as target sub-segments; the preset number corresponding to the event category is positively correlated with the importance of the target event of the event category, and the preset number corresponding to the event category is positively correlated with the duration of the video; at least one target sub-segment is cached until the current time reaches the next cycle of acquiring the video; Based on the number of the at least one target sub-segment, a preset video template is obtained, where the preset video template is used to indicate the editing method of the at least one target sub-segment, and different numbers of target sub-segments correspond to different preset templates; based on the preset video template, the at least one target sub-segment is edited to obtain an edited video.

2. The event recognition method according to claim 1, characterized in that: The fusing the at least one first event information and the at least one second event information to obtain at least one target event information includes: For each first event information, in the case where at least one audio event includes a target audio event that matches the image event indicated by the first event information, the first event information is processed to obtain the target event information, and the audio event matches the image event means that the audio event and the image event belong to the same event category, and the occurrence time period of the audio event matches the occurrence time point of the image event; or For each second event information, when at least one image event includes a target image event matching the audio event indicated by the second event information, the first event information corresponding to the target image event is processed to obtain the target event information.

3. The event recognition method according to claim 2, characterized in that: The processing of the first event information corresponding to the target image event to obtain the target event information includes: Determine a target time period based on the occurrence time point of the image event indicated by the first event information, wherein the occurrence time point of the image event indicated by the first event information is a time point in the target time period; The image event indicated by the first event information is used as the target event indicated by the target event information, and the target time period is used as the occurrence time period of the target event indicated by the target event information.

4. The event recognition method according to claim 1, characterized in that: The multiple frames of images are obtained from a terminal, and the method further includes: In response to receiving the first N frames of images among the multiple frames of images sent by the terminal, the scene type of the target scene is determined, where N is an integer greater than or equal to 1.

5. The event recognition method according to claim 1, characterized in that: The performing image event recognition based on the multiple frames of images by using an image event recognition model corresponding to the scene type to obtain at least one first event information includes: For any two adjacent frames of the multiple frames of images, the two frames of images are compared, and when the event scores of the two frames of images change, a central image of the two frames of images is obtained from the video, the timestamp of the central image of the two frames of images is the average of the timestamps of the two frames of images, and the event score is used to indicate the number of occurrences of an image event; The central image of the two frames of images is respectively combined with any frame image of the two frames of images to form any two adjacent frames of images, and the above steps are repeated to obtain the central image until the event score of the any two adjacent frames of images formed based on the obtained central image changes, and the difference between the timestamps of the any two adjacent frames of images formed based on the obtained central image is less than the preset difference, the image event corresponding to the changed event score is used as the image event indicated by the first event information, and the average of the timestamps of the any two adjacent frames of images formed based on the obtained central image is used as the occurrence time point of the image event indicated by the first event information.

6. The event recognition method according to claim 1, characterized in that: The method further comprises: Performing text recognition on the multiple frames of images respectively; When the text recognition result of any frame image matches the target text, the image event corresponding to the target text is used as the image event indicated by the first event information, and the timestamp of the image is used as the occurrence time point of the image event indicated by the first event information.

7. The event recognition method according to claim 1, characterized in that: The video also includes a recorded video, the frame rate of multiple frames of images in the live video is greater than the frame rate of multiple frames of images in the recorded video, and the sampling rate of the audio in the live video is greater than the sampling rate of the audio in the recorded video.

8. An event recognition device, characterized in that: include: an acquisition unit, configured to periodically acquire a video and cache the acquired video, wherein the video includes a live data stream showing a target scene; Extracting multiple frames of images and audio from the video; A module configured to perform the following steps: caching the extracted multiple frames of images and audio, and processing the extracted audio and each frame of image in real time before reaching the next cycle; an image event recognition unit configured to determine the scene type of the target scene based on the multiple frames of images; perform image event recognition based on the multiple frames of images by using an image event recognition model corresponding to the scene type to obtain at least one first event information, each first event information being used to indicate an image event and a time point of occurrence of the image event, the image event being an image representation of the target event, varying according to the scene type, and different scene types corresponding to different image event recognition models; An audio event recognition unit is configured to input the audio into an audio event recognition model corresponding to the scene type, so that the audio event recognition model performs audio event recognition based on the audio features of the audio, and obtains a target audio event matching the audio features, and different scene types correspond to different audio event recognition models; based on the audio event output by the audio event recognition model and the time period of occurrence of the audio event, obtain at least one second event information, each second event information is used to indicate an audio event and the time period of occurrence of the audio event, the audio event is the audio performance of the target event, and the image event recognition and the audio event recognition are performed simultaneously; An information fusion unit is configured to fuse the at least one first event information and the at least one second event information to obtain at least one target event information, each target event information is used to indicate a target event and an occurrence time period of the target event; The editing unit is configured as follows: Based on the at least one target event information, determining at least one original sub-segment from the video, wherein the start and end times of the at least one original sub-segment are respectively the start and end times of the occurrence time period of the target event indicated by the at least one target event information; Clustering the at least one original sub-segment to determine at least one original sub-segment corresponding to each of the plurality of event categories; For any event category, when the number of original sub-segments belonging to the event category is greater than the preset number corresponding to the event category, the preset number of original sub-segments are screened out from the original sub-segments belonging to the event category, and the screened preset number of original sub-segments are used as target sub-segments; when the number of original sub-segments belonging to the event category is not greater than the preset number corresponding to the event category, the original sub-segments belonging to the event category are used as target sub-segments; The preset number corresponding to the event category is positively correlated with the importance of the target event of the event category, and the preset number corresponding to the event category is positively correlated with the duration of the video; Based on the number of the at least one target sub-segment, a preset video template is obtained, where the preset video template is used to indicate a way of editing the at least one target sub-segment, and different numbers of target sub-segments correspond to different preset templates; based on the preset video template, the at least one target sub-segment is edited to obtain an edited video; The module is configured to perform the following steps: buffering at least one target sub-segment until the current time reaches a next cycle of acquiring a video.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the event recognition method as described in any one of claims 1 to 7. 10 . A computer-readable storage medium, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the event recognition method according to any one of claims 1 to 7.

Images