An event recognition method, an electronic device, and a computer-readable storage medium

Abstract

This application discloses an event recognition method, an electronic device, and a computer-readable storage medium. The method includes: splitting the target video stream into at least two video segments based on at least two event timestamps; inputting each video segment into a target video feature extraction network for feature extraction to obtain image dimension features of each video segment in at least two dimensions, wherein the target video feature extraction network is obtained by aligning the image dimension features and text dimension features corresponding to each video segment sample in the video stream sample; and matching the image dimension features corresponding to each video segment with the preset image dimension features corresponding to each preset video stream in a preset video feature library to obtain target event information corresponding to each video segment. This improves the generalization ability and accuracy of event recognition.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to an event recognition method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the explosive growth of video data, efficient video content recognition and retrieval technologies have become crucial. Traditional video event recognition methods typically involve directly inputting the received video stream into a recognition model for event identification, or dividing the video stream into segments according to preset time periods and then performing event recognition on the resulting video segments. However, this approach may result in the same event being divided into multiple video segments, or multiple events existing within a single video segment, affecting the accuracy of event recognition. Summary of the Invention

[0003] The main technical problem addressed by this application is to provide an event recognition method, electronic device, and computer-readable storage medium that can improve the accuracy of event recognition.

[0004] To address the aforementioned technical problems, this application provides an event recognition method comprising: splitting the target video stream into at least two video segments based on at least two event timestamps in the acquired target video stream; inputting each video segment into a target video feature extraction network for feature extraction to obtain image dimension features of each video segment in at least two dimensions, wherein the target video feature extraction network is obtained by aligning the image dimension features corresponding to each video segment sample in the video stream sample with the text dimension features corresponding to each video segment sample; and performing matching processing based on the image dimension features corresponding to each video segment with the preset image dimension features corresponding to each preset video stream in a preset video feature library to obtain target event information corresponding to each video segment.

[0005] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide an electronic device, including a memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the above-mentioned event recognition method.

[0006] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium including program data, which is used to implement the above-mentioned event recognition method when executed by a processor.

[0007] The event recognition method of this application splits the target video stream into at least two video segments based on the time of at least two events in the acquired target video stream. Each video segment is then input into a target video feature extraction network for feature extraction, yielding image dimensional features in at least two dimensions. The target video feature extraction network is obtained by aligning the image dimensional features and text dimensional features corresponding to each video segment sample in the video stream sample. The image dimensional features of each video segment are then matched with the preset image dimensional features of each preset video stream in a preset video feature library to obtain the target event information corresponding to each video segment. This method splits the target video stream according to event time, ensuring that each video segment includes only one event, reducing interference and improving the accuracy of event recognition. Furthermore, the target video feature extraction network is trained based on image-text alignment, possessing strong visual and semantic information extraction capabilities, thereby enriching image dimensional features and improving the generalization ability and accuracy of event recognition. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein: Figure 1 This is a flowchart illustrating an exemplary embodiment of the event recognition method shown in this application; Figure 2 This is a schematic diagram of an exemplary embodiment of the initial video prediction module shown in this application; Figure 3 This application illustrates a schematic diagram of an exemplary embodiment of the target video feature extraction network training process; Figure 4 This is a schematic diagram of an exemplary embodiment of the event recognition device shown in this application; Figure 5 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application; Figure 6 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0009] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are only for explaining this application and not for limiting it. Furthermore, it should be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, not all structures. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0010] First, it's important to note that a video stream may contain one or more events. Currently, a common approach is to input the entire video stream into an event recognition model for event identification, thereby identifying multiple events occurring within the stream. Alternatively, the video stream can be divided into multiple video segments based on time intervals, and event identification can be performed on each segment to determine one or more events occurring within it. However, with these methods, multiple events are prone to interference, affecting the accuracy of event identification.

[0011] Based on this, embodiments of this application propose an event recognition method, electronic device, and computer-readable storage medium. By splitting video streams into event time segments, the integrity and uniqueness of events within video clips are ensured, thereby improving the accuracy of event information recognition. For details, please refer to... Figure 1 , Figure 1 This is a flowchart illustrating an exemplary embodiment of the event recognition method shown in this application.

[0012] The execution entity of the event recognition method can be a terminal device, a server, or other processing device. The terminal device can be a user equipment (UE), computer, mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The execution entity of the event recognition method can also be an event recognition device. In some possible implementations, the event recognition method can be implemented by a processor calling computer-readable instructions stored in memory.

[0013] Specifically, the event recognition method in this embodiment includes the following steps: S110: Perform event splitting processing on the target video stream based on at least two event times in the acquired target video stream to obtain at least two video segments.

[0014] The target video stream refers to the video stream for which event recognition needs to be performed in this embodiment. For example, the target video stream can be a segment of video selected from a video collection, or it can be a video stream captured by a video capture device. The target video stream can be any video in the video collection, or it can be the video with the highest quality selected from the video collection; this embodiment does not limit this. When the target video stream belongs to the video stream captured by the video capture device, the video stream during the period of screen change can be used as the target video stream when the video capture device detects a change in the screen.

[0015] Event time is used to characterize the stages of an event. An event can include an event start time, an event midtime, and an event end time. In some embodiments, a trained start / end keyframe splitting model can be used to predict the target video stream to obtain multiple event times in the target video stream. Specifically, a set of video segment samples is obtained, and each video frame in each video segment sample includes an event time label, which includes a normal frame, an event start frame, and an event end frame. The video segment samples are then input into the start / end keyframe splitting model to be trained to obtain the trained start / end keyframe splitting model. In other embodiments, video segment samples including event time labels can also be input into a target video feature extraction network to train the target video feature extraction network, enabling the target video feature extraction network to have keyframe splitting capabilities. To improve annotation accuracy, when the detected event start frame or event end frame in a video segment sample is located between 1 / 4 and 3 / 4 of the video segment sample, the corresponding video frame is labeled as the corresponding event start frame or event end frame; otherwise, it is labeled as a normal frame.

[0016] A video segment is a video composed of partial video frames extracted from a target video stream. In some embodiments, a trained start and end keyframe splitting model can be used to predictively split the target video stream to obtain multiple video segments. In other embodiments, the video stream can be manually split based on event timestamps in the target video stream to obtain multiple video segments.

[0017] S120: Input each video segment into the target video feature extraction network for feature extraction, and obtain the image dimension features of each video segment in at least two dimensions. The target video feature extraction network is obtained by aligning the image dimension features and text dimension features corresponding to each video segment sample in the video stream sample.

[0018] A target video feature extraction network is used to extract video features. For example, during training, in addition to training with video clip samples, the network also aligns the video clip samples with corresponding text samples, thereby enabling it to extract semantic information effectively. In some embodiments, the target video feature extraction network includes a target video encoding module and a target image feature mapping module. Video clips are input into the target video encoding module for encoding to obtain video features; the video features are then input into the target image feature mapping module and mapped to multiple dimensions to obtain multi-dimensional image features.

[0019] Video stream samples are pre-acquired video streams used for training. Video segment samples are videos extracted from the video stream samples. For example, for any set containing multiple video stream samples... , Video stream samples contain For each video stream sample, a frame skipping rate is randomly selected and extracted to obtain video segment samples. Typically, the frame skipping rate is between 2 and 16.

[0020] Text dimension features are the features of the text descriptions corresponding to video clip samples across different dimensions. For example, each video clip sample can be pre-annotated to obtain its corresponding text description; then, multi-dimensional feature extraction is performed on the text descriptions to obtain the text dimension features. Text dimension features correspond to image dimension features. As an example, text and image dimensions can include one or more of motion information, appearance information, and background information dimensions, but are not limited to these dimensions.

[0021] S130: Match the image dimension features corresponding to each video segment with the preset image dimension features corresponding to each preset video stream in the preset video feature library to obtain the target event information corresponding to each video segment.

[0022] A preset video feature library is a pre-built database. It includes the correspondence between preset image dimension features and preset text dimension features corresponding to preset video streams. For example, the preset video feature library includes the correspondence between preset moving image features and preset moving text features. Exemplarily, several preset video streams can be input into a target feature extraction network to obtain several preset image dimension features; text annotation can then be performed on each preset image dimension feature to obtain the corresponding text dimension features. The preset video streams can be video streams of interest obtained from online video libraries or historical video sets captured by video capture devices. Preset image dimension features are image features extracted from preset video streams in different dimensions; each preset video stream corresponds to at least two preset image dimension features.

[0023] Target event information refers to the description of an event that occurs in a video segment. In some embodiments, the image dimension features of each video segment are matched with the preset image dimension features of each preset video stream in a preset video feature library to obtain the preset event information corresponding to each image dimension feature; the target event information of the video segment is determined based on the preset event information corresponding to each image dimension feature. In other embodiments, for a video segment, when the feature similarity between each image dimension feature of the video segment and the preset dimension feature corresponding to a preset video stream in the preset video feature library is greater than a preset threshold, the target event information is determined based on the preset event information corresponding to the preset dimension feature in the preset video stream. In other embodiments, the preset video feature library includes the correspondence between preset video streams and preset video description information. The image dimension features corresponding to the video segment are weighted and summed with the preset image dimension features of the preset video stream to obtain the matching value between the video segment and each preset video stream; when the matching value is greater than a preset matching threshold, the preset video description information of the corresponding preset video stream is determined as the target event information of the video segment.

[0024] As can be seen, the event recognition method of this application splits the target video stream into at least two video segments based on at least two event times in the acquired target video stream. Each video segment is then input into a target video feature extraction network for feature extraction, yielding image dimension features of each video segment in at least two dimensions. The target video feature extraction network is obtained by aligning the image dimension features and text dimension features corresponding to each video segment sample in the video stream sample. The image dimension features of each video segment are then matched with the preset image dimension features corresponding to each preset video stream in a preset video feature library to obtain the target event information corresponding to each video segment. This splitting of the target video stream according to event time ensures that each video segment includes only one event, reducing interference and improving the accuracy of event recognition. Furthermore, the target video feature extraction network, trained based on image-text alignment, possesses strong visual and semantic information extraction capabilities, thereby enriching image dimension features and improving the generalization and accuracy of event recognition.

[0025] In some embodiments, the event recognition device can also adjust the weights of each dimension to adjust the attention level of each dimension according to the actual application. For example, the image dimension features of each video segment are weighted according to preset weights to obtain target image dimension features of the video segment; the feature similarity between each target image dimension feature and the preset image dimension features corresponding to each preset video stream in a preset video feature library is obtained; in response to a feature similarity greater than or equal to a preset similarity threshold, the target event information of the video segment is determined based on the preset event information corresponding to the preset image dimension features. Thus, the attention level of each dimension is adaptively adjusted through weighting.

[0026] Dimensions are used to represent different types of information of interest. For example, dimensions may include, but are not limited to, motion information dimensions, appearance information dimensions, and background information dimensions. Preset weights are used to adjust the importance of each dimension. The higher the preset weight, the higher the attention given to that dimension; the lower the preset weight, the lower the attention given to that dimension.

[0027] The image dimensional features corresponding to the video segment are weighted using preset weights for each dimension to obtain the target image dimensional features. The target image dimensional features are then compared with each preset image dimensional feature to obtain several feature similarities. When a feature similarity is greater than a preset similarity threshold, the corresponding preset image dimensional feature and the target image dimensional feature are considered as a matching pair, and the preset event information corresponding to the preset image dimensional feature is determined as the initial event information corresponding to the target image dimensional feature. The target event information is determined based on the initial event information corresponding to each target image dimensional feature in the video segment.

[0028] In other embodiments, the image dimension features of each video segment are weighted and summed with the preset image dimension features corresponding to each preset video stream in a preset video feature library to obtain the matching value between the video segment and the preset video stream; the video text description information corresponding to the preset video stream is determined as the target event information of the video segment. It should be noted that the previous embodiment did not restrict the preset event information corresponding to each image dimension feature of the video segment to belong to the same preset video stream, that is, it matched each image dimension feature individually; this embodiment compares the video segment with the preset video stream one by one, requiring the image dimension features of the video segment to match the preset image dimension features of the same preset video stream, thereby obtaining the preset video stream corresponding to the video segment.

[0029] After obtaining the target event information of a video segment, to prevent the event from being split too finely into multiple events, the target event information of each video segment can be merged to obtain event information that better reflects actual needs. For example, an event merging analysis is performed on the target event information of the video segment and the target event information of its adjacent video segments. The analysis results show that there are video frames in the video segment and its adjacent video segments whose frame order differs by a preset value. In response to the event merging analysis result indicating that the target event information of the video segment and the target event information of its adjacent video segments meet the merging requirements, the video segment and its adjacent video segments, as well as the target event information of the video segment and its adjacent video segments, are merged to obtain the merged event of the merged video segment. This merges mergeable events in adjacent video segments to obtain events that better reflect reality.

[0030] Adjacent video segments refer to video segments in the target video stream whose frame order differs by a preset value. This preset value can be 1. For example, following the frame order in the target video stream from smallest to largest, the next video segment is considered an adjacent video segment. In this method, the frame order difference between the last frame of the current video segment and the first frame of the adjacent video segment is 1. Alternatively, the previous video segment can also be considered an adjacent video segment. In this method, the frame order difference between the first frame of the current video segment and the last frame of the adjacent video segment is 1. In other embodiments, video segments in the target video stream whose events occur at adjacent times can also be considered adjacent video segments in this embodiment.

[0031] As an example, following the frame order in the target video stream from smallest to largest, the event recognition device maintains a current merging object. Starting from the first video segment of the target video stream, it iterates sequentially, determining whether the target event information of the next video segment can be merged with the target event information of the video segments in the current merging object. If so, the next video segment is merged with the current merging object and their respective target event information to obtain new merged data. This new merged data is then used as the current merging object to continue judging the remaining video segments. This process continues until a video segment that cannot be merged is reached. At this point, the current merging object is resolved, and the video segment that cannot be merged is used as the new current merging object to begin a new merging attempt. For example, if the target video stream includes events A, B, C, D, and E, first determine whether events A and B can be merged. If the executing subject, execution time, and execution action type of events A and B meet the requirements, then events A and B can be merged to obtain event AB. Next, determine whether events AB can be merged with event C. If the executing subject, execution time, and execution action type of events AB and C meet the requirements, then events AB and C are merged to obtain event ABC. Then, determine whether events ABC can be merged with event D. If the executing subject, execution time, and execution action type of events AB and C do not meet the requirements, then event ABC is obtained, and starting from event D, determine whether it can be merged with event E.

[0032] The training process of the target video feature extraction network for feature extraction may include: acquiring a set of video segment samples; inputting the video segment samples into an event recognition model to obtain predicted events output by the event recognition model, the event recognition model including an initial video feature extraction network; calculating a loss value based on the difference between the predicted event and the actual event; adjusting the parameters of the initial video feature extraction network with the goal of reducing the loss value until the loss value meets the requirements, thus obtaining the target video feature extraction network. In other embodiments, the training process may also include: acquiring a set of video segment samples, the set of video segment samples including several video segment samples; training an initial video prediction module using several video segment samples to obtain a first trained initial video prediction module; training the first trained initial video prediction module and an initial image feature mapping module using several video segment samples to obtain a second trained initial video prediction module and a trained initial image feature mapping module; determining the target video feature extraction network based on the second trained initial video prediction module and the trained initial image feature mapping module. In other embodiments, the target video feature extraction network may be determined only based on the first trained initial video prediction module. Alternatively, the initial video prediction module and the initial image feature mapping module can be trained using only a few video clips to obtain the trained initial video prediction module and the trained initial image feature mapping module; the target video feature extraction network can then be determined based on the trained initial video prediction module and the trained initial image feature mapping module.

[0033] The video clip sample set includes several video clip samples. For example, video clip samples can be obtained by extracting frames from video stream samples; alternatively, video stream samples can be directly used as video clip samples.

[0034] In some embodiments, when training the initial video prediction module using several video segment samples, at least two training video frames can be obtained from the video segment samples according to a preset frame extraction method to obtain training samples. The training samples include the last training video frame and other training video frames besides the last training video frame. The other training video frames in the training samples are input into the initial video prediction module to obtain the predicted frame difference output by the initial video prediction module. The predicted frame difference represents the difference between the predicted last training video frame and its adjacent training video frames. The frame difference loss value of the loss function between the predicted frame difference output by the initial video prediction module and the actual frame difference is calculated. The actual frame difference represents the difference between the last training video frame and its adjacent training video frames. The initial video prediction module is trained with the goal of reducing the frame difference loss value to obtain the first trained initial video prediction module. Thus, using the last frame as the output supervision signal of the initial video prediction module reduces the data annotation process and improves training efficiency.

[0035] The preset frame extraction method can be to extract a preset number of video frames from a video segment according to frame order to obtain training samples. For example, it can start from the first frame of the video segment and extract training samples sequentially according to the preset frame number; alternatively, it can extract training samples from the video segment according to the frame skipping rate. The training samples include at least two training video frames, which are sequentially divided into the last training video frame and all other training video frames except the last training video frame. The last training video frame refers to the training video frame that is last in the training sample according to frame order.

[0036] Training samples refer to combinations consisting of at least two training video frames. For example, multiple training samples can be extracted from a single video clip, or a single training sample can be extracted from a single video clip.

[0037] Other training video frames from the training samples are input into the initial video prediction module to obtain the predicted frame difference output by the initial video prediction module. The predicted frame difference refers to the difference between the predicted last training video frame and the previous training video frame of the actual last training video frame. The parameters of the initial video prediction module are adjusted with the goal of reducing the loss value between the predicted frame difference and the actual frame difference until the loss value meets the requirements, thus obtaining the initial video prediction module after the first training.

[0038] As an example, see [link to relevant documentation] Figure 2 , Figure 2 This is a schematic diagram of an exemplary embodiment of the initial video prediction module shown in this application. Assuming the training samples include k+1 training video frames, the first to kth training video frames are input to the initial video prediction module. The initial video prediction module includes an initial video encoding unit and an initial video decoding unit. The first to kth training video frames are first input to the initial video encoding unit to obtain video features; the video features are then input to the initial video decoding unit to obtain the prediction frame difference. The prediction frame difference is... , This represents the predicted tail frame of the training video frame. This represents the previous training video frame after the last training frame. The initial video encoding unit can include a video embedding layer and a network of N cascaded transformer blocks. The video embedding layer can be designed as two convolutional layers: the first layer is 1*k*k, used to process spatial information, and the second layer is 2*1*1, used to process temporal information. The initial video decoding unit can include M cascaded transformer blocks and an upsampling layer. The upsampling layer can include a pixshuffle and an MLP (Multilayer Perceptron) module.

[0039] In other embodiments, the initial video prediction module further includes a classification module for obtaining predicted events of video segment samples. When the initial video prediction module is trained using several video segment samples, the video segment samples can be input into the initial video prediction module to obtain the predicted events output by the initial video prediction module. The initial video prediction module is trained with the goal of reducing the loss value between the predicted events and the actual events to obtain the first trained initial video prediction module.

[0040] In other embodiments, a distillation learning approach can be used to train the initial video prediction module. Specifically, training video frames from several video segment samples are input into the teacher video prediction module and the initial video prediction module, respectively, to obtain the teacher video features and teacher prediction frame differences output by the teacher video prediction module, and the student video features and student prediction frame differences output by the initial video prediction module. A first distillation loss value and a second distillation loss value are calculated for the loss function between the teacher video features and the student video features, and between the teacher prediction frame differences and the student prediction frame differences. The initial video prediction module is trained with the goal of minimizing the first and second distillation loss values, resulting in a first trained initial video prediction module. Thus, the initial video prediction module is used as a student model, and its training performance is improved through learning from video features and frame differences.

[0041] The teacher video prediction module is pre-trained and used to optimize the network of the initial video prediction module. Typically, the model complexity of the teacher video prediction module is higher than that of the initial video prediction module. The structures of the teacher video prediction module and the initial video prediction module can be roughly the same, such as... Figure 3 As shown in the thick solid line frame, Figure 3 This application illustrates a schematic diagram of an exemplary embodiment of the training process of a target video feature extraction network. The teacher video prediction module includes a teacher video encoding unit and a teacher video decoding unit, and the initial video prediction module includes an initial video encoding unit and an initial video decoding unit. Distillation learning is performed using the video features output by the video encoding unit and the frame differences output by the video decoding unit, respectively, thereby enabling the initial video prediction module to fully learn from the teacher video prediction module.

[0042] The same video segment sample is input into the teacher video prediction module and the initial video prediction module, respectively. This yields the teacher video features output by the teacher video encoding unit and the teacher prediction frame difference output by the teacher video decoding unit in the teacher video prediction module, as well as the student video features output by the initial video encoding unit and the student prediction frame difference output by the initial video decoding unit in the initial video prediction module. A first distillation loss value is calculated based on the difference between the teacher video features and the student video features, and a second distillation loss value is calculated based on the difference between the teacher prediction frame difference and the student prediction frame difference. The sum of the first and second distillation loss values ​​is calculated, and the parameters of the initial video encoding unit and the initial video decoding unit are adjusted with the goal of minimizing the sum of the first and second distillation loss values, resulting in the first trained initial video prediction module.

[0043] In other embodiments, the initial video prediction module can be trained with the goal of reducing either the first or second distillation loss value, resulting in a first trained initial video prediction module. In other embodiments, for the initial video prediction module, at least two training video frames can be first obtained from the video segment samples according to a preset frame extraction method to obtain training samples. The training samples include the last training video frame and other training video frames besides the last training video frame. The other training video frames in the training samples are input into the initial video prediction module to obtain the predicted frame difference output by the initial video prediction module. The frame difference loss value of the loss function between the predicted frame difference output by the initial video prediction module and the actual frame difference is calculated. The initial video prediction module is trained with the goal of reducing the frame difference loss value, resulting in the initially trained initial video prediction module. Then, several videos... The training video frames from the segment samples are input into the teacher video prediction module and the initial video prediction module after initial training, respectively, to obtain the teacher video features and teacher prediction frame difference output by the teacher video prediction module, and the student video features and student prediction frame difference output by the initial video prediction module after initial training. The first distillation loss value of the loss function between the teacher video features and the student video features, and the second distillation loss value of the loss function between the teacher prediction frame difference and the student prediction frame difference are calculated. The initial video prediction module after initial training is trained with the goal of minimizing the first distillation loss value and the second distillation loss value, resulting in the first trained initial video prediction module. This embodiment does not limit the order of the two training methods. It should be noted that the video segment sample sets used in the two training methods can be the same or different. For example, the video segment sample set for the training method is obtained by extracting frames from the video stream sample set D1, while the video segment sample set for the distillation learning training method is obtained by extracting frames from the video stream sample set D2.

[0044] After obtaining the initial video prediction module after the first training, the target video feature extraction network can be directly determined based on the initial video prediction module after the first training; alternatively, the initial video prediction module and the initial image feature mapping module after the first training can be retrained using several video segment samples to obtain the initial video prediction module and the initial image feature mapping module after the second training; the target video feature extraction network can then be determined based on the initial video prediction module and the initial image feature mapping module after the second training.

[0045] In some embodiments, after obtaining the first trained initial video prediction module, the training video frames in several video segment samples are sequentially processed by the first trained initial video coding unit and the initial image feature mapping module to obtain predicted image dimension features in at least two dimensions; the text descriptions corresponding to each training video frame in the several video segment samples are input into the text feature extraction module to obtain text dimension features in at least two dimensions; each predicted image dimension feature and each text dimension feature are aligned to obtain a prediction alignment result, which includes each predicted image dimension feature and the predicted corresponding text dimension feature; the alignment loss value of the loss function between the predicted alignment result and the actual alignment result is calculated; the first trained initial video prediction module and the initial image feature mapping module are trained with the goal of reducing the alignment loss value to obtain the second trained initial video prediction module and the trained initial image feature mapping module.

[0046] The initial image feature mapping module is used to map the initial video features output by the initial video coding unit to multiple dimensions to obtain the predicted image dimensional features in multiple dimensions.

[0047] Text annotation is performed on each video segment sample in the video segment sample set to obtain text descriptions of each video segment sample in different dimensions. These text descriptions are then input into a text feature extraction module to obtain the corresponding text dimension features. For example, text information such as the actions performed, target appearance descriptions, and video background can be annotated in video segments to obtain text descriptions of each video segment in different dimensions. For details, please refer to [link to relevant documentation]. Figure 3As shown in the thick dashed box, the text feature extraction module includes a text encoding unit and a text feature mapping unit. Text descriptions of each video segment sample in different dimensions are input into the text encoding unit for encoding processing to obtain text features of each text description. The text features of each text description are then input into the text feature mapping unit to obtain text dimension features in multiple dimensions. Video segment samples are input into the initial video encoding unit for processing to obtain initial video features. The initial video features are output to the image feature mapping unit for processing to obtain predicted image dimension features in multiple dimensions. Alignment training is performed on the text dimension features in multiple dimensions and the predicted image dimension features in multiple dimensions to obtain the second trained initial video prediction module and the trained initial image feature mapping module.

[0048] In other embodiments, the initial video prediction module may be trained directly on image-text alignment without undergoing an initial training, resulting in a trained initial video prediction module; thereby, the target video feature extraction network is determined based on the trained initial video prediction module and the trained initial image feature mapping module.

[0049] As an example, the text description of each training video frame can be a structured text description, that is, it includes the motion text description of the training video frame in the motion information dimension, the appearance text description in the appearance information dimension, and the background text description in the background information dimension. Initial feature extraction processing is performed on the text descriptions corresponding to each training video frame in several video segment samples to obtain motion text features, appearance text features, and background text features. The motion text features and appearance text features are fused to obtain motion appearance text features. The motion text features, appearance text features, and background text features are fused to obtain comprehensive text features. The motion text features, appearance text features, background text features, motion appearance text features, and comprehensive text features are used as text dimension features in at least two dimensions.

[0050] Based on this, the training video frames are processed sequentially through the first trained initial video coding unit and the initial image feature mapping module to obtain motion image features, appearance image features, background image features, motion appearance image features, and comprehensive image features.

[0051] As another example, initial feature extraction is performed on the text descriptions corresponding to each training video frame in several video segment samples to obtain motion text features, appearance text features, and background text features; the motion text features and background text features are fused to obtain motion background text features; the motion text features, appearance text features, and background text features are fused to obtain comprehensive text features; the motion text features, appearance text features, background text features, motion background text features, and comprehensive text features are used as text dimension features in at least two dimensions.

[0052] As another example, initial feature extraction is performed on the text descriptions corresponding to each training video frame in several video segment samples to obtain motion text features, appearance text features, and background text features; the appearance text features and background text features are fused to obtain appearance and background text features; the motion text features, appearance text features, and background text features are fused to obtain comprehensive text features; the motion text features, appearance text features, background text features, appearance and background text features, and comprehensive text features are used as text dimension features in at least two dimensions.

[0053] To further illustrate the application of this application in event recognition methods, the details are as follows: Training process of target video feature extraction network: from video stream sample set Each video stream sample is sampled to obtain video segment samples; then, k+1 consecutive frames are randomly selected from the video segment samples as training samples; the training video frames 1 to k are input into the initial video prediction module to obtain the predicted frame difference output by the initial video prediction module; the initial video prediction module is trained based on the difference between the predicted frame difference and the actual frame difference to obtain the initial video prediction module after initial training.

[0054] From video stream sample set Each video stream sample is sampled to obtain video segment samples; the video segment samples are input into the teacher video prediction module and the initial video prediction module after initial training, respectively; the teacher video prediction module is used to perform distillation learning on the initial video prediction module after initial training to obtain the first initial video prediction module after training.

[0055] Text annotation is performed on video clip samples to obtain text descriptions of each video clip sample in different dimensions. The annotated sample set can be represented as follows: The text description is input into the text feature extraction module to obtain multi-dimensional text dimension features. Video segment samples are sequentially passed through the first trained initial video prediction module and the initial image feature mapping module to obtain multi-dimensional predicted image dimension features. The first trained initial video prediction module and the initial image feature mapping module are aligned and trained using the text dimension features and the predicted image dimension features in each dimension to obtain the second trained initial video prediction module and the trained initial image feature mapping module. The second trained initial video coding unit in the second trained initial video prediction module is used as the target video coding unit, and the trained initial image feature mapping module is used as the target image feature mapping module. The target video coding unit and the target image feature mapping module are used to construct a target video feature extraction network.

[0056] The training process of the start and end keyframe splitting model: using the labeled sample set, it can be represented as Each video segment is labeled with its event start frame, event end frame, and normal frames, for example, {0: normal frame, 1: start frame, 2: end frame}. The aforementioned target video feature extraction network can be used as a pre-trained model to extract frames from the video segment, obtaining sampled videos. These sampled videos are then fed into the target video feature extraction network for model training, enabling the network to perform start and end keyframe splitting. In other embodiments, a separate start and end keyframe splitting model can also be created for this purpose.

[0057] The process of creating a preset video feature library is as follows: The video clip of interest is input as a preset video stream into the target video feature extraction network for feature extraction, resulting in preset image dimension features of the preset video stream; the preset image dimension features of the preset video stream are annotated with text, resulting in preset text dimension features corresponding to the preset image dimension features; and a preset video feature library is constructed based on the correspondence between the preset image dimension features of the preset video stream and the corresponding preset text dimension features.

[0058] Event recognition application process: After receiving the target video stream, the start and end keyframe splitting model can be used to split the target video stream into at least two video segments based on at least two event times. Each video segment is then input into the target video feature extraction network for feature extraction, obtaining image dimension features of each video segment in at least two dimensions. The image dimension features of each video segment are then matched with the preset image dimension features corresponding to each preset video stream in the preset video feature library to obtain the target event information corresponding to each video segment. Finally, the target event information corresponding to each video segment is merged and judged to obtain the final merged event.

[0059] Please see Figure 4 , Figure 4This is a schematic diagram of an exemplary embodiment of the event recognition device shown in this application. The event recognition device 400 includes a splitting module 410, a feature extraction module 420, and a matching module 430. The splitting module 410 is used to split the target video stream into at least two video segments based on at least two event times in the acquired target video stream. The feature extraction module 420 is used to input each video segment into a target video feature extraction network for feature extraction, obtaining image dimension features of each video segment in at least two dimensions. The target video feature extraction network is obtained by aligning the image dimension features corresponding to each video segment sample in the video stream sample with the text dimension features corresponding to each video segment sample. The matching module 430 is used to perform matching processing based on the image dimension features corresponding to each video segment and the preset image dimension features corresponding to each preset video stream in a preset video feature library, to obtain target event information corresponding to each video segment.

[0060] The above scheme involves an event recognition device that splits the target video stream into at least two video segments based on the time of at least two events. Each video segment is then input into a target video feature extraction network for feature extraction, yielding image dimensional features in at least two dimensions. This network is created by aligning the image dimensional features and text dimensional features of each video segment sample within the video stream sample. Finally, the image dimensional features of each video segment are matched with the preset image dimensional features of each preset video stream in a preset video feature library to obtain the target event information for each video segment. This method splits the target video stream according to event time, ensuring that each video segment contains only one event, reducing interference and improving event recognition accuracy. Furthermore, the target video feature extraction network, trained based on image-text alignment, possesses strong visual and semantic information extraction capabilities, thereby enriching image dimensional features and improving the generalization and accuracy of event recognition.

[0061] The functions of each module can be found in the event recognition method implementation examples, and will not be repeated here.

[0062] To implement the event recognition method of the above embodiments, this application proposes another electronic device, please refer to [link / reference needed]. Figure 5 , Figure 5 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application.

[0063] Electronic device 500 includes memory 510 and processor 520, wherein memory 510 and processor 520 are coupled together.

[0064] The memory 510 is used to store program data, and the processor 520 is used to execute the program data to implement the event recognition method of the above embodiment.

[0065] In this embodiment, processor 520 can also be referred to as CPU (Central Processing Unit). Processor 520 may be an integrated circuit chip with signal processing capabilities. Processor 520 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The general-purpose processor can be a microprocessor, or processor 520 can be any conventional processor.

[0066] This application also provides a computer-readable storage medium, such as Figure 6 As shown, the computer-readable storage medium 600 is used to store program data 610, which, when executed by a processor, is used to implement the event recognition method as described in the method embodiments of this application.

[0067] The methods involved in the event recognition method embodiments of this application, when implemented as software functional units and sold or used as independent products, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0068] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0069] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. The term "and / or" is merely a description of the association of related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, "many" in this document means two or more. In addition, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of elements, such as including at least one of A, B, and C, and may mean including any one or more elements selected from the set consisting of A, B, and C.

Claims

1. An event recognition method, characterized in that, The event recognition method includes: The target video stream is split into at least two video segments based on at least two event times in the acquired target video stream. Each video segment is input into the target video feature extraction network for feature extraction, and the image dimension features of each video segment in at least two dimensions are obtained. The target video feature extraction network is obtained by aligning the image dimension features and text dimension features of each video segment sample in the video stream sample. The target event information corresponding to each video segment is obtained by matching the image dimension features corresponding to each video segment with the preset image dimension features corresponding to each preset video stream in the preset video feature library.

2. The event recognition method according to claim 1, characterized in that, The step of matching the image dimension features corresponding to each video segment with the preset image dimension features corresponding to each preset video stream in the preset video feature library to obtain the target event information corresponding to each video segment includes: The features of each image dimension of the video segment are weighted according to the preset weights of each dimension to obtain the features of each target image dimension of the video segment. Obtain the feature similarity between the dimensional features of each target image and the dimensional features of each preset video stream in the preset video feature library; In response to the feature similarity being greater than or equal to a preset similarity threshold, the target event information of the video segment is determined based on the preset event information corresponding to the preset image dimension features.

3. The event recognition method according to claim 1, characterized in that, After the step of matching the image dimension features corresponding to each video segment with the preset image dimension features corresponding to each preset video stream in the preset video feature library to obtain the target event information corresponding to each video segment, the method further includes: The target event information of the video segment and the target event information of the adjacent video segments are analyzed by event merging to obtain the event merging analysis result. There are video frames in the video segment and the adjacent video segments whose frame order differs by a preset value. In response to the event merging analysis result indicating that the target event information of the video segment and the target event information of the adjacent video segments of the video segment meet the merging requirements, the video segment and the adjacent video segments, as well as the target event information of the video segment and the target event information of the adjacent video segments of the video segment, are merged to obtain the merged event of the merged video segment.

4. The event recognition method according to claim 1, characterized in that, Before the step of inputting the video segments into the target video feature extraction network for feature extraction to obtain image dimensional features of each video segment in at least two dimensions, the method includes: Obtain a set of video clip samples, wherein the set of video clip samples includes several video clip samples; The initial video prediction module is trained using the aforementioned video segment samples to obtain the first trained initial video prediction module. The first trained initial video prediction module and initial image feature mapping module are trained using the aforementioned video segment samples to obtain the second trained initial video prediction module and initial image feature mapping module. The target video feature extraction network is determined based on the second trained initial video prediction module and the trained initial image feature mapping module.

5. The event recognition method according to claim 4, characterized in that, The step of training the initial video prediction module using the several video segment samples to obtain the first trained initial video prediction module includes: At least two training video frames are obtained from the video segment sample according to a preset frame extraction method to obtain a training sample. The training sample includes the last training video frame and other training video frames except for the last training video frame. Other training video frames in the training samples are input into the initial video prediction module to obtain the prediction frame difference output by the initial video prediction module. The prediction frame difference represents the difference between the predicted last training video frame and the adjacent training video frames of the last training video frame. Calculate the frame difference loss value of the loss function between the predicted frame difference output by the initial video prediction module and the actual frame difference, where the actual frame difference represents the difference between the last training video frame and its adjacent training video frames. The initial video prediction module is trained with the goal of reducing the frame difference loss value, thus obtaining the first trained initial video prediction module.

6. The event recognition method according to claim 4, characterized in that, The step of training the initial video prediction module using the several video segment samples to obtain the first trained initial video prediction module includes: The training video frames from the several video segment samples are respectively input into the teacher video prediction module and the initial video prediction module to obtain the teacher video features and teacher prediction frame difference output by the teacher video prediction module, and the student video features and student prediction frame difference output by the initial video prediction module. Calculate the first distillation loss value of the loss function between the teacher video features and the student video features, and the second distillation loss value of the loss function between the teacher predicted frame difference and the student predicted frame difference; The initial video prediction module is trained with the goal of reducing the first distillation loss value and the second distillation loss value, thus obtaining the first trained initial video prediction module.

7. The event recognition method according to claim 4, characterized in that, The initial video prediction module includes an initial video encoding unit. The step of training the first trained initial video prediction module and the initial image feature mapping module using the several video segment samples to obtain a second trained initial video prediction module and a trained initial image feature mapping module includes: The training video frames in the plurality of video segment samples are processed sequentially by the first trained initial video coding unit and the initial image feature mapping module to obtain predicted image dimensional features in at least two dimensions. Input the text descriptions corresponding to each training video frame in the plurality of video segment samples into the text feature extraction module to obtain text dimension features in at least two dimensions. Align the predicted image dimension features with the predicted text dimension features to obtain the prediction alignment result, which includes the predicted image dimension features and the predicted corresponding text dimension features. Calculate the alignment loss value of the loss function between the predicted alignment result and the actual alignment result; The first trained initial video prediction module and initial image feature mapping module are trained with the goal of reducing the alignment loss value, to obtain the second trained initial video prediction module and the trained initial image feature mapping module.

8. The event recognition method according to claim 7, characterized in that, The step of inputting the preset text description corresponding to each training video frame in the plurality of video segment samples into the text feature extraction module to obtain text dimension features in at least two dimensions includes: Initial feature extraction processing is performed on the text descriptions corresponding to each training video frame in the aforementioned video segment samples to obtain motion text features, appearance text features, and background text features; The motion text features and the appearance text features are fused to obtain motion appearance text features; The motion text features, appearance text features, and background text features are fused to obtain comprehensive text features; The motion text feature, the appearance text feature, the background text feature, the motion appearance text feature, and the comprehensive text feature are used as text dimension features under the at least two dimensions.

9. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to perform the method as claimed in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, include: The system stores program data, which, when executed by a processor, is used to implement the method as described in any one of claims 1-8.

Images