Screen display video content identification method and device, electronic equipment and storage medium

Abstract

The invention provides a screen display video content identification method and device, electronic equipment and a storage medium, and belongs to the technical field of video analysis, and the method comprises the steps: calculating a difference value between two adjacent frame images in a screen display video; based on each difference value, extracting a key frame from the screen display video; and performing event category identification based on the key frame, and generating a content report of the screen display video. According to the screen display video content identification method provided by the invention, the key frame is effectively extracted by calculating the difference value of the adjacent frame images in the screen display video, high-value information can be efficiently positioned from a large amount of video data, then event category identification is carried out by utilizing the extracted key frame, and finally the content report of the screen display video is generated. The intelligent analysis and rapid understanding of the screen display video content are realized, the automation level and processing efficiency of screen display video content identification are improved, and the video information mining speed and depth can be improved to a great extent when the screen display video content is analyzed subsequently.

Description

Technical Field

[0001] The present invention relates to the field of video analysis technology, and in particular to a method, device, electronic device and storage medium for identifying screen-displayed video content. Background Art

[0002] In today's rapidly developing world, electronic information technology has permeated every aspect of flight training, from situational awareness to intelligence processing. Aircraft on-screen video is a product of this technology. By identifying the content of on-screen video, key events can be quickly located, significantly improving the efficiency of flight training review and debriefing.

[0003] Existing on-screen video analysis methods mainly rely on manpower. Faced with numerous on-screen videos, they can only analyze them one by one, making it difficult to quickly locate key event nodes in the video, and the efficiency is low. Summary of the Invention

[0004] The present invention provides a method, device, electronic device and storage medium for identifying the content of screen-displayed videos, which are used to solve the problem of low efficiency in the prior art of manually analyzing screen-displayed videos one by one.

[0005] In a first aspect, the present invention provides a method for identifying on-screen video content, comprising: Calculate the difference between two adjacent frame images in the screen display video; extracting key frames from the screen-displayed video based on the difference values; Event categories are identified based on the key frames, and a content report of the screen-displayed video is generated.

[0006] In one embodiment, calculating the difference between two adjacent frame images in the screen-displayed video includes: Calculating the difference between the pixel values ​​of the two adjacent frame images according to the same coordinate position to obtain the pixel difference value corresponding to each coordinate position; The pixel difference values ​​corresponding to each coordinate position are summed to obtain the difference value between the two adjacent frame images.

[0007] In one embodiment, extracting key frames from the screen-displayed video based on the difference values ​​includes: Determine two frame images used for calculating a target difference value as a segmentation point; the target difference value is a difference value greater than a preset difference threshold among the difference values; dividing the screen-displayed video into at least one segment according to the segmentation point; At least one image frame is extracted from each of the segments to generate a key frame for each of the segments.

[0008] In one embodiment, the performing of event category recognition based on the key frames and generating a content report of the screen-displayed video includes: Inputting each of the key frames into an event recognition model in timeline order to obtain an event category recognition result corresponding to each of the key frames output by the event recognition model; the event category includes one or a combination of any two or more of aircraft takeoff, aircraft climb, aircraft landing, radar capture, and instrument failure; Based on the event category identification results, a content report of the screen display video is generated.

[0009] In one embodiment, generating a content report of the screen-displayed video based on each of the event category identification results includes: If the event category recognition result corresponding to any key frame is that the event category is not recognized, then the key frame is removed; If the event category identification result corresponding to any key frame is the event category corresponding to the key frame, a content report of the screen display video is generated according to the timestamps and corresponding event categories of each key frame.

[0010] In one embodiment, generating a content report of the screen-displayed video based on the timestamp of each key frame and the corresponding event category includes: If two adjacent key frames in each of the key frames belong to the same event category, determining the time period between the timestamps of the two adjacent key frames as the event occurrence time period, and updating the content reports of the two adjacent key frames according to the event occurrence time period and the event categories corresponding to the two adjacent key frames; The content reports of the key frames are integrated into the content report of the screen display video according to the timeline sequence.

[0011] In one embodiment, the event recognition model is trained in the following manner: Get key frame image samples; Preprocessing the key frame image samples to obtain target frame image samples; Performing event category labeling on the target frame image sample to obtain label data corresponding to the target frame image sample; Model training is performed based on the target frame image samples and their label data to obtain the event recognition model.

[0012] In a second aspect, the present invention further provides a device for identifying on-screen video content, comprising: A difference calculation module is used to calculate the difference between two adjacent frame images in the screen display video; a frame image extraction module, configured to extract key frames from the screen-displayed video based on the difference values; The screen display video content recognition module is used to identify event categories based on the key frames and generate a content report of the screen display video.

[0013] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-described methods for identifying screen video content are implemented.

[0014] In a fourth aspect, the present invention also provides a computer-readable storage medium, which includes a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the screen display video content recognition method as described in any one of the above are implemented.

[0015] The screen video content recognition method, device, electronic device and storage medium provided by the present invention effectively extract key frames by calculating the difference values ​​of adjacent frame images in the screen video, and can efficiently locate high-value information from a large amount of video data, and then use the extracted key frames to identify event categories, and finally generate a content report of the screen video, thereby realizing intelligent analysis and rapid understanding of the screen video content, improving the automation level and processing efficiency of screen video content recognition, and helping to greatly improve the speed and depth of video information mining in the subsequent analysis of the screen video content. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 It is a flow chart of the screen display video content recognition method provided by the present invention.

[0018] Figure 2 It is a flowchart of the model training process provided by the present invention.

[0019] Figure 3 It is a structural diagram of the screen display video content recognition device provided by the present invention.

[0020] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0022] The terms "first," "second," and the like in the present invention are used to distinguish similar objects and are not used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present invention can be implemented in orders other than those illustrated or described herein.

[0023] The following combination Figures 1-4 The present invention describes a method, device, electronic device and storage medium for identifying screen video content.

[0024] It should be noted that the on-screen video content recognition method provided in the embodiments of the present invention is implemented based on an on-screen video content recognition device. This method can identify the approximate content of a video before analysis by an on-screen video analyst. It segments the video and generates a content report. Based on the content report, the analyst can quickly locate the corresponding video segment for further analysis, significantly improving the speed and depth of video information mining.

[0025] In the embodiment of the present invention, a method for identifying content in a screen-displayed video is described by taking a device for identifying content in a screen-displayed video as an example.

[0026] Combine Figure 1 , Figure 1 It is a flow chart of the screen display video content recognition method provided by the present invention.

[0027] like Figure 1 As shown, the method includes the following: Step 101: Calculate the difference between two adjacent frame images in the screen display video; Step 102: extracting key frames from the screen-displayed video based on the difference values; Step 103: Perform event category recognition based on the key frames and generate a content report of the screen display video.

[0028] It should be noted that the embodiments of the present invention are described around aircraft screen display videos. Of course, the embodiments of the present invention can also be applied to screen display videos in other fields. Here, the screen display videos in the aviation field are refined and can be applied to screen display videos in other fields.

[0029] In scenarios such as pilot training, flight accident investigation, flight safety monitoring, and flight equipment maintenance, real-time images of these scenes will be recorded through acquisition equipment. The acquired video content will be displayed on the display screen for analysts to analyze the screen video based on the important data and visual information of the flight process provided by the screen video. For example, the analysis of operating specifications, system performance, potential problems, and emergency handling during the flight process will be carried out, thereby improving flight safety, pilot training quality, equipment maintenance level and other safety management.

[0030] Specifically, screen video is obtained. Since screen video is long, frame-by-frame image analysis is performed on the entire screen video. This process is not only time-consuming but also consumes a lot of computing resources. In addition, the screen video includes at least one event category. This method is to know the approximate content of the video in advance before the analyst performs the screen video analysis process. Therefore, it is only necessary to extract key frames from the video clips of the same event category and analyze and identify the key frames. The analyst can then quickly locate the video clip of a certain event category based on the identification results for further analysis, effectively improving the analysis efficiency.

[0031] It can be understood that the difference value between two adjacent frame images in the screen-displayed video is first calculated to obtain the difference value between the two adjacent frame images in the screen-displayed video.

[0032] Furthermore, if the difference value between two adjacent frame images is too large, the two adjacent frame images are used as segmentation points to segment the screen video to obtain at least one segment; if the difference value between two adjacent frame images is not large, it means that the two adjacent frame images belong to the same event category and need to be put into the same segment for analysis.

[0033] Furthermore, to extract a key frame from each segment, it is only necessary to perform event category recognition on the key frame, and the event category corresponding to the key frame can be considered as the event category corresponding to the segment where the key frame is located.

[0034] Furthermore, based on the key frames of each segment, a content report of each segment is generated, and the content reports of each segment are further integrated into a content report of the screen video in timeline order. Therefore, analysts can quickly locate the corresponding segment for further analysis through the content report of the screen video.

[0035] Furthermore, analysts can quickly locate the clip corresponding to a certain event category based on the content report of the on-screen video for further confirmation and verification. This can include analyzing and comparing key frames, objects, behaviors, etc. to ensure the accuracy and reliability of the event, which can greatly improve the speed and depth of video information mining.

[0036] The screen video content recognition method provided by the present invention effectively extracts key frames by calculating the difference values ​​of adjacent frame images in the screen video, can efficiently locate high-value information from a large amount of video data, and then use the extracted key frames to identify event categories, and finally generate a content report of the screen video, thereby realizing intelligent analysis and rapid understanding of the screen video content, improving the automation level and processing efficiency of screen video content recognition, and helping to greatly improve the speed and depth of video information mining in the subsequent analysis of the screen video content.

[0037] In some embodiments, based on step 101, calculating the difference between two adjacent frame images in the screen-displayed video specifically includes: Calculating the difference between the pixel values ​​of the two adjacent frame images according to the same coordinate position to obtain the pixel difference value corresponding to each coordinate position; The pixel difference values ​​corresponding to each coordinate position are summed to obtain the difference value between the two adjacent frame images.

[0038] It should be noted that the pixel values ​​of each frame of an image can usually be represented by a matrix. Therefore, each pixel value has a corresponding coordinate position in the matrix. When performing a pixel-by-pixel difference analysis between two frames of images, the difference value needs to be calculated based on the same coordinate position.

[0039] Specifically, according to the same coordinate position, the pixel values ​​of two adjacent frame images are difference-calculated to obtain the pixel difference value corresponding to each coordinate position. Further, the pixel difference values ​​of each coordinate position are summed to obtain the difference value between the two adjacent frame images. The algorithm adopted for calculating the difference value includes but is not limited to absolute difference and square difference.

[0040] In one embodiment, the difference value between the frame image A and the frame image B is calculated. For grayscale images, the pixel values ​​can be directly compared. For color images, the pixel values ​​of the red, green, and blue channels need to be compared respectively.

[0041] For the frame images A and B of the grayscale image, for each pixel value position (i, j), the difference between the pixel value A(i, j) and the pixel value B(i, j) is calculated, and the absolute difference can be used.

[0042] The formula is: pixel difference value (i, j) = |A(i, j) - B(i, j)|.

[0043] For frame images A and frame images B of a color image, for each pixel value position (i, j), first calculate the difference in each color channel between the pixel value A(i, j) and the pixel value B(i, j), you can use the absolute difference, and then integrate the difference value into the difference between the pixel value A(i, j) and the pixel value B(i, j).

[0044] The formula is: pixel difference value (i,j) = |A_R(i,j)-B_R(i,j)| + |A_G(i,j)-B_G(i,j)| + |A_B(i,j)-B_B(i,j)|.

[0045] Therefore, the difference value between the frame image A and the frame image B is the sum of the pixel difference values ​​(i, j) corresponding to each coordinate position.

[0046] In addition, the squared difference can also be used to calculate the pixel value difference.

[0047] As in the above example, for the grayscale image frame A and frame B, the formula is: pixel difference value (i, j) = |A(i, j)-B(i, j)| 2 .

[0048] As in the above example, for the color image frame A and frame B, the formula is: pixel difference value (i, j) = |A_R(i, j)-B_R(i, j)| 2 +|A_G(i,j)-B_G(i,j)| 2 +|A_B(i,j)-B_B(i,j)| 2 .

[0049] In another embodiment, a structural similarity index (SSIM) may be used. The SSIM function in the scikit-image library may be used to calculate the SSIM. The SSIM is an algorithm for measuring the similarity between two images. The SSIM considers differences in brightness, contrast, and structure, and generates a score between -1 and 1.

[0050] Among them, x and y represent the pixel values ​​of the two frame images respectively; and represent the means of x and y respectively; and represent the standard deviation of x and y respectively; Represents the covariance of x and y; C1 and C2 are constants used to stabilize calculations and are usually small positive numbers.

[0051] By calculating the SSIM for each pixel value and averaging the SSIMs across multiple pixels, we can obtain the SSIM value for the two frames, which is the difference between them. The SSIM value ranges from -1 to 1, with values ​​closer to 1 indicating greater similarity between the two frames and values ​​closer to -1 indicating less similarity between the two frames.

[0052] You can also use the Mean Squared Error (MSE) and use the OpenCV library to calculate MSE. MSE is a method to measure the difference between two frame images. It calculates a value by calculating the sum of the squares of the differences between each pixel. A lower MSE value indicates that the two images are more similar.

[0053] Where n represents the number of samples; Represents the predicted value of the i-th sample; Represents the true value of the i-th sample.

[0054] MSE measures the model's fit by calculating the mean of the sum of squares of the differences between the predicted values ​​and the true values. A smaller MSE value indicates a better model fit and a smaller difference between the predicted values ​​and the true values.

[0055] The embodiment of the present invention utilizes different difference algorithms to calculate the difference value of the pixel values ​​of two adjacent frame images, and finally obtains the difference value between the two adjacent frame images, thereby improving the flexibility of the difference analysis between images.

[0056] In some embodiments, based on step 102, extracting key frames from the screen-displayed video based on the difference values ​​specifically includes: Determine two frame images used for calculating a target difference value as a segmentation point; the target difference value is a difference value greater than a preset difference threshold among the difference values; dividing the screen-displayed video into at least one segment according to the segmentation point; At least one image frame is extracted from each of the segments to generate a key frame for each of the segments.

[0057] Specifically, each difference value is compared with a preset difference threshold value. According to the comparison result, a difference value greater than the preset difference threshold value is selected from the difference values ​​and used as the target difference value, wherein the preset difference threshold value is set according to actual conditions.

[0058] Furthermore, the two frame images used to calculate the target difference value are determined as segmentation points, and the screen video is segmented into at least one segment based on the segmentation points. For example, if the screen video has one segmentation point, the screen video after segmentation is divided into two segments; if the screen video has two segmentation points, the screen video after segmentation is divided into three segments. Similarly, if the screen video has n segmentation points, the screen video after segmentation is divided into n+1 segments. In the special case, if the segmentation point is 0, it means that the screen video content is similar overall and it is treated as a single segment.

[0059] Furthermore, at least one image frame is extracted from each segment to generate a key frame for each segment. For example, if the on-screen video has one segment and one key frame is extracted from each segment, the on-screen video has one key frame in total. If the on-screen video has two segments and one key frame is extracted from each segment, the on-screen video has two key frames in total. If the on-screen video has two segments and two key frames are extracted from each segment, the on-screen video has four key frames in total. If the on-screen video has n segments and m key frames are extracted from each segment, the on-screen video has n×m key frames in total.

[0060] An embodiment of the present invention uses two adjacent frame images with large differences as segmentation points, divides the screen video into video segments with different contents according to these segmentation points, extracts key frames from each video segment as key image information of its corresponding video segment, and uses the extracted key frames to identify event categories, thereby realizing intelligent analysis and rapid understanding of the screen video content, and improving the automation level and processing efficiency of screen video content recognition.

[0061] In some embodiments, based on step 103, event category identification is performed based on the key frames to generate a content report of the screen display video, specifically including: Inputting each of the key frames into an event recognition model in timeline order to obtain an event category recognition result corresponding to each of the key frames output by the event recognition model; the event category includes one or a combination of any two or more of aircraft takeoff, aircraft climb, aircraft landing, radar capture, and instrument failure; Based on the event category identification results, a content report of the screen display video is generated.

[0062] Specifically, the key frames of each segment are input into the event recognition model in timeline order, and the event category recognition results corresponding to each key frame output by the event recognition model are obtained.

[0063] Among them, event categories include aircraft takeoff, aircraft climb, aircraft landing, radar capture, instrument failure, or any combination of two or more. These event contents can provide important data and insights for aviation safety, flight operation optimization, and training. Subsequent detailed analysis of these video contents can help better understand the key links in the flight process, identify potential risks or improvement areas, and improve the safety and efficiency of flight operations.

[0064] Furthermore, generating a content report for each key frame based on the event category recognition result corresponding to each key frame can also be understood as generating a content report for each segment based on the event category recognition result corresponding to each segment.

[0065] Furthermore, the content reports of each key frame or each segment are integrated into a content report of the screen display video in a timeline order.

[0066] The embodiment of the present invention utilizes the extracted key frames, performs analysis and prediction through an event recognition model, outputs the corresponding event category, realizes intelligent image event recognition, improves the efficiency of image event recognition, and also ensures the recognition accuracy, and finally generates a content report of the screen video, realizing intelligent analysis and rapid understanding of the screen video content.

[0067] Furthermore, according to the above content, based on the identification results of each event category, generating a content report of the screen display video specifically includes: If the event category recognition result corresponding to any key frame is that the event category is not recognized, then the key frame is removed; If the event category identification result corresponding to any key frame is the event category corresponding to the key frame, a content report of the screen display video is generated according to the timestamps and corresponding event categories of each key frame.

[0068] Specifically, if the event category recognition result corresponding to a key frame is that the event category is not recognized, these unrecognized key frames are removed.

[0069] If there is an event category corresponding to a key frame and the identification result is the event category corresponding to the key frame, a content report for each key frame is generated based on the timestamp and corresponding event category of each key frame, and the content report of each key frame is further integrated into a content report of the screen display video in timeline order.

[0070] For example, after segmentation and frame extraction, a total of 3 key frames are obtained, which are input into the event recognition model in chronological order. If key frame 1 does not hit the event category, the frame is removed and key frame 2 is input. If key frame 2 successfully hits a certain event category, the timestamp of key frame 2 and the event category it hits are recorded, and then key frame 3 is input, and so on.

[0071] The embodiment of the present invention uses the extracted key frames to identify event categories and ultimately generates a content report for the screen video, thereby realizing intelligent analysis and rapid understanding of the screen video content, improving the automation level and processing efficiency of screen video content recognition, and helping to greatly improve the speed and depth of video information mining in the subsequent analysis of the screen video content.

[0072] Furthermore, according to the above content, generating a content report of the screen display video according to the timestamp of each key frame and the corresponding event category specifically includes: If two adjacent key frames in each of the key frames belong to the same event category, determining the time period between the timestamps of the two adjacent key frames as the event occurrence time period, and updating the content reports of the two adjacent key frames according to the event occurrence time period and the event categories corresponding to the two adjacent key frames; The content reports of the key frames are integrated into the content report of the screen display video according to the timeline sequence.

[0073] Specifically, a content report of each key frame is generated according to the timestamp of each key frame and the corresponding event category.

[0074] Furthermore, if two adjacent key frames in each key frame belong to the same event category, the time period between the timestamps of the two adjacent key frames is determined as the event occurrence time period. According to the time occurrence time period and the event category corresponding to the two adjacent key frames, the content reports of the two adjacent key frames are updated, that is, the content reports of the two key frames are merged together.

[0075] Furthermore, the content reports of the key frames are integrated into a content report of the screen display video according to the timeline sequence.

[0076] For example, all recognition results are sorted and listed according to the timeline. For key frames that hit the same event category at adjacent times, their times will be merged into the same event occurrence time period. For example, the event category hit at 01:30:12:41 (hour: minute: second: millisecond) is launch, and the next time 01:30:15:05 also hits the event category of launch. Then the two event results will be merged and the final report will be reflected as "01:30:12:41 to 01:30:15:05 launch". And so on. Finally, the content report will be output according to the timeline, and the on-screen video content analysis is completed.

[0077] The embodiment of the present invention generates a content report for each key frame based on the timestamp of each key frame and the corresponding event category, and then merges adjacent content reports belonging to the same event category to finally generate a complete content report for the screen video, thereby realizing intelligent analysis and rapid understanding of the screen video content, improving the automation level and processing efficiency of screen video content recognition, and helping to greatly improve the speed and depth of video information mining in the subsequent analysis of the screen video content.

[0078] Furthermore, the event recognition model described above is trained in the following way: Get key frame image samples; Preprocessing the key frame image samples to obtain target frame image samples; Performing event category labeling on the target frame image sample to obtain label data corresponding to the target frame image sample; Model training is performed based on the target frame image samples and their label data to obtain the event recognition model.

[0079] The following content is combined Figure 2 ,picture Figure 2 It is a flowchart of the model training process provided by the present invention.

[0080] Specifically, a large amount of image data is collected for training the model. The source of these image data is the key frames described in the above embodiment. In order not to limit the number, they are described here as key frame image samples. The types of key frame image samples should include representative images of various events, including: aircraft takeoff, aircraft climb, aircraft landing, radar capture, instrument failure, one or a combination of any two or more, so that the model can learn different objects and features.

[0081] Furthermore, the collected key frame image samples are preprocessed to obtain target frame image samples, wherein the preprocessing method includes one or a combination of any two or more operations of image scaling, cropping, rotation and grayscale, with the purpose of converting the image data into a format that can be processed by the model.

[0082] Furthermore, event categories are annotated on the target frame image samples to obtain label data corresponding to the target frame image samples, wherein the label data at least includes: aircraft takeoff, aircraft climb, aircraft landing, radar capture, and instrument failure.

[0083] Furthermore, features are extracted from the target frame image samples using techniques such as convolutional neural networks. Convolutional neural networks are a widely used deep learning model that can automatically learn features in images, such as edges, textures, and shapes.

[0084] The extracted features are then fed into a classifier for training. Through training, the model learns the characteristics of different categories and associates them with the corresponding labeled data. The classifier can be a support vector machine, random forest, or deep learning model.

[0085] Furthermore, the performance of the trained model is evaluated using a test set. Evaluation metrics may include accuracy, recall, etc. If the model performs poorly, the model architecture can be adjusted, training data can be added, or hyperparameters can be adjusted until the expected effect is achieved. The test set used here is sourced from video key frames, but the images themselves do not intersect with the key frame image samples used for training.

[0086] Furthermore, the trained model is deployed in actual applications to perform image recognition tasks by taking key frames as input and associating the key frames with the above-mentioned label data as event categories. The key frames here are images generated by real business.

[0087] The event recognition model of the embodiment of the present invention can accurately identify the characteristics of different events by preprocessing and labeling key frame image samples and combining them with event category labels for training. By learning from a large amount of labeled data, the model can effectively extract key information related to the event from the image, thereby achieving efficient event category recognition, significantly improving the accuracy and automation of image analysis, and is suitable for a variety of event detection and monitoring scenarios.

[0088] The screen display video content recognition device provided by the present invention is described below. The screen display video content recognition device described below and the screen display video content recognition method described above can be referenced to each other.

[0089] Reference Figure 3 , Figure 3 It is a structural diagram of the screen display video content recognition device provided by the present invention.

[0090] The screen display video content recognition device includes: The difference calculation module 310 is used to calculate the difference value between two adjacent frame images in the screen display video.

[0091] The frame image extraction module 320 is configured to extract key frames from the screen display video based on the difference values.

[0092] The screen display video content recognition module 330 is configured to perform event category recognition based on the key frames and generate a content report of the screen display video.

[0093] The screen video content recognition device provided by the present invention effectively extracts key frames by calculating the difference values ​​of adjacent frame images in the screen video, can efficiently locate high-value information from a large amount of video data, and then use the extracted key frames to identify event categories, and finally generate a content report of the screen video, thereby realizing intelligent analysis and rapid understanding of the screen video content, improving the automation level and processing efficiency of screen video content recognition, and helping to greatly improve the speed and depth of video information mining in the subsequent analysis of the screen video content.

[0094] Furthermore, the difference calculation module 310 is further configured to: Calculating the difference between the pixel values ​​of the two adjacent frame images according to the same coordinate position to obtain the pixel difference value corresponding to each coordinate position; The pixel difference values ​​corresponding to each coordinate position are summed to obtain the difference value between the two adjacent frame images.

[0095] Furthermore, the frame image extraction module 320 is further configured to: Determine two frame images used for calculating a target difference value as a segmentation point; the target difference value is a difference value greater than a preset difference threshold among the difference values; dividing the screen-displayed video into at least one segment according to the segmentation point; At least one image frame is extracted from each of the segments to generate a key frame for each of the segments.

[0096] Furthermore, the screen display video content recognition module 330 is further configured to: Inputting each of the key frames into an event recognition model in timeline order to obtain an event category recognition result corresponding to each of the key frames output by the event recognition model; the event category includes one or a combination of any two or more of aircraft takeoff, aircraft climb, aircraft landing, radar capture, and instrument failure; Based on the event category identification results, a content report of the screen display video is generated.

[0097] Furthermore, the screen display video content recognition module 330 is further configured to: If the event category recognition result corresponding to any key frame is that the event category is not recognized, then the key frame is removed; If the event category identification result corresponding to any key frame is the event category corresponding to the key frame, a content report of the screen display video is generated according to the timestamps and corresponding event categories of each key frame.

[0098] Furthermore, the screen display video content recognition module 330 is further configured to: If two adjacent key frames in each of the key frames belong to the same event category, determining the time period between the timestamps of the two adjacent key frames as the event occurrence time period, and updating the content reports of the two adjacent key frames according to the event occurrence time period and the event categories corresponding to the two adjacent key frames; The content reports of the key frames are integrated into the content report of the screen display video according to the timeline sequence.

[0099] Furthermore, the screen display video content recognition device is also used for: Get key frame image samples; Preprocessing the key frame image samples to obtain target frame image samples; Performing event category labeling on the target frame image sample to obtain label data corresponding to the target frame image sample; Model training is performed based on the target frame image samples and their label data to obtain the event recognition model.

[0100] It should be noted that the on-screen video content recognition device provided by the present invention can execute the on-screen video content recognition method described in any of the above embodiments during specific operation, which will not be described in detail in this embodiment.

[0101] Figure 4 Schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4 As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other via the communications bus 440. The processor 410 may invoke logic instructions in the memory 430 to execute a method for identifying content in an on-screen video. The method includes: calculating a difference value between two adjacent frame images in an on-screen video; extracting key frames from the on-screen video based on each of the difference values; and identifying event categories based on the key frames to generate a content report for the on-screen video.

[0102] Furthermore, the logic instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0103] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the screen video content recognition method provided by the above-mentioned embodiments, the method including: calculating the difference value between two adjacent frame images in the screen video; extracting key frames from the screen video based on each of the difference values; performing event category recognition based on the key frames, and generating a content report of the screen video.

[0104] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the screen video content recognition method provided by the above-mentioned embodiments, the method including: calculating the difference value between two adjacent frame images in the screen video; extracting key frames from the screen video based on each of the difference values; performing event category recognition based on the key frames, and generating a content report of the screen video.

[0105] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement the present invention without inventive effort.

[0106] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for identifying screen display video content, characterized in that: include: Calculate the difference between two adjacent frame images in the screen display video; Extracting key frames from the screen-displayed video based on the difference values; Event categories are identified based on the key frames, and a content report of the screen display video is generated.

2. The method for identifying screen display video content according to claim 1, characterized in that: The calculating the difference value between two adjacent frame images in the screen display video includes: According to the same coordinate position, the pixel values ​​of the two adjacent frame images are calculated to obtain the pixel difference value corresponding to each coordinate position; The pixel difference values ​​corresponding to each coordinate position are summed to obtain the difference value between the two adjacent frame images.

3. The method for identifying screen display video content according to claim 2, characterized in that: The step of extracting a key frame from the screen-displayed video based on each of the difference values ​​includes: Determine two frame images used to calculate a target difference value as a segmentation point; the target difference value is a difference value greater than a preset difference threshold among the difference values; According to the segmentation point, the screen display video is segmented into at least one segment; At least one image frame is extracted from each of the segments to generate a key frame for each of the segments.

4. The method for identifying screen display video content according to claim 3, characterized in that: The performing event category recognition based on the key frame and generating a content report of the screen-displayed video includes: Input each of the key frames into the event recognition model in timeline order to obtain the event category recognition results corresponding to each of the key frames output by the event recognition model; the event category includes one of aircraft takeoff, aircraft climb, aircraft landing, radar capture, instrument failure, or a combination of any two or more thereof; Based on the event category identification results, a content report of the screen display video is generated.

5. The method for identifying screen display video content according to claim 4, characterized in that: The generating a content report of the screen-displayed video based on the event category identification results includes: If the event category recognition result corresponding to any key frame is that the event category is not recognized, then each key frame is removed; If the event category recognition result corresponding to any key frame is the event category corresponding to the key frame, a content report of the screen display video is generated according to the timestamps of each key frame and the corresponding event category.

6. The method for identifying screen display video content according to claim 5, characterized in that: Generating a content report of the screen-displayed video according to the timestamp of each key frame and the corresponding event category includes: If two adjacent key frames in each of the key frames belong to the same event category, the time period between the timestamps of the two adjacent key frames is determined as the event occurrence time period, and the content reports of the two adjacent key frames are updated according to the event occurrence time period and the event categories corresponding to the two adjacent key frames; The content reports of the key frames are integrated into the content report of the screen display video according to the timeline sequence.

7. The method for identifying screen display video content according to claim 4, characterized in that: The event recognition model is trained in the following way: Get key frame image samples; Preprocessing the key frame image samples to obtain target frame image samples; Annotating the target frame image sample with an event category to obtain label data corresponding to the target frame image sample; Model training is performed based on the target frame image samples and their label data to obtain the event recognition model.

8. A device for identifying screen display video content, characterized in that: include: A difference calculation module is used to calculate the difference value between two adjacent frame images in the screen display video; A frame image extraction module, used for extracting key frames from the screen display video based on each of the difference values; The screen display video content recognition module is used to perform event category recognition based on the key frames and generate a content report of the screen display video.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the screen display video content recognition method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, the computer-readable storage medium comprising a non-transitory computer-readable storage medium, having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the screen display video content recognition method according to any one of claims 1 to 7 are implemented.

Images