A method and system for understanding streaming video and extracting key information structuredly

Abstract

The present application discloses a method and system for understanding streaming videos and extracting key information in a structured manner, the method comprising: collecting video data and determining the application type, decomposing the video data into a video frame sequence according to the application type and the preset duration of each frame sequence; using a multi-feature comprehensive decision algorithm to identify key frames in the video frame sequence; using a preset voice model to identify voice information in the video frame sequence and convert it into text, and using natural language processing technology to obtain key text information; mapping and associating key frames with key text information according to a preset mapping method; fusing the feature vectors of the key frames with the feature vectors of the associated key text information, and performing joint reasoning in combination with a deep learning model to obtain the joint reasoning results of the video; extracting key information from the video and storing it in a structured form according to the joint reasoning results of the video. The present application can improve the efficiency and accuracy of users' understanding of video content.

Description

Technical Field

[0001] The present application relates to the technical field of video data processing, and in particular to a method and system for understanding streaming videos and extracting structured key information. Background Art

[0002] With the rapid development of the Internet, video data has become an important carrier of information dissemination. However, faced with massive video content, how to efficiently understand video content, extract key information and present it in a structured manner has become an urgent problem to be solved.

[0003] Traditional video analysis methods are mostly based on image processing and video coding technologies. The intelligent algorithms involved are all methods for extracting, analyzing and generating single features, which makes it difficult to deeply understand the semantic information in the video. In recent years, with the maturity and development of multimodal information fusion technology, technologies have emerged that achieve in-depth understanding of videos through the fusion of video, audio and text information. However, the fusion of existing multimodal fusion technologies does not take into account the differences in key information in videos under different application scenarios, and lacks accuracy in key information extraction and fusion. It is difficult to truly achieve a deep understanding of streaming video content, and automatically extract and structure the key information in the video to improve the efficiency and accuracy of video content understanding and enhance user experience. Summary of the invention

[0004] In order to achieve a deep understanding of streaming video content, automatically extract and structure the key information in the video, and improve the efficiency and accuracy of the user's understanding of the video content, the present application provides a method and system for streaming video understanding and structured extraction of key information.

[0005] In a first aspect, the present application provides a method for understanding streaming video and extracting structured key information, specifically comprising: collecting video data and determining its application type; obtaining a preset per-frame sequence duration of the determined application type from a first database according to the determined application type, and decomposing the video data into a video frame sequence according to the obtained preset per-frame sequence duration of the determined application type;

[0006] According to the determined application type, a preset feature weight of the determined application type is obtained from the second database, and a multi-feature comprehensive decision algorithm with the preset feature weight of the determined application type is used to identify key frames in the video frame sequence; the multi-feature decision method includes: a motion detection decision method, a color histogram comparison decision method and an image entropy analysis decision method;

[0007] According to the determined application type, a preset speech recognition model of the determined application type is obtained from a third database, the preset speech recognition model of the determined application type is used to recognize speech information in the video frame sequence and convert the recognized speech into text, and key text information is obtained by using natural language processing technology;

[0008] According to the determined application type, a preset mapping association method for the determined application type is obtained from the fourth database, and the key frames identified in the video frame sequence are mapped and associated with the key text information according to the obtained preset mapping method for the determined application type; the feature vectors of the key frames and the feature vectors of the associated key text information are subjected to feature fusion through a cross-modal learning network model, and joint reasoning is performed in combination with a deep learning model to obtain a joint reasoning result of the video; the preset mapping association method includes: mapping association based on timestamp, mapping association based on scene description, and mapping association based on event description;

[0009] Based on the joint reasoning results of the video, the key information in the video is extracted and stored in a structured form.

[0010] By adopting the above scheme, taking into account the differences in key information in videos of different application types, the duration of the corresponding type of preset frame sequence is obtained to complete the video frame processing, the feature weight is preset to complete the key frame extraction, the speech recognition module is preset to complete the key text information acquisition, and the mapping association method is preset to complete the video text fusion, finally realizing the joint reasoning of multimodal fused video content, and then extracting accurate key information.

[0011] Preferably, determining the application type of the video data specifically includes:

[0012] Determine whether the collected video data has a video application type label. If so, determine the corresponding application type based on the application type label; otherwise, input the video data into a constructed video application type recognition model, and output the application type of the video data; the video application type recognition model is generated by training a number of historical video data marked with application types.

[0013] By adopting the above scheme, considering that some video data will record the type of current video data record in the form of labels, the application type can be determined quickly; for video data without labels, neural networks are used to identify the application type to achieve accurate determination of the application type.

[0014] Preferably, it also includes:

[0015] After collecting video data and determining its application type, continue to determine the sub-application type based on the application type classification based on the video data; obtain the adjustment weight of the preset feature weight for determining the application type from the second database according to the determined sub-application type; and use a multi-feature comprehensive decision algorithm with the preset feature weights adjusted after the weights for determining the application type to identify key frames in the video frame sequence.

[0016] By adopting the above solution, considering different application types and dynamically adjusting the preset feature weights under different time or content orientations, the key frames in the video frame sequence can be more accurately identified.

[0017] Preferably, it also includes:

[0018] According to the determined application type, a preset speech recognition model of the determined application type is obtained from the third database, and at the same time, a preset duration of using the preset speech model of the determined application type to recognize the speech information in a frame of video in the video frame sequence is obtained; the actual duration of using the preset speech model of the determined application type to recognize the speech information in a frame of video in the video frame sequence is counted, and if the counted duration is less than the preset duration and the number of video frames is greater than the preset number of frames, the spare preset speech recognition model of the corresponding type is switched to re-recognize the speech information in the video frame sequence.

[0019] By adopting the above solution, taking into account the possibility of failures in the speech recognition process, when the language recognition time is detected to have timed out, the backup speech recognition model is switched accordingly to complete the speech recognition in a timely manner.

[0020] Preferably, it also includes: obtaining a satisfaction score corresponding to the key information of the joint reasoning result of the video from user feedback; if the satisfaction score is lower than a preset score threshold, optimizing the model parameters of the deep learning model or the cross-modal fusion network model until the preset score threshold is reached.

[0021] By adopting the above solution, we can judge based on user feedback whether the current joint reasoning results using fusion features to obtain key information meet the user's requirements for in-depth understanding of the video, and then continuously optimize the model to achieve in-depth understanding of the video content.

[0022] Preferably, it also includes: designing a visualization display module to visualize the extracted key information in the form of charts, timelines, and tag clouds.

[0023] By adopting the above solution, key information is visualized in the form of timelines, charts, tag clouds, etc., helping users to quickly understand the video content.

[0024] Preferably, the application types include: security application types and media application types divided according to application scenarios, content attributes and usage purposes.

[0025] By adopting the above scheme, the application type is specifically disclosed according to the classification standard of the application type to clarify the in-depth understanding of the video content in the application field.

[0026] In a second aspect, the present application provides a system for understanding streaming videos and extracting structured key information, specifically comprising: a video preprocessing module for collecting video data and determining its application type; obtaining a preset per-frame sequence duration of the determined application type from a first database according to the determined application type, and decomposing the video data into a video frame sequence according to the obtained preset per-frame sequence duration of the determined application type;

[0027] A key frame extraction module is used to obtain a preset feature weight of a determined application type from a second database according to the determined application type, and identify key frames in a video frame sequence using a multi-feature comprehensive decision algorithm with the preset feature weight of the determined application type; the multi-feature decision method includes: a motion detection decision method, a color histogram comparison decision method, and an image entropy analysis decision method;

[0028] A natural language processing module, configured to obtain a preset speech recognition model of the determined application type from a third database according to the determined application type, recognize speech information in the video frame sequence using the preset speech model of the determined application type and convert the recognized speech into text, and obtain key text information using natural language processing technology;

[0029] A video text fusion reasoning module is used to obtain a preset mapping association method for a determined application type from a fourth database according to the determined application type, and map and associate key frames and key text information identified by a video frame sequence according to the obtained preset mapping method for the determined application type; feature vectors of the key frames and feature vectors of the associated key text information are feature-fused through a cross-modal learning network model, and joint reasoning is performed in combination with a deep learning model to obtain a joint reasoning result of the video; the preset mapping association method includes: mapping association based on timestamp, mapping association based on scene description, and mapping association based on event description;

[0030] The key information structured extraction module is used to extract key information from the video based on the joint reasoning results of the video and store it in a structured form.

[0031] By adopting the above solution, considering the application type of video data, applying natural language processing technology and key frame analysis technology in a targeted manner, integrating multimodal data, realizing semantic understanding and key information extraction of streaming video, and improving the efficiency and accuracy of users' understanding of video content.

[0032] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method as described above.

[0033] In a fourth aspect, the present application provides a computer device, comprising a memory, a processor, and a program stored and executable on the memory, wherein the program implements the steps of the above method when executed by the processor.

[0034] In summary, this application has the following beneficial effects:

[0035] Taking into account the differences in key information in videos of different application types, the corresponding preset method is searched based on the application type to complete video decomposition, key frame extraction, key text information extraction, and key frame and key text feature fusion. Finally, the features of multimodal fusion are used to complete the joint reasoning of video content. Based on the joint reasoning, the key information of the video is accurately extracted in a structured form, which improves the efficiency and accuracy of users' understanding of video content. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flowchart of the method for understanding streaming video and extracting key information structuredly in a specific embodiment;

[0037] Figure 2 It is a schematic diagram of the system structure of streaming video understanding and key information structured extraction described in a specific embodiment. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0039] like Figure 1 As shown, the embodiment of the present application discloses a method for understanding streaming video and extracting key information structuredly, which specifically includes the following steps:

[0040] S1. Collect video data and determine its application type.

[0041] Specifically, during the process of collecting video stream data, part of the video data will record the application type label of the video, thereby determining the application type of the video data; specifically including: judging whether the collected video data has an application type label, if so, determining the corresponding application type according to the application type label; otherwise, inputting the video data into a constructed video application type recognition model, and outputting the application type of the video data; the video application type recognition model adopts a CNN neural network, which is specifically generated by training a number of historical video data marked with application types.

[0042] In this embodiment, the application types are specifically divided into security application types and media application types according to application scenarios, content attributes and usage purposes; among them, security videos are mainly used in the field of security monitoring, with relatively simple content, mainly shooting real-time images of the monitoring area for the purpose of ensuring safety; media videos are mainly used in the field of media communication, with relatively rich and diverse content, including news reports, interviews, dialogues, plot interpretations and other forms, with the main purpose of delivering news and promoting culture.

[0043] S2. Query the preset duration of each frame sequence according to the determined application type and complete video decomposition.

[0044] Specifically, taking into account the differences in video data content corresponding to different application types, for example, the security video content is relatively simple, and when the video is segmented into frames, it can be set to be decomposed according to a longer frame of video, while the media video content is relatively rich, and when the video is segmented into frames, it is necessary to increase the decomposition frequency and set it to be decomposed according to a shorter frame of video. Therefore, the preset duration of each frame sequence is pre-set for different application types.

[0045] Obtain the preset per-frame sequence duration for a determined application type from the first database; the first database stores the preset per-frame sequence durations for video data of different application types, such as: the first preset per-frame sequence duration (3s) for the security type, and the second preset per-frame sequence duration (0.5s) for the media type.

[0046] The streaming media parsing technology is used to decompose the collected video according to the preset frame sequence length of the determined application type, and the corresponding video frame sequence {I1, I2, ..., I n}, where I n Indicates the nth frame of the video. The duration of each frame is the preset sequence duration of each frame.

[0047] In addition to decomposing the collected video, the decomposed frame sequence is subjected to image denoising and enhancement processing, that is, the video denoising is performed using filters and image enhancement technology, and the frame sequence is enhanced with an image enhancement algorithm. Specifically, the bilateral filter algorithm is selected for filtering processing, and the calculation formula is:

[0048]

[0049] Where p and q are pixels in the image. is the spatial distance weight function, is the pixel value difference weight function.

[0050] S3. Query preset feature weights according to the determined application type and use a multi-feature comprehensive decision algorithm to identify key frames in the video frame sequence.

[0051] Specifically, taking into account the differences in video data content corresponding to different application types, such as most security video content is static and repetitive images, when there is motion change in the frame, it is likely to be a key frame, so the corresponding motion detection weight is relatively heavy, while the media video content is richer, the weights of various feature detections can be set more evenly. Therefore, preset feature weights are pre-set for video data of different application types.

[0052] According to the determined application type, the preset feature weights for determining the application type are obtained from the second database, which respectively correspond to the weight ratios of extracting features using each decision method to determine whether it is a key frame in the multi-feature comprehensive decision algorithm; the decision methods included in the multi-feature comprehensive decision algorithm include: weights of motion detection decision method, color histogram comparison decision method and image entropy analysis decision method; the second database stores the preset feature weights for determining the application type, such as: the preset feature weights for the security type are α1, β1, γ1, and the preset feature weights for the media type are α2, β2, γ2.

[0053] A multi-feature comprehensive decision algorithm with preset feature weights for a certain application type is used to identify key frames in a video frame sequence, and a key frame collection {K1, K2, …, K n}, where K n Indicates the nth important scene or change point of the video. Among them, the multi-feature comprehensive decision algorithm identifies the key frame in the video frame sequence, including extracting features of the video frame sequence by different decision methods, weighting the features according to the preset feature weights, and completing the key frame recognition.

[0054] S4. According to the determined application type, the preset speech recognition model is queried and speech recognition is completed and converted into text, and the key text information is obtained using natural language processing technology.

[0055] Specifically, taking into account the differences in languages ​​contained in video data of different application types, a speech recognition model generated by training video data of a certain application type is selected to achieve more accurate speech content recognition, such as: a first neural network trained for language recognition using historical security application type videos, and a second neural network trained for language recognition using historical media application type videos.

[0056] A preset speech recognition model for the determined application type is obtained from a third database according to the determined application type; the third database stores preset speech recognition modules for video data of different application types.

[0057] A preset speech model of a certain application type is used to recognize speech information in a video frame sequence and convert the recognized speech into text; wherein the recognition process can be expressed as a transformation of a sequence model:

[0058] T=ASR(A)

[0059] Where A is the audio input and T is the text sequence output.

[0060] Natural language processing technology is used to obtain key text information; in this embodiment, the NLP model is used to process the recognized text, including word segmentation, part-of-speech tagging, syntactic analysis, named entity recognition, sentiment analysis, etc., to extract key information. The specific text processing process can be formalized as follows:

[0061] Entities = NLP(T)

[0062] In the formula, T is the input text, and Entities is the extracted key text information, such as: name, place name, time, etc.

[0063] S5. According to the determined application type, a preset mapping association method is queried to complete the mapping association between key frames and key text information, feature fusion is completed according to the associated key frames and key text information, and deep understanding and reasoning of the video content are completed in combination with a deep learning network.

[0064] Specifically, in order to further ensure the fusion of relevant feature information, the identified key frames and key texts are associated in advance. However, considering the differences in key frames and key texts extracted by different application types, the corresponding preset mapping methods are set to complete the association mapping. The preset mapping association methods include: mapping association based on timestamp, mapping association based on scene description, and mapping association based on event description, etc.

[0065] Among them, the key frames and key texts of security application types mainly contain records of abnormal activities and events, and there are relatively few descriptions of scenes in the key texts. Therefore, the corresponding preset mapping method adopts a mapping association based on timestamps as the basis, supplemented by mapping associations based on scene descriptions and mapping associations based on event descriptions to complete the association between key frames and key texts; specifically: pre-match key frames and key texts with the same timestamps, and on this basis, match and adjust the key frame scenes and events according to the scenes and events described in the key texts; while the key frames and key texts of media application types have relatively rich main contents, which are more descriptions of scenes and events. Therefore, the corresponding preset mapping method adopts a mapping association based on scene and event descriptions as the main basis, supplemented by mapping associations based on timestamps to complete the association between key frames and key texts; specifically: pre-match the scenes and events described in the key texts with the key frame scenes and events, and adjust the key frames and key texts to have as close timestamps as possible on the basis of the match.

[0066] According to the determined application type, a preset mapping association method of the determined application type is obtained from a fourth database; the fourth database stores preset mapping association methods of video data of different application types. According to the obtained preset mapping method of the determined application type, the key frames identified in the video frame sequence are mapped and associated with the key text information.

[0067] The feature vectors of key frames and the feature vectors of associated key text information are fused through a cross-modal learning network model, and joint reasoning is performed with a deep learning model to obtain the joint reasoning results of the video.

[0068] Specifically, the feature vectors of key frames and the feature vectors of key text information are extracted, and the feature vectors of key frames and key text information are mapped to the same (semantic) space vector for fusion through a pre-built and trained cross-modal learning network. The fused feature vectors are input into a pre-built and trained deep learning model (such as BERT) to output the joint reasoning results for the video, such as early warning prompts for videos of security application type and news summaries for videos of media application type.

[0069] S6. Based on the joint reasoning results of the video, extract the key information in the video and store it in a structured form.

[0070] Specifically, according to the joint reasoning results of the video, natural language processing technology is used to extract key information from the reasoning results and store them in a structured format such as JSON or XML. For example, a news summary of a video of a media application type:

[0071] "Event": "xxx",

[0072] "Person": ["Person", "Li Si"],

[0073] "Location": "Somewhere",

[0074] "Time": "2024 x month x day"

[0075] }

[0076] S7. Use the designed visual interface to display the extracted key information.

[0077] Specifically, a visual display module is designed to visualize the extracted key information in the form of charts, timelines, and tag clouds, so that users can quickly and deeply understand the video content.

[0078] In a specific embodiment, considering that users have feedback on key information of video content, the efficiency and accuracy of outputting key information can be further improved based on user feedback. The method also includes:

[0079] Obtain user feedback on the satisfaction score of the joint reasoning results of the video corresponding to the key information, such as:

[0080] Score = f(K i ,T i )

[0081] In the formula, Score is the satisfaction score of the key information corresponding to the joint reasoning result of the video fed back by the user, f is the cross-modal fusion network, and K i is the feature vector of the key frame, T i is the feature vector of the key text associated with it;

[0082] If the satisfaction score is lower than the preset score threshold, the model parameters of the deep learning model or the cross-modal fusion network model are optimized until the preset score threshold is reached.

[0083] In a specific embodiment, in order to further accurately acquire key frames of video data of different application types, the method further includes:

[0084] After collecting the video data and determining its application type, the sub-application type divided based on the application type is further determined according to the video data.

[0085] Specifically, each application type is pre-set with a sub-application type, and the pre-set sub-application type is uploaded to the second database and exists in the form of association between the application type and the sub-application type. For example, the videos of the security application type are divided into videos of the daytime sub-application type and videos of the nighttime sub-application type according to time, and the videos of the media application type are divided into news report sub-application type, interview dialogue sub-application type, plot interpretation sub-application type and other sub-application types according to content orientation. Specifically, the method of determining the self-application type can still be determined based on the video tag or by using a neural network model that is trained to identify the sub-application type.

[0086] According to the determined sub-application type, the adjustment weight of the preset feature weight of the determined application type is obtained from the second database; the second database also stores sub-application types associated with different application types, such as: when the light video appears at night in the security video content, the color difference is large, thereby increasing the weight of the feature extracted by the color histogram comparison decision method at night, and correspondingly setting the weight of the sub-application type adjustment: α 1, , β1 , , γ 1, ; The interview dialogue sub-application type in the media video has relatively less obvious motion changes compared to other sub-application types, so the weight of the features extracted by motion detection decision-making for this sub-application type can be reduced, and the weight adjusted for this sub-application type is set accordingly: α 2, , β2 , , γ 2, .

[0087] A multi-feature comprehensive decision algorithm with preset feature weights adjusted with weights for a certain application type is used to identify key frames in a video frame sequence; wherein the adjusted preset feature weights are generated by merging the preset feature weights with the adjusted weights of the preset feature weights.

[0088] In a specific embodiment, considering that there may be situations where speech recognition cannot be recognized during the speech recognition process, in order to ensure timely recognition of key text information, the method further includes:

[0089] According to the determined application type, a preset speech recognition model of the determined application type is obtained from the third database, and at the same time, a preset time length for using the preset speech model of the determined application type to recognize speech information in a frame of video in a video frame sequence is obtained; that is, the third database also stores preset time lengths for different application types to recognize a frame of corresponding type of video data, and the preset time lengths are obtained by statistically analyzing the time lengths spent on historical recognition of a frame of corresponding type of video data by preset speech models of different application types.

[0090] Statistics actually use the preset speech model of a certain application type to recognize the duration of a frame of video voice information in a video frame sequence. If the counted duration is less than the preset duration and the number of video frames is greater than the preset number of frames, it indicates that there are problems in the recognition process of a large number of frames. The corresponding type of preset speech recognition model is switched to the standby one and the voice information in the video frame sequence is re-recognized.

[0091] Alternatively, statistics are provided for the actual duration of using a preset speech model of a certain application type to recognize the speech information of all frames in a video frame sequence. If the counted duration is less than n times the preset duration, where n is the number of all frames, then the corresponding type of preset speech recognition model is switched to a standby one and the speech information in the video frame sequence is recognized again.

[0092] like Figure 2 As shown, the present application also provides a system for streaming video understanding and key information structured extraction, specifically including:

[0093] The video preprocessing module 101 is used to collect video data and determine its application type; obtain a preset per-frame sequence duration of the determined application type from the first database according to the determined application type, and decompose the video data into a video frame sequence according to the obtained preset per-frame sequence duration of the determined application type;

[0094] The key frame extraction module 102 is used to obtain the preset feature weight of the determined application type from the second database according to the determined application type, and use a multi-feature comprehensive decision algorithm with the preset feature weight of the determined application type to identify the key frames in the video frame sequence; the multi-feature decision method includes: motion detection decision method, color histogram comparison decision method and image entropy analysis decision method;

[0095] A natural language processing module 103 is used to obtain a preset speech recognition model of the determined application type from a third database according to the determined application type, recognize speech information in the video frame sequence using the preset speech model of the determined application type and convert the recognized speech into text, and obtain key text information using natural language processing technology;

[0096] The video-text fusion reasoning module 104 is used to obtain a preset mapping association method for the determined application type from the fourth database according to the determined application type, and map and associate the key frames identified by the video frame sequence with the key text information according to the obtained preset mapping method for the determined application type; perform feature fusion on the feature vectors of the key frames and the feature vectors of the associated key text information through a cross-modal learning network model, and perform joint reasoning in combination with a deep learning model to obtain a joint reasoning result of the video; the preset mapping association method includes: mapping association based on timestamp, mapping association based on scene description, and mapping association based on event description;

[0097] The key information structured extraction module 105 is used to extract key information from the video according to the joint reasoning result of the video and store it in a structured form.

[0098] The system further comprises:

[0099] The key information display module 106 is used to design a visual display module to visually display the extracted key information in the form of charts, timelines, and tag clouds.

[0100] In a specific embodiment, the key frame extraction module 102 in the system is also used to collect video data and determine its application type, and then continue to determine the sub-application type based on the application type according to the video data; obtain the adjustment weight of the preset feature weight of the determined application type from the second database according to the determined sub-application type; and use a multi-feature comprehensive decision algorithm with the preset feature weights adjusted after the weights of the determined application type to identify key frames in the video frame sequence.

[0101] In a specific embodiment, the natural language processing module 103 in the system is also used to obtain a preset speech recognition model of a determined application type from a third database according to the determined application type, and at the same time obtain a preset duration of using the preset speech model of the determined application type to recognize speech information in a frame of video in a video frame sequence; count the actual duration of using the preset speech model of the determined application type to recognize speech information in a frame of video in a video frame sequence, and if the counted duration is less than the preset duration and the number of video frames is greater than the preset number of frames, switch to a spare preset speech recognition model of the corresponding type and re-perform recognition of the speech information in the video frame sequence.

[0102] The embodiment of the present application also discloses a computer-readable storage medium.

[0103] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and execute the method of streaming video understanding and structured extraction of key information as described above. The computer-readable storage medium includes, for example: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0104] The embodiment of the present application also discloses a computer device.

[0105] Specifically, the computer device includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and execute the above-mentioned method of streaming video understanding and key information structured extraction.

[0106] The above are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any feature disclosed in this specification (including the abstract and drawings), unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

Claims

1. A method for understanding streaming video and extracting key information structuredly, characterized in that: include: Collect video data and determine its application type; Acquire a preset per-frame sequence duration of the determined application type from a first database according to the determined application type, and decompose the video data into a video frame sequence according to the acquired preset per-frame sequence duration of the determined application type; According to the determined application type, a preset feature weight of the determined application type is obtained from the second database, and a multi-feature comprehensive decision algorithm with the preset feature weight of the determined application type is used to identify key frames in the video frame sequence; The multi-feature comprehensive decision algorithm includes: motion detection decision mode, color histogram comparison decision mode and image entropy analysis decision mode; According to the determined application type, a preset speech recognition model of the determined application type is obtained from a third database, the preset speech recognition model of the determined application type is used to recognize speech information in the video frame sequence and convert the recognized speech into text, and key text information is obtained by using natural language processing technology; According to the determined application type, a preset mapping association method for the determined application type is obtained from the fourth database, and the key frames identified in the video frame sequence are mapped and associated with the key text information according to the obtained preset mapping method for the determined application type; the feature vectors of the key frames and the feature vectors of the associated key text information are subjected to feature fusion through a cross-modal learning network model, and joint reasoning is performed in combination with a deep learning model to obtain a joint reasoning result of the video; the preset mapping association method includes: mapping association based on timestamp, mapping association based on scene description, and mapping association based on event description; Based on the joint reasoning results of the video, the key information in the video is extracted and stored in a structured form.

2. The method for understanding streaming video and extracting key information structuredly according to claim 1, characterized in that: Determine the application type of video data includes: Determine whether the collected video data has a video application type label. If so, determine the corresponding application type based on the application type label; otherwise, input the video data into a constructed video application type recognition model, and output the application type of the video data; the video application type recognition model is generated by training a number of historical video data marked with application types.

3. The method for understanding streaming video and extracting key information structuredly according to claim 1, characterized in that: Also includes: After collecting the video data and determining its application type, continue to determine the sub-application type based on the application type classification according to the video data; According to the determined sub-application type, an adjustment weight of a preset feature weight of a determined application type is obtained from a second database; and a multi-feature comprehensive decision algorithm having the preset feature weight after weight adjustment of the determined application type is used to identify key frames in a video frame sequence.

4. The method for understanding streaming video and extracting key information structuredly according to claim 1, characterized in that: Also includes: Acquire a preset speech recognition model of the determined application type from a third database according to the determined application type, and at the same time, acquire a preset duration for recognizing speech information in a frame of video in a video frame sequence using the preset speech model of the determined application type; Statistics actually use a preset speech model of a determined application type to recognize the duration of a frame of video voice information in a video frame sequence. If the counted duration is less than the preset duration and the number of video frames is greater than the preset number of frames, the corresponding type of preset speech recognition model is switched to the spare one and the voice information in the video frame sequence is re-recognized.

5. The method for understanding streaming video and extracting key information structuredly according to claim 1, characterized in that: Also includes: The satisfaction score of the key information corresponding to the joint reasoning result of the video fed back by the user is obtained. If the satisfaction score is lower than the preset score threshold, the model parameters of the deep learning model or the cross-modal fusion network model are optimized until the preset score threshold is reached.

6. The method for understanding streaming video and extracting key information structured according to claim 1, characterized in that: Also includes: Design a visual display module to visualize the extracted key information in the form of charts, timelines, and tag clouds.

7. The method for understanding streaming video and extracting key information structuredly according to claim 1, characterized in that: The application types include: security application type and media application type according to application scenarios, content attributes and usage purposes.

8. A system for understanding streaming video and extracting key information structuredly, characterized in that: include: Video preprocessing module, used to collect video data and determine its application type; Acquire a preset per-frame sequence duration of the determined application type from a first database according to the determined application type, and decompose the video data into a video frame sequence according to the acquired preset per-frame sequence duration of the determined application type; A key frame extraction module, used for obtaining a preset feature weight of the determined application type from the second database according to the determined application type, and identifying key frames in the video frame sequence using a multi-feature comprehensive decision algorithm with the preset feature weight of the determined application type; The multi-feature comprehensive decision algorithm includes: motion detection decision mode, color histogram comparison decision mode and image entropy analysis decision mode; A natural language processing module, configured to obtain a preset speech recognition model of the determined application type from a third database according to the determined application type, recognize speech information in the video frame sequence using the preset speech model of the determined application type and convert the recognized speech into text, and obtain key text information using natural language processing technology; A video text fusion reasoning module is used to obtain a preset mapping association method for a determined application type from a fourth database according to the determined application type, and map and associate key frames and key text information identified by a video frame sequence according to the obtained preset mapping method for the determined application type; feature vectors of the key frames and feature vectors of the associated key text information are feature-fused through a cross-modal learning network model, and joint reasoning is performed in combination with a deep learning model to obtain a joint reasoning result of the video; the preset mapping association method includes: mapping association based on timestamp, mapping association based on scene description, and mapping association based on event description; The key information structured extraction module is used to extract key information from the video based on the joint reasoning results of the video and store it in a structured form.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A computer device, characterized in that: The computer device comprises a memory, a processor and a program stored and executable on the memory, and the program implements the steps of the method according to any one of claims 1 to 7 when executed by the processor.

Images