Video key frame acquisition method and device, computer equipment and storage medium

Abstract

The embodiment of the invention belongs to the field of digital medical treatment, and relates to a video key frame obtaining method and device, computer equipment and a storage medium, and the method comprises the following steps: obtaining audio data and video data; converting the audio data into an audio text; performing correlation analysis on the audio text and the video data to obtain a correlation corresponding relation; performing logic point detection on the audio text and the video data to obtain an audio logic point and a video logic point; generating an effective logic point according to the association corresponding relation, the audio logic point and the video logic point, and marking the video data according to the effective logic point to obtain marked video data; and carrying out key frame positioning based on the marked video data to obtain a video key frame. According to the invention, the accuracy of video key frame acquisition can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, specifically to the field of digital medicine, and in particular to a method, device, computer equipment and storage medium for acquiring video key frames. Background Art

[0002] In the medical field, double recording (audio and video) has become an important business process, aimed at recording customers' business transactions, especially important processes involving risk disclosure. Double recording refers to audio and video recording. Audio recording is information recorded based on the customer's voice, and video recording is information recorded based on the scene and picture where the customer is. Double recording can be used to effectively inspect or verify the business transactions conducted by customers.

[0003] In the process of inspection or verification of the business handled by the customer, the customer is usually required to show his or her ID card, insurance record, etc. For the recorded video, manual quality inspection is usually required to determine whether the recording process is compliant. The traditional method is to manually watch the video from the beginning to determine whether each environment is normal. Watching the video frame by frame is too inefficient. Usually we only need to look at the key frames of some links to determine whether this link is compliant. The existing technology uses specific algorithms to locate key frames, but it is often difficult to accurately distinguish between key frames and non-key frames in the video through specific algorithms, resulting in false detection (misjudging non-key frames as key frames) and missed detection (missing key frames), which seriously affects the accuracy and reliability of the key frame determination results. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, apparatus, computer device and storage medium for obtaining video key frames to solve the problem of being unable to accurately and effectively locate the key frames of a video.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a method for obtaining a video key frame, which adopts the following technical solution:

[0006] Get audio data and video data;

[0007] Converting the audio data into audio text;

[0008] Performing correlation analysis on the audio text and the video data to obtain a correlation correspondence relationship;

[0009] Performing logic point detection on the audio text and the video data to obtain audio logic points and video logic points;

[0010] Generate a valid logic point according to the associated correspondence, the audio logic point, and the video logic point, and mark the video data according to the valid logic point to obtain marked video data;

[0011] Key frame positioning is performed based on the marked video data to obtain video key frames.

[0012] Furthermore, the step of obtaining audio data and video data specifically includes:

[0013] Acquire extraction request information, and extract original audio data and original video data from a database according to the extraction request information;

[0014] Performing integrity and synchronization detection on the original audio data and the original video data to obtain valid audio data and valid video data;

[0015] The valid audio data and the valid video data are preprocessed to obtain the audio data and the video data.

[0016] Furthermore, the step of converting the audio data into an audio text specifically includes:

[0017] Extracting features from the audio data to obtain an audio feature vector;

[0018] Inputting the audio feature vector into a preset acoustic model for phoneme mapping to obtain audio phoneme information;

[0019] The audio phoneme information is input into a preset language model to obtain the audio text.

[0020] Furthermore, the step of performing correlation analysis on the audio text and the video data to obtain a correlation correspondence specifically includes:

[0021] Extracting feature vectors from the audio text to obtain audio text feature vectors;

[0022] Extracting feature vectors from the video data to obtain feature vectors of the video data;

[0023] Mapping the audio text feature vector and the video data feature vector to a pre-constructed embedding space to obtain a feature vector embedding space;

[0024] Performing distance measurement on feature vector points in the feature vector embedding space to obtain feature vector distance;

[0025] The feature vector distance is mapped to the associated correspondence relationship.

[0026] Furthermore, the step of performing logic point detection on the audio text and the video data to obtain audio logic points and video logic points specifically includes:

[0027] Performing semantic analysis on the audio text to obtain audio key information;

[0028] Performing visual analysis on the video data to obtain video visual elements;

[0029] Based on the audio key information and the video visual elements, logic point detection is performed on the audio text and the video data respectively to obtain the audio logic point and the video logic point.

[0030] Further, the step of generating a valid logic point according to the associated correspondence, the audio logic point, and the video logic point, and marking the video data according to the valid logic point to obtain the marked video data specifically includes:

[0031] Arranging the associated correspondence, the audio logic point, and the video logic point to obtain an associated logic point;

[0032] Acquire preset key point information, and screen the associated logic points according to the key point information to obtain valid logic points;

[0033] The time frame information corresponding to the valid logic point is obtained, and the valid logic point is added to the video data according to the time frame information to obtain the marked video data.

[0034] Furthermore, the step of performing key frame positioning based on the marked video data to obtain the video key frame specifically includes:

[0035] Segmenting the marked video data according to the valid logic points to obtain marked video segments;

[0036] Acquire logic point information corresponding to the valid logic point, and add a corresponding content tag to the marked video segment according to the logic point information;

[0037] Key frame information is obtained, and key frame search is performed on the marked video segment with the content tag added according to the key frame information to obtain the video key frame.

[0038] In order to solve the above technical problems, the embodiment of the present application further provides a video key frame acquisition device, which adopts the following technical solution:

[0039] A data acquisition module, used for acquiring audio data and video data;

[0040] A data conversion module, used for converting the audio data into audio text;

[0041] A relationship analysis module, used for performing association analysis on the audio text and the video data to obtain an associated corresponding relationship;

[0042] A logic detection module, used for performing logic point detection on the audio text and the video data to obtain audio logic points and video logic points;

[0043] A video marking module, used to generate a valid logic point according to the associated correspondence, the audio logic point, and the video logic point, and mark the video data according to the valid logic point to obtain marked video data;

[0044] The key frame positioning module is used to perform key frame positioning based on the marked video data to obtain video key frames.

[0045] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0046] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of any one of the above methods for acquiring video key frames when executing the computer-readable instructions.

[0047] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0048] A computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the video key frame acquisition method as described in any one of the above items are implemented.

[0049] Compared with the prior art, the embodiment of the present application has the following beneficial effects: the embodiment acquires audio data and video data; converts the audio data into audio text; performs correlation analysis on the audio text and the video data to obtain a correlation correspondence; performs logic point detection on the audio text and the video data to obtain audio logic points and video logic points; generates valid logic points according to the correlation correspondence, the audio logic points, and the video logic points, and marks the video data according to the valid logic points to obtain marked video data; performs key frame positioning based on the marked video data to obtain video key frames. Thus, video key frame detection is effectively performed according to the logic of audio and video to improve the accuracy of video key frame detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0052] Figure 2 A flowchart of an embodiment of a method for acquiring a video key frame according to the present application;

[0053] Figure 3 yes Figure 2 A flowchart of a specific implementation of step S10;

[0054] Figure 4 yes Figure 2 A flowchart of a specific implementation of step S20;

[0055] Figure 5 yes Figure 2 A flowchart of a specific implementation of step S30;

[0056] Figure 6 yes Figure 2 A flowchart of a specific implementation of step S40;

[0057] Figure 7 yes Figure 2 A flowchart of a specific implementation of step S50;

[0058] Figure 8 yes Figure 2 A flowchart of a specific implementation of step S60;

[0059] Fig. 9 is a structural schematic diagram of an embodiment of a video key frame acquisition device according to the present application;

[0060] Fig.10 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0062] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor does it refer to non-related or alternative embodiments that are mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0063] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0064] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0065] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0066] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0067] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0068] It should be noted that the video key frame acquisition method provided in the embodiment of the present application is generally executed by a server, and accordingly, the video key frame acquisition device is generally set in the server.

[0069] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0070] Continue to refer Figure 2 , shows a flow chart of an embodiment of the method for image recognition according to the present application. The method for obtaining video key frames comprises the following steps:

[0071] Step S10, obtaining audio data and video data;

[0072] In this embodiment, audio data refers to voice recordings collected during the medical insurance authentication process. Voice recordings are recordings of the applicant's voice communication in a telephone authentication, remote consultation, or self-service system, and are used to verify the applicant's identity, confirm their application intention, or understand their health status. Video data refers to video recordings collected during the medical insurance authentication process. Video recordings are video images that display identity information, facial images, and physical conditions, and are used to verify the applicant's identity, confirm their health status, and determine whether they meet specific conditions. In this embodiment, audio data and video data correspond in real time, and the time frame length corresponding to the audio signal of the audio data is the same as the time frame length corresponding to the video data, and the audio data and video data can be extracted from the same video stream data.

[0073] Step S20, converting the audio data into audio text;

[0074] In this embodiment, the audio data is a recording record, which refers to the sound information captured and saved by a recording device (such as a mobile phone, a computer, a system terminal, etc.), and the recording record can be used to verify the identity of the applicant. For example, during the authentication process, the voice characteristics of the applicant can be recorded and compared with the previously registered voice sample to confirm the authenticity of their identity. The information provided by the applicant during the authentication process, such as basic personal information, description of health status, willingness to apply, etc., can also be recorded. This information is essential for subsequent review and evaluation of the applicant's qualifications. The audio text is the text corresponding to the recording record. In addition to the text content, the audio text also includes the time frame information corresponding to the text. By converting the audio data into audio text, it is convenient to process it.

[0075] Step S30, performing correlation analysis on the audio text and the video data to obtain a correlation correspondence relationship;

[0076] In this embodiment, the association analysis refers to the analysis of the association between the sound field in the audio text and the image element in the video data, and the association correspondence refers to the correspondence between the sound field in the audio text and the image element in the video data. Specifically, the association correspondence refers to the correspondence consistency between the semantic information corresponding to the sound field and the action or scene depicted by the image element. For example, if the sound field is "Please put the identity document in the specified box range", the image displayed in the image element corresponds to the identity document appearing in the specified box range.

[0077] Step S40, performing logic point detection on the audio text and the video data to obtain audio logic points and video logic points;

[0078] In this embodiment, in the audio text, the audio logic points may include: Keywords or phrases: such as names of people, places, time, numbers, etc., which are crucial to understanding the content of the text. Changes in sentence structure: such as the introduction of clauses, the appearance of transition words, etc., these changes often mark the transformation of the logical structure of the text. Emotional transitions: such as changes from positive to negative emotions, these changes help to understand the speaker's intentions and attitudes. In video data, video logic points may include: Important pictures: representative pictures in the video that can summarize the main content or emotions of the video. Scene switching: The transition between different scenes usually marks the development of the story or plot. Action changes: such as the beginning, end or change of a character's action, these changes help to understand the events and dynamics in the video.

[0079] Step S50, generating a valid logic point according to the associated correspondence, the audio logic point, and the video logic point, and marking the video data according to the valid logic point to obtain marked video data;

[0080] In this embodiment, the valid logic point refers to the video logic point after being screened by preset key point information, wherein the preset key point information is an artificially set video key attention point, such as the identity authentication stage in the video, the document inspection stage in the video, the face authentication stage in the video, etc. The preset key point information is used to perform logic point screening to obtain the logic point corresponding to the preset key point information, and the corresponding logic point is used as the valid logic point.

[0081] Step S60: performing key frame positioning based on the marked video data to obtain video key frames.

[0082] In this embodiment, the key frame refers to the image frame of the important link in the video data, which is used to determine whether there is an abnormality in the important link and whether the image frame content appearing in the important link is the correct image frame content. For example, in the identity authentication link, the key frame of the video should be the image frame of the user presenting his / her identification document, and this image frame is used to determine whether the identity authentication link has been effectively executed.

[0083] In this embodiment, the above method can be applied to the insurance verification of health insurance. In the insurance verification, the audio and video records submitted by the user are collected, and the key frames of the video records are checked according to the needs to determine whether the user has executed the corresponding steps in the video records. The system corresponding to the above health insurance can be one or more of the medical insurance system and the disease insurance system. The audio data and video data correspond to the above audio and video records respectively. The audio data and video data are stored in the medical insurance system and the disease insurance system and obtained from the database of the above system to verify the above health insurance. Specifically, in this embodiment, the video key frame corresponds to the video key frame of the insurance verification in the health insurance. The insurance verification is the insurance information inspection and identity inspection of the insured object submitted in the health insurance, which can be used for health insurance claims or annual inspections; the video data is the video recording of the insured object of the health insurance; and the audio data is the recording of the insured object of the health insurance.

[0084] This embodiment obtains a multi-dimensional scar facial image; extracts features from the multi-dimensional scar facial image to obtain multi-dimensional scar facial features; constructs a three-dimensional scar facial model based on the multi-dimensional scar facial image and the multi-dimensional scar facial features; extracts features based on the three-dimensional scar facial model to obtain effective multi-dimensional scar facial features; performs model training based on the effective multi-dimensional scar facial features to obtain a scar facial recognition model; obtains a real facial image, inputs the real facial image into the scar facial recognition model, and obtains a facial scar recognition result. This embodiment can effectively identify facial scars on a human face and improve the accuracy of recognition.

[0085] Continue to refer Figure 3 In some optional implementations of this embodiment, step S10 includes the following steps:

[0086] Step S101, obtaining extraction request information, and extracting original audio data and original video data from a database according to the extraction request information;

[0087] In this embodiment, the extraction request information is an extraction information identifier corresponding to the original audio data and the original video data, and the extraction information identifier can be a unique identifier. The original audio data and the original audio data are correspondingly synchronized and can be extracted from the same video stream data. By matching the extraction request information with the data identifiers of the original audio data and the original video data, a search is performed in the database to effectively obtain the original audio data and the original video data. The original audio data refers to the recording data collected directly through a recording device, a microphone, etc., and the recording data has not been processed in any way; the original video data refers to the video data collected directly through a video recording device, a camera, etc., and the video data has not been processed in any way.

[0088] Step S102, performing integrity and synchronization detection on the original audio data and the original video data to obtain valid audio data and valid video data;

[0089] In this embodiment, the integrity detection of the original audio data includes: checking whether the size of the audio file is consistent with its expectation or standard, and whether the file format is correct (such as MP3, WAV, etc.); decoding the original audio data, and checking whether errors or exceptions occur during the decoding process. If the decoding fails, it may indicate that the file is damaged; calculating the MD5 checksum of the audio file and comparing it with the known correct checksum. If the two do not match, the file may have been tampered with or damaged; checking the continuity of the audio content of the original audio data to determine whether it is complete. The integrity detection of the original video data includes: checking whether the video file size and format of the original video data meet expectations; checking whether it can be played normally without error prompts through the video decoder; checking whether each frame in the video is complete and there is no missing or damaged situation through the video processing software. The synchronization detection of the original audio data and the original video data includes: obtaining the timestamp information in the original audio data and the original video data, and comparing whether they are consistent, or the deviation between the two is within an acceptable range (such as within 200ms), so as to determine whether the original audio data and the original video data meet the synchronization requirements.

[0090] Step S103: pre-process the valid audio data and the valid video data to obtain the audio data and the video data.

[0091] In this embodiment, the preprocessing of effective audio data includes: denoising: using filters and other algorithms to remove background noise in the audio; enhancement: improving the sound quality through equalization, compression and other technologies. The preprocessing of effective video data includes: frame rate adjustment: adjusting the frame rate of the video as needed; resolution adjustment: adjusting the resolution of the video according to the playback device or network conditions; color correction: adjusting the color balance and saturation of the video to make the picture more natural.

[0092] This embodiment obtains extraction request information, extracts original audio data and original video data from a database according to the extraction request information; performs integrity and synchronization detection on the original audio data and the original video data to obtain valid audio data and valid video data; and pre-processes the valid audio data and the valid video data, thereby effectively obtaining complete, valid and synchronized audio data and video data to facilitate subsequent processing.

[0093] Continue to refer Figure 4 In some optional implementations of this embodiment, step S20 includes the following steps:

[0094] Step S201, extracting features from the audio data to obtain an audio feature vector;

[0095] In this embodiment, the audio feature vector includes Mel-frequency cepstral coefficients (MFCCs), linear prediction coefficients (LPCs), and spectral features, wherein Mel-frequency cepstral coefficients (MFCCs): are a feature widely used in speech recognition, which simulates the way the human ear perceives frequency and extracts important frequency information from the audio signal. Linear prediction coefficients (LPCs): are used to represent the linear prediction model of the audio signal, which can capture characteristics such as resonance peaks in the audio signal. Spectral features: such as short-time energy, short-time zero-crossing rate, etc., are used to describe the spectral characteristics of the audio signal. In this embodiment, feature extraction of audio data is achieved through the following steps: pre-emphasis processing is performed on the audio data through a Gaussian filter to enhance the high-frequency components of the audio data; the audio data is framed through a sliding window technique to obtain a plurality of audio short-time frames; the audio short-time frames are windowed to make the transition between frames more natural; the windowed frames are fast Fourier transformed to obtain a frequency spectrum representation of the frame, i.e., a frame spectrum; the frame spectrum is filtered through a Mel filter group; a logarithmic operation is performed on the filtered frame spectrum to obtain a logarithmic result of the frame spectrum; and a discrete cosine transform is performed on the logarithmic result of the frame spectrum to obtain an audio feature vector.

[0096] Step S202, inputting the audio feature vector into a preset acoustic model for phoneme mapping to obtain audio phoneme information;

[0097] In this embodiment, phoneme is the smallest unit in phonetics, which is the smallest speech segment constituting a syllable and the smallest speech unit divided from the perspective of sound quality. Phoneme mapping is the process of converting audio feature vectors into phonemes. In this embodiment, the preset acoustic model can adopt an acoustic model based on a hidden Markov model (HMM). HMM is a statistical model used to describe a Markov process with hidden unknown parameters. The corresponding phoneme pronunciation or word sequence is identified by modeling the time series of the speech signal and performing statistical analysis of the speech features.

[0098] Step S203: input the audio phoneme information into a preset language model to obtain the audio text.

[0099] In this embodiment, the audio phoneme information includes all phonemes corresponding to the audio data. The preset language model may adopt a statistical language model. The statistical language model can obtain the probability distribution of a given word sequence by processing a large amount of text corpus based on statistical methods. In this embodiment, the statistical language model adopts a neural network language model to model the probability distribution of word sequences through a neural network (such as RNN, LSTM, Transformer, etc.), so as to capture more complex language rules and contextual information, so as to generate corresponding audio text according to the audio phoneme information.

[0100] This embodiment obtains an audio feature vector by extracting features from the audio data; inputs the audio feature vector into a preset acoustic model for phoneme mapping to obtain audio phoneme information; and inputs the audio phoneme information into a preset language model, thereby effectively obtaining an audio text converted accordingly to the audio data, to facilitate subsequent association analysis with video data.

[0101] Continue to refer Figure 5 In some optional implementations of this embodiment, step S30 includes the following steps:

[0102] Step S301, extracting a feature vector from the audio text to obtain an audio text feature vector;

[0103] In this embodiment, the audio text is processed by word embedding to obtain the audio text feature vector corresponding to the audio text, and the word embedding process includes the following steps: the vector representation of each word in the model is searched in the audio text by the preset BERT model, and a corresponding feature vector is obtained for each word in the audio text. In order to represent the entire audio text, the feature vectors of all words in the audio text can be aggregated, such as taking the average, taking the average after weighting with TF-IDF, encoding using a more complex sequence model (such as LSTM, GRU), etc., so as to obtain the audio text feature vector.

[0104] Step S302, extracting feature vectors from the video data to obtain feature vectors of the video data;

[0105] In this embodiment, by performing frame extraction on the video data, a video extraction image frame is obtained, and then the video extraction image frame is subjected to the following operations to obtain a video data feature vector: Grayscale feature extraction: Calculate the statistical features of the grayscale value of the video extraction image frame, such as mean, variance, energy, entropy, etc. These features can reflect the distribution of different grayscale levels in the image. Color feature extraction: Extract the value of the RGB color channel of each pixel in the video extraction image frame, and calculate its statistical features, such as color histogram, color mean, variance, etc. to obtain color features. The color histogram is constructed by counting the frequency of occurrence of each color value in the image, which can reflect the distribution of different colors in the image. Texture feature extraction: The texture feature is represented by calculating the relationship between the pixels in the local area of ​​the video extraction image frame. Common algorithms include local binary pattern (Local Binary Pattern, LBP), gray-level co-occurrence matrix (Gray-Level Co-occurrence Matrix, GLCM), etc. After obtaining the grayscale features, color features and texture features of the video extraction image frame, the grayscale features, color features and texture features are combined to obtain a video data feature vector.

[0106] Step S303, mapping the audio text feature vector and the video data feature vector to a pre-constructed embedding space to obtain a feature vector embedding space;

[0107] In this embodiment, the pre-constructed embedding space refers to the representation space of word vectors or higher-level text units (such as sentences and paragraphs) pre-trained on a large amount of text data. This representation space captures the semantic and grammatical relationships between vocabulary or text units, so that the feature vector can be effectively mapped to the embedding space to obtain the corresponding representation, and the embedding space can be constructed by BERT.

[0108] Step S304, performing distance measurement on the feature vector points in the feature vector embedding space to obtain the feature vector distance;

[0109] In this embodiment, the distance metric is to calculate the distance or similarity between each pair of audio text embedding vectors and video data embedding vectors, which can be achieved by calculating the Euclidean distance of feature vector points in the feature vector embedding space. When the calculation is completed, a distance or similarity matrix is ​​obtained, and each element in the matrix represents the distance or similarity between a pair of feature vectors.

[0110] Step S305: Map the feature vector distance to the associated correspondence relationship.

[0111] In this embodiment, the association correspondence is judged according to the set distance threshold or similarity threshold. When the distance between two feature vectors is less than the distance threshold or the similarity is greater than the similarity threshold, it is considered that there is an association between them. By traversing the distance or similarity matrix, the feature vectors that meet the threshold conditions are mapped to the association correspondence, and the association correspondence is output.

[0112] This embodiment extracts feature vectors from the audio text to obtain audio text feature vectors; extracts feature vectors from the video data to obtain video data feature vectors; maps the audio text feature vectors and the video data feature vectors to a pre-constructed embedding space to obtain a feature vector embedding space; measures the distance of feature vector points in the feature vector embedding space to obtain feature vector distances; and maps the feature vector distances to the associated correspondence. This effectively achieves the acquisition of the associated correspondence between the audio text and the video data, so as to facilitate the subsequent generation of effective logic points.

[0113] Continue to refer Figure 6 In some optional implementations of this embodiment, step S40 includes the following steps:

[0114] Step S401, performing semantic analysis on the audio text to obtain audio key information;

[0115] In this embodiment, the semantic analysis of the audio text includes: Named Entity Recognition (NER): identifying named entities in the text, such as names of people, places, and institutions; Dependency Syntactic Analysis: analyzing the dependency relationship between words in a sentence to construct the syntactic structure of the sentence; Semantic Role Labeling (SRL): identifying the relationship between predicates and arguments in a sentence to further understand the semantics of the sentence; Based on the above analysis results, extracting audio key information in the audio text, such as events, links, etc.

[0116] Step S402, performing visual analysis on the video data to obtain video visual elements;

[0117] In this embodiment, the visual analysis of video data includes: object detection: identifying objects in video frames, such as people, documents, etc.; scene recognition: analyzing scene types in video frames, such as indoors, outdoors, etc.; action recognition: detecting the actions of people or objects in video frames, such as walking, waving, etc.; based on the above analysis results, extracting key visual elements in the video, such as main characters, key actions, etc.

[0118] Step S403: Perform logic point detection on the audio text and the video data based on the audio key information and the video visual elements to obtain the audio logic point and the video logic point.

[0119] In this embodiment, the audio logic point detection includes: determining the audio logic point according to key information in the audio text, such as the turning point of the main event, the starting point of the link, etc. The video logic point detection includes: determining the video logic point according to key visual elements in the video, such as the change of the subject, the occurrence of key actions of the characters, etc., and by analyzing the action and scene coherence between video frames.

[0120] This embodiment obtains audio key information by performing semantic analysis on the audio text; obtains video visual elements by performing visual analysis on the video data; and performs logic point detection on the audio text and the video data based on the audio key information and the video visual elements, thereby obtaining audio logic points related to audio logic and video logic points related to video logic, so as to facilitate the subsequent acquisition of effective logic points.

[0121] Continue to refer Figure 7 In some optional implementations of this embodiment, step S403 includes the following steps:

[0122] Step S501, sorting the associated correspondence, the audio logic point, and the video logic point into logic point correspondences to obtain associated logic points;

[0123] In this embodiment, the steps of organizing the correspondence of logical points are as follows: integrating the audio logical points and the video logical points, aligning them by means of timestamps or frame numbers, etc., to ensure the logical consistency of the audio and video content, and then selecting the logical points with associated correspondence from the audio logical points and the video logical points as associated logical points.

[0124] Step S502, obtaining preset key point information, and screening the associated logic points according to the key point information to obtain valid logic points;

[0125] In this embodiment, the effective logic point refers to the video logic point after being screened by preset key point information. The preset key point information is an artificially set key attention point of the video, for example, the identity authentication stage in the video, the document inspection stage in the video, the face authentication stage in the video, etc. The preset key point information can be set accordingly according to actual needs to ensure that the effective logic points obtained after screening can meet the corresponding needs.

[0126] Step S503, obtaining the time frame information corresponding to the valid logic point, and adding the valid logic point to the video data according to the time frame information to obtain the marked video data.

[0127] In this embodiment, the time frame information of the valid logic point can be obtained according to the audio timestamp of the valid logic point in the audio text or the video timestamp in the video data, wherein the audio timestamp is the time corresponding to the word or phrase in the audio text, and the video timestamp is the time corresponding to the video frame image in the video data. After the time frame information is obtained, the valid logic point can be added to the video data as a marked text by superimposing the marked text on the video frame of the video data, so as to obtain the marked video data showing the description or identifier of the valid logic point.

[0128] This embodiment obtains associated logic points by sorting out the associated correspondence, the audio logic points, and the video logic points through logic point correspondence; obtains preset key point information, and filters the associated logic points according to the key point information to obtain valid logic points; obtains time frame information corresponding to the valid logic points, and adds the valid logic points to the video data according to the time frame information, thereby obtaining marked video data with logic points that meet the key point information, so as to facilitate subsequent key frame positioning.

[0129] Continue to refer Figure 8 In some optional implementations of this embodiment, step S60 includes the following steps:

[0130] Step S601, dividing the marked video data according to the valid logic points to obtain marked video segments;

[0131] In this embodiment, based on the time frame information (start time and end time) of each valid logical point, these time points are found in the marked video data, and the marked video data is divided into multiple marked video segments based on the time points, wherein each marked video segment contains content starting from a valid logical point and ending at the next valid logical point (if it is the last valid logical point, then to the end of the video).

[0132] Step S602, obtaining logic point information corresponding to the valid logic point, and adding a corresponding content tag to the marked video segment according to the logic point information;

[0133] In this embodiment, the logic point information is a description of a valid logic point, such as an event or action occurring in a time period after the valid logic point. The content tag corresponding to the marked video segment is descriptive information of the content corresponding to the video segment. The logic point information is added to the content tag to display the events or actions occurring in the time period corresponding to the marked video segment.

[0134] Step S603, obtaining key frame information, and performing key frame search on the marked video segment with the content tag added according to the key frame information to obtain the video key frame.

[0135] In this embodiment, a key frame refers to an image frame of an important link in the video data, and the image frame is used to determine whether there is an abnormality in the important link and whether the image frame content appearing in the important link is the correct image frame content. For example, in the identity verification link, the key frame of the video should be the image frame of the user showing the identity document, and the image frame is used to determine whether the identity verification link has been effectively executed. The key frame information is the marking information corresponding to the key frame, which contains the characteristics of the corresponding key frame. The key frame search is performed through the key frame information to accurately obtain the video key frame in the marked video segment.

[0136] This embodiment obtains marked video segments by dividing the marked video data according to the valid logical points; obtains the logical point information corresponding to the valid logical points, and adds corresponding content tags to the marked video segments according to the logical point information; obtains key frame information, and performs key frame search on the marked video segments with added content tags according to the key frame information, thereby obtaining video key frames corresponding to the requirements to facilitate subsequent judgment or review operations.

[0137] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0138] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0139] Further references Fig. 9 , as a response to the above Figure 1 The present application provides an embodiment of a video key frame acquisition device, and the device embodiment is similar to Figure 1Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0140] like Fig. 9 As shown, the video key frame acquisition device 700 described in this embodiment includes: a data acquisition module 701, a data conversion module 702, a relationship analysis module 703, a logic detection module 704, a video marking module 705, and a key frame positioning module 706. Among them:

[0141] Data acquisition module 701, used to acquire audio data and video data;

[0142] A data conversion module 702, used for converting the audio data into audio text;

[0143] A relationship analysis module 703 is used to perform a correlation analysis on the audio text and the video data to obtain a correlation correspondence relationship;

[0144] A logic detection module 704 is used to perform logic point detection on the audio text and the video data to obtain audio logic points and video logic points;

[0145] The video marking module 705 is used to generate a valid logic point according to the associated correspondence, the audio logic point, and the video logic point, and mark the video data according to the valid logic point to obtain marked video data;

[0146] The key frame positioning module 706 is used to perform key frame positioning based on the marked video data to obtain video key frames.

[0147] By adopting the above-mentioned device, this embodiment can obtain audio data and video data; convert the audio data into audio text; perform correlation analysis on the audio text and the video data to obtain a correlation correspondence relationship; perform logic point detection on the audio text and the video data to obtain audio logic points and video logic points; generate effective logic points according to the correlation correspondence relationship, the audio logic points, and the video logic points, and mark the video data according to the effective logic points to obtain marked video data; perform key frame positioning based on the marked video data to obtain video key frames. In this way, video key frame detection is effectively performed according to the logic of audio and video to improve the accuracy of video key frame detection.

[0148] To solve the above technical problems, the present application also provides a computer device. Fig.10 , Fig.10 This is a basic structural block diagram of the computer device in this embodiment.

[0149] The computer device 8 includes a memory 81, a processor 82, and a network interface 83 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 8 with components 81-83, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0150] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0151] The memory 81 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 81 can be an internal storage unit of the computer device 8, such as a hard disk or memory of the computer device 8. In other embodiments, the memory 81 can also be an external storage device of the computer device 8, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 8. Of course, the memory 81 can also include both the internal storage unit of the computer device 8 and its external storage device. In this embodiment, the memory 81 is generally used to store the operating system and various application software installed on the computer device 8, such as computer-readable instructions of the video key frame acquisition method, etc. In addition, the memory 81 can also be used to temporarily store various types of data that have been output or are to be output.

[0152] The processor 82 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 82 is generally used to control the overall operation of the computer device 8. In this embodiment, the processor 82 is used to run computer-readable instructions stored in the memory 81 or process data, such as computer-readable instructions for running the video key frame acquisition method.

[0153] The network interface 83 may include a wireless network interface or a wired network interface. The network interface 83 is generally used to establish a communication connection between the computer device 8 and other electronic devices.

[0154] By adopting the above-mentioned computer device, this embodiment can obtain audio data and video data; convert the audio data into audio text; perform correlation analysis on the audio text and the video data to obtain a correlation correspondence relationship; perform logic point detection on the audio text and the video data to obtain audio logic points and video logic points; generate effective logic points according to the correlation correspondence relationship, the audio logic points, and the video logic points, and mark the video data according to the effective logic points to obtain marked video data; perform key frame positioning based on the marked video data to obtain video key frames. In this way, video key frame detection is effectively performed according to the logic of audio and video to improve the accuracy of video key frame detection.

[0155] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the video key frame acquisition method as described above.

[0156] By adopting the above-mentioned computer-readable storage medium, this embodiment can obtain audio data and video data; convert the audio data into audio text; perform association analysis on the audio text and the video data to obtain an association correspondence relationship; perform logic point detection on the audio text and the video data to obtain audio logic points and video logic points; generate effective logic points according to the association correspondence relationship, the audio logic points, and the video logic points, and mark the video data according to the effective logic points to obtain marked video data; perform key frame positioning based on the marked video data to obtain video key frames. In this way, video key frame detection is effectively performed according to the logic of audio and video to improve the accuracy of video key frame detection.

[0157] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0158] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for obtaining a video key frame, characterized in that: The steps include: Get audio data and video data; Converting the audio data into audio text; Performing correlation analysis on the audio text and the video data to obtain a correlation correspondence relationship; Performing logic point detection on the audio text and the video data to obtain audio logic points and video logic points; Generate a valid logic point according to the associated correspondence, the audio logic point, and the video logic point, and mark the video data according to the valid logic point to obtain marked video data; Key frame positioning is performed based on the marked video data to obtain video key frames.

2. The video key frame acquisition method according to claim 1, characterized in that: The step of obtaining audio data and video data specifically includes: Acquire extraction request information, and extract original audio data and original video data from a database according to the extraction request information; Performing integrity and synchronization detection on the original audio data and the original video data to obtain valid audio data and valid video data; The valid audio data and the valid video data are preprocessed to obtain the audio data and the video data.

3. The video key frame acquisition method according to claim 1, characterized in that: The step of converting the audio data into an audio text specifically comprises: Extracting features from the audio data to obtain an audio feature vector; Inputting the audio feature vector into a preset acoustic model for phoneme mapping to obtain audio phoneme information; The audio phoneme information is input into a preset language model to obtain the audio text.

4. The video key frame acquisition method according to claim 1, characterized in that: The step of performing correlation analysis on the audio text and the video data to obtain a correlation correspondence specifically includes: Extracting feature vectors from the audio text to obtain audio text feature vectors; Extracting feature vectors from the video data to obtain feature vectors of the video data; Mapping the audio text feature vector and the video data feature vector to a pre-constructed embedding space to obtain a feature vector embedding space; Performing distance measurement on feature vector points in the feature vector embedding space to obtain feature vector distance; The feature vector distance is mapped to the associated correspondence relationship.

5. The video key frame acquisition method according to claim 1, characterized in that: The step of performing logic point detection on the audio text and the video data to obtain audio logic points and video logic points specifically includes: Performing semantic analysis on the audio text to obtain audio key information; Performing visual analysis on the video data to obtain video visual elements; Based on the audio key information and the video visual elements, logic point detection is performed on the audio text and the video data respectively to obtain the audio logic point and the video logic point.

6. The video key frame acquisition method according to claim 1, characterized in that: The step of generating a valid logic point according to the associated correspondence, the audio logic point, and the video logic point, and marking the video data according to the valid logic point to obtain the marked video data specifically includes: Arranging the associated correspondence, the audio logic point, and the video logic point to obtain an associated logic point; Acquire preset key point information, and screen the associated logic points according to the key point information to obtain valid logic points; The time frame information corresponding to the valid logic point is obtained, and the valid logic point is added to the video data according to the time frame information to obtain the marked video data.

7. The video key frame acquisition method according to claim 1, characterized in that: The step of performing key frame positioning based on the marked video data to obtain the video key frame specifically includes: Segmenting the marked video data according to the valid logic points to obtain marked video segments; Acquire logic point information corresponding to the valid logic point, and add a corresponding content tag to the marked video segment according to the logic point information; Key frame information is obtained, and key frame search is performed on the marked video segment with the content tag added according to the key frame information to obtain the video key frame.

8. A video key frame acquisition device, characterized in that: include: A data acquisition module, used for acquiring audio data and video data; A data conversion module, used for converting the audio data into audio text; A relationship analysis module, used for performing association analysis on the audio text and the video data to obtain an associated corresponding relationship; A logic detection module, used for performing logic point detection on the audio text and the video data to obtain audio logic points and video logic points; A video marking module, used to generate a valid logic point according to the associated correspondence, the audio logic point, and the video logic point, and mark the video data according to the valid logic point to obtain marked video data; The key frame positioning module is used to perform key frame positioning based on the marked video data to obtain video key frames.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the video key frame acquisition method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the video key frame acquisition method according to any one of claims 1 to 7 are implemented.

Images