Interrogation record and audio and video recording linkage method and system

Abstract

The invention provides an interrogation record and audio and video recording linkage method and system, and belongs to the technical field of information processing, and the method comprises the steps: obtaining interrogation record text data and segmented video recording data, and respectively labeling category labels; performing semantic extraction on the record text data to obtain semantic feature information, and converting the feature information into text semantic feature vectors; frame extraction is carried out on the video data, image features of each frame of the video data are extracted, similarity screening is carried out, and the image features are converted into key frame semantic feature vectors; constructing a cross-modal embedding space model, respectively encoding the text semantic feature vector and the key frame semantic feature vector of the category label through two encoders, then splicing and fusing to obtain a fused feature vector, converting and mapping the fused feature vector into a sharing layer, completing feature association of the record text data and the segmented video recording data, and obtaining the segmented video recording data. And subsequently, cross-modal search is carried out. According to the method, cross-modal information association is realized through association of the semantic feature vectors of the record and video recording information, and a basis is provided for subsequent rapid calling, research and judgment.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a method and system for linking interrogation records with audio and video recordings. Background Art

[0002] The practice of correlating interrogation transcripts with audio and video recordings is a common practice in modern judicial procedures to ensure transparency and fairness in the interrogation process. Interrogation transcripts and audio and video recordings must be recorded simultaneously. During the interrogation, both transcripts and audio and video recordings are recorded simultaneously to ensure consistency between the written and audio and video recordings. After the interrogation, relevant personnel must review the transcripts and audio and video recordings to ensure consistency. Any discrepancies must be corrected within the legal scope.

[0003] Since interrogation information needs to be repeatedly called and analyzed during the interrogation process or subsequent information calls, due to the diversity of interrogation materials, the data types of transcripts and audio and video recordings are completely different, and they need to be classified and stored during storage. In order to better associate the text data of the transcripts with the audio and video recordings for subsequent analysis and judgment, the existing technology usually divides the text data and audio and video recordings into multiple time periods and packages them separately to achieve a better correspondence. However, this method of storing multiple types of data in segmented packages greatly increases the workload of data storage and is not convenient for retrieval and calling during subsequent analysis and judgment. Summary of the Invention

[0004] To solve the above problems, the present invention provides a method and system for linking interrogation records with audio and video recordings. This method achieves cross-modal information association by associating the semantic feature vectors of the record and video information, providing a basis for subsequent rapid retrieval and analysis.

[0005] To achieve the above objectives, the present invention provides the following technical solutions.

[0006] A method for linking interrogation records with audio and video recordings, comprising the following steps:

[0007] Obtaining the interrogation transcript and video data; labeling the transcript and video data with category labels, and matching transcripts with a certain segment of video data with the same category label;

[0008] Performing semantic extraction on the transcript text data to obtain semantic feature information of the transcript text data, and converting the feature information into a text semantic feature vector;

[0009] Perform frame extraction on the video data, extract the image features of each frame of the video data, and perform similarity screening on the image features to obtain key frame image features; obtain key frame semantic features through the semantic extraction model and convert them into key frame semantic feature vectors;

[0010] Construct a cross-modal embedding space model, including two different modality encoders and a shared layer with a high-dimensional vector space;

[0011] The text semantic feature vector of the category label and the key frame semantic feature vector are encoded by two encoders respectively, and then concatenated and fused to obtain a fused feature vector, which is then converted and mapped into the shared layer to complete the feature association between the transcript text data and the segmented video data;

[0012] Input transcript text query data or video data, convert it into a text semantic feature vector or a keyframe semantic feature vector as an input feature vector;

[0013] According to the input feature vector, the trained cross-modal embedding space model is used to obtain the fused feature vector with the highest similarity, and the corresponding interrogation transcript text data and segmented video data are used as the results.

[0014] Preferably, it also includes determining the corresponding key frame semantic feature vector based on the fusion feature vector with the highest similarity; determining the key frame position of the video data based on the time series information of the key frame semantic feature vector, and retrieving the video segment with preset time length requirements and containing key frames.

[0015] Preferably, the dot product between the fused feature vector and the input feature vector is used to determine similarity, and the candidate objects are sorted according to the similarity; and the sorted candidate objects are output to the user as retrieval results.

[0016] Preferably, the text semantic feature vector and the key frame semantic feature vector are converted to a shared layer and before being fused, a fully connected layer output mapping is used to convert feature vectors from different modalities into vectors with the same dimension.

[0017] Preferably, the semantic extraction of the transcript text data to obtain semantic feature information of the transcript text data includes the following steps:

[0018] Clean, segment, fill, and remove stop words from the transcript text data to obtain valid transcript text data;

[0019] The valid transcript text data is used to extract semantic feature information through the BERT model and convert it into numerical vectorized semantic feature information.

[0020] Preferably, the step of performing frame extraction on the video data, extracting image features of each frame of the video data, and performing similarity screening on the image features to obtain key frame image features comprises the following steps:

[0021] Extract frames at specific time intervals to obtain image data for each frame;

[0022] Edge detection is used to obtain image contour features in image data, and a number of image contour feature contour points are extracted to form a contour point set;

[0023] Calculate the bidirectional Hausdorff distance between the contour point sets of each frame image data as the basis for the similarity matching degree, that is, the similarity value between each frame image data;

[0024] According to the preset similarity threshold, frames with similarity values lower than the threshold are regarded as key frames.

[0025] Preferably, the step of obtaining key frame semantic features through a semantic extraction model and converting the key frame semantic feature vectors includes the following steps:

[0026] Extract key frame semantic feature vectors through the pre-trained Transformer model;

[0027] The key frame semantic feature vector is reduced in dimension by the t-SNE algorithm to keep the same dimension as the text semantic feature vector and normalized.

[0028] A system for linking interrogation records with audio and video recordings, comprising:

[0029] processor;

[0030] a memory having stored thereon a computer program executable on the processor;

[0031] When the computer program is executed by the processor, the steps of the method for linking the interrogation record with audio and video recording are implemented.

[0032] A computer-readable storage medium stores a data processing program, which, when executed by a processor, implements the steps of the method for linking interrogation records with audio and video recordings.

[0033] Beneficial effects of the present invention:

[0034] The present invention proposes a method and system for linking interrogation transcripts with audio and video recordings. In order to achieve the purpose of synchronously linking multimodal data of interrogations, the method proposes a method of extracting and splicing multimodal semantic feature vectors to perform information association, thereby realizing the linkage between transcripts and audio and video recordings without the need for multi-type data packaging. In addition, when the subsequent interrogation information is called, it can be accurately located through semantic features and cross-modal retrieval can be performed, providing a basis for subsequent rapid calling and analysis. The method adopts a cross-modal embedding space model to map the features of different modalities to the common space of the cross-modal embedding space model to obtain vector representations of the same dimension, so that the text semantic feature vectors and the key frame semantic feature vectors can be aligned and in the same dimension, thereby improving the feasibility of subsequent feature association and the accuracy of cross-modal retrieval. The method realizes cross-modal information association by linking the semantic feature vectors of transcript and video information, providing a basis for subsequent rapid calling and analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is an overall flow chart of an embodiment of the present invention;

[0036] Figure 2 is a flow chart of the key frame image feature extraction steps of an embodiment of the present invention;

[0037] Figure 3 Flowchart of the steps for extracting semantic feature information from transcript text data according to an embodiment of the present invention;

[0038] Figure 4 4 is a flowchart of the key frame semantic feature vector extraction steps of an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0040] Example 1

[0041] The interrogation information needs to be repeatedly called and analyzed during the interrogation process or the subsequent information calling process. However, due to the diversity of interrogation data categories, it is very inconvenient to store the interrogation information, and the purpose of common storage cannot be achieved; furthermore, there is related information in the time sequence between the two, but the separate storage method leads to poor correlation between the two, and it is impossible to quickly match and retrieve valid information when calling for review, which greatly affects the subsequent judgment and review. In view of the above content, this embodiment proposes a method for linking interrogation records with audio and video recordings, such as Figure 1 As shown, Figure 1This is a flow chart of the method for linking interrogation records with audio and video recordings, which specifically includes the following steps:

[0042] S1: Obtain the interrogation transcript text data and video data; label the transcript text data and video data with category labels respectively, and correspond the transcript text data with the same category label to a certain section of video data.

[0043] S2: Perform semantic extraction on the transcript text data to obtain semantic feature information of the transcript text data, and convert the feature information into a text semantic feature vector.

[0044] S3: Perform frame extraction on the video data, extract the image features of each frame of the video data, and perform similarity screening on the image features to obtain key frame image features; obtain key frame semantic features through the semantic extraction model and convert them into key frame semantic feature vectors.

[0045] S4: Construct a cross-modal embedding space model, including two different modality encoders and a shared layer with a high-dimensional vector space.

[0046] S5: The text semantic feature vector of the category label and the key frame semantic feature vector are encoded by two encoders respectively, and then spliced and fused to obtain a fused feature vector, which is then converted and mapped to the shared layer to complete the feature association between the transcript text data and the segmented video data.

[0047] S6: Input transcript text query data or video data, and convert it into a text semantic feature vector or a keyframe semantic feature vector as an input feature vector.

[0048] S7: Based on the input feature vector, the trained cross-modal embedding space model is used to obtain the fused feature vector with the highest similarity, and the corresponding interrogation transcript text data and segmented video data are used as the results.

[0049] Among them, when cross-modal information retrieval is performed based on the interrogation transcript text data, in order to reduce the processing volume and increase the subsequent retrieval accuracy and efficiency during data processing, key frame image features are targeted. Therefore, similarity recognition and screening are specifically used to determine the key frames of the video data, that is, by performing frame extraction on the video data, extracting the image features of each frame of the video data, and performing similarity screening on the image features to obtain the key frame image features. Specifically, Figure 2 As shown, Figure 2 The specific steps for key frame image feature extraction include:

[0050] S3.1: Extract frames at specific time intervals and obtain image data of each frame.

[0051] S3.2: Use edge detection to obtain image contour features in the image data, and extract a number of image contour feature contour points to form a contour point set.

[0052] S3.3: Calculate the bidirectional Hausdorff distance between the contour point sets of each frame image data as a basis for the similarity matching degree, that is, the similarity value between each frame image data.

[0053] In this embodiment, the bidirectional Hausdorff distance is used to determine the similarity of the contour point sets of each frame image data. Specifically:

[0054] According to two sets of contour points A and B, where A={a1,a2,...,a p}, B={b1,b2,...,b q}, its bidirectional Hausdorff distance H(A,B) is:

[0055] H(A,B)=max[h(A,B),h(B,A)]

[0056] Among them, h(A,B) is the one-way Hausdorff distance from contour point set A to contour point set B, and h(B,A) is the one-way Hausdorff distance from contour point set B to contour point set A.

[0057] The calculation formula of one-way Hausdorff distance is:

[0058] h(A,B)=max(a∈A)min(b∈B)‖ab‖

[0059] h(B,A)=max(b∈B)min(a∈A)‖ba‖

[0060] Here, ‖·‖ represents some distance form between the contour point sets A and B, usually the Euclidean distance.

[0061] S3.4: According to a preset similarity threshold, ie, a Hausdorff distance threshold, frames with similarity values lower than the threshold are regarded as key frames.

[0062] Furthermore, when performing cross-modal information retrieval, that is, when retrieving video data from interrogation transcript data, it is necessary to determine the corresponding key frame semantic feature vector based on the fusion feature vector with the highest similarity, and determine the key frame position of the video data based on the time series information of the key frame semantic feature vector, and retrieve the video segment with the preset time length requirement and containing key frames.

[0063] In this embodiment, vector dot product is used to determine similarity. The dot product of two vectors a and b is defined as:

[0064] a·b=|a||b|cos(θ)

[0065] Here, |a| and |b| are the moduli (lengths) of vectors a and b, respectively, and θ is the angle between the two vectors. The dot product of vectors has two elements: directional similarity and magnitude similarity.

[0066] When the angle θ between two vectors is small, their dot product is large, indicating that the directions of the two vectors are relatively similar. In particular, when the directions of the two vectors are the same (θ=0), the dot product reaches its maximum value, i.e. |a||b|. The dot product is also affected by the vector modulus. When the moduli of both vectors are large, the dot product may be large even if their directions are not exactly the same. Therefore, the dot product takes into account both the size and direction of the vector. In this embodiment, the dot product between the fused feature vector and the input feature vector is used to judge the similarity, and the candidate objects are sorted according to the similarity; the sorted candidate objects are output to the user as retrieval results.

[0067] Furthermore, before the text semantic feature vector and the keyframe semantic feature vector are converted to a shared layer and fused, a fully connected layer output mapping is used to convert the feature vectors from different modalities into vectors with the same dimension.

[0068] Further, such as Figure 3 As shown, performing semantic extraction on the transcript text data to obtain semantic feature information of the transcript text data includes the following steps:

[0069] S2.1: Clean, segment, fill, and remove stop words from the transcript text data to obtain valid transcript text data.

[0070] S2.2: Use the BERT model to extract semantic feature information from valid transcript text data and convert it into numerical vectorized semantic feature information.

[0071] Further, such as Figure 4 As shown in FIG, the key frame semantic features are obtained through the semantic extraction model and converted into a key frame semantic feature vector, including the following steps:

[0072] S3.5: Extract keyframe semantic feature vectors through the pre-trained Transformer model.

[0073] S3.6: Use the t-SNE algorithm to reduce the dimension of the key frame semantic feature vector to keep it the same dimension as the text semantic feature vector and normalize it.

[0074] The method proposed in this embodiment, in order to achieve the purpose of synchronous association of multimodal data of interrogation, proposes to use the method of extracting and splicing multimodal semantic feature vectors to perform information association, realizes the linkage between transcripts and audio and video recordings, and can accurately locate and cross-modally retrieve when the subsequent interrogation information is called, providing a basis for subsequent rapid calling and analysis. This method adopts a cross-modal embedding space model to map the features of different modalities to the common space of the cross-modal embedding space model to obtain vector representations of the same dimension, so that the text semantic feature vectors and the key frame semantic feature vectors can be aligned and in the same dimension, thereby improving the feasibility of subsequent feature association and the accuracy of cross-modal retrieval. This method realizes cross-modal information association by associating the semantic feature vectors of transcripts and video information, providing a basis for subsequent rapid calling and analysis.

[0075] The above is a method for linking interrogation records with audio and video recordings provided in one embodiment of this embodiment. Based on the same idea, this embodiment also provides a corresponding system for linking interrogation records with audio and video recordings. For the specific definition of the system for linking interrogation records with audio and video recordings, please refer to the definition of the method for linking interrogation records with audio and video recordings above, which will not be repeated here. Each module in the above-mentioned system for linking interrogation records with audio and video recordings can be implemented in whole or in part through software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of the above-mentioned modules.

[0076] This embodiment also provides a computer-readable storage medium, which stores a computer program that can be used to execute the above Figure 1 Provides a method for linking interrogation records with audio and video recordings.

[0077] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0078] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for linking interrogation records with audio and video recordings, characterized in that: The following steps are involved: Obtaining the interrogation transcript and video data; labeling the transcript and video data with category labels, and matching transcripts with a certain segment of video data with the same category label; Performing semantic extraction on the transcript text data to obtain semantic feature information of the transcript text data, and converting the feature information into a text semantic feature vector; Perform frame extraction on the video data, extract the image features of each frame of the video data, and perform similarity screening on the image features to obtain key frame image features; obtain key frame semantic features through the semantic extraction model and convert them into key frame semantic feature vectors; Construct a cross-modal embedding space model, including two different modality encoders and a shared layer with a high-dimensional vector space; The text semantic feature vector of the category label and the key frame semantic feature vector are encoded by two encoders respectively, and then concatenated and fused to obtain a fused feature vector, which is then converted and mapped into the shared layer to complete the feature association between the transcript text data and the segmented video data; Input transcript text query data or video data, convert it into a text semantic feature vector or a keyframe semantic feature vector as an input feature vector; According to the input feature vector, the trained cross-modal embedding space model is used to obtain the fused feature vector with the highest similarity, and the corresponding interrogation transcript text data and segmented video data are used as the results.

2. The method for linking interrogation records with audio and video recordings according to claim 1, characterized in that: The method further includes determining a corresponding key frame semantic feature vector based on the fused feature vector with the highest similarity; The key frame position of the video data is determined according to the time series information of the key frame semantic feature vector, and the video segment containing the key frame with the preset time length requirement is retrieved.

3. The method for linking interrogation records with audio and video recordings according to claim 1, characterized in that: The dot product between the fused feature vector and the input feature vector is used to determine similarity, and the candidate objects are sorted according to the similarity; the sorted candidate objects are output to the user as retrieval results.

4. The method for linking interrogation records with audio and video recordings according to claim 1, characterized in that: It also includes converting the text semantic feature vector and the key frame semantic feature vector to a shared layer and before fusing them, using a fully connected layer output mapping to convert feature vectors from different modalities into vectors with the same dimension.

5. The method for linking interrogation records with audio and video recordings according to claim 1, characterized in that: The process of performing semantic extraction on the transcript text data to obtain semantic feature information of the transcript text data includes the following steps: Clean, segment, fill, and remove stop words from the transcript text data to obtain valid transcript text data; The valid transcript text data is used to extract semantic feature information through the BERT model and convert it into numerical vectorized semantic feature information.

6. The method for linking interrogation records with audio and video recordings according to claim 1, characterized in that: The process of extracting frames from the video data, extracting image features from each frame of the video data, and performing similarity screening on the image features to obtain key frame image features comprises the following steps: Extract frames at specific time intervals to obtain image data for each frame; Edge detection is used to obtain image contour features in image data, and a number of image contour feature contour points are extracted to form a contour point set; Calculate the bidirectional Hausdorff distance between the contour point sets of each frame image data as the basis for the similarity matching degree, that is, the similarity value between each frame image data; According to the preset similarity threshold, frames with similarity values lower than the threshold are regarded as key frames.

7. The method for linking interrogation records with audio and video recordings according to claim 1, characterized in that: The step of obtaining key frame semantic features through a semantic extraction model and converting the key frame semantic feature vectors includes the following steps: Extract key frame semantic feature vectors through the pre-trained Transformer model; The key frame semantic feature vector is reduced in dimension by the t-SNE algorithm to keep the same dimension as the text semantic feature vector and normalized.

8. A system for linking interrogation records with audio and video recordings, characterized in that: include: processor; a memory having stored thereon a computer program executable on the processor; When the computer program is executed by the processor, the steps of the method for linking interrogation records with audio and video recordings as described in any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a data processing program, which, when executed by a processor, implements the steps of the method for linking an interrogation record with audio and video recording according to any one of claims 1 to 7.

Images