Multi-modal data labeling system and method based on deep learning

Abstract

The invention discloses a multi-modal data labeling system and method based on deep learning, and relates to the technical field of data labeling, the multi-modal data labeling method based on deep learning specifically comprises the following steps: step 1, inputting an original video data stream, performing modal decomposition and preprocessing on the original video data to obtain multi-modal data, the method comprises a first step of collecting multi-modal data, a second step of classifying and analyzing the collected multi-modal data, and setting feature coefficients corresponding to multi-modals by extracting features of the multi-modal data, and a third step of setting a joint feature vector based on the feature coefficients corresponding to the multi-modals, and realizing real-time labeling quality monitoring according to the joint feature vector. According to the method, the image data feature, the text data feature and the audio data feature coefficient are self-defined by extracting the multi-modal data feature, the joint feature vector is set by referring to different feature coefficients, and real-time labeling quality monitoring is achieved according to the joint feature vector.

Description

Technical Field

[0001] The present invention belongs to the technical field of data annotation, and specifically relates to a multimodal data annotation system and method based on deep learning. Background Art

[0002] With the advent of the big data era, the processing and analysis of multimodal data (such as images, text, audio, etc.) has become increasingly important. Traditional data annotation methods usually rely on the data features of a single modality and are unable to fully utilize the correlation information between multimodal data, resulting in uneven annotation quality. In the prior art, although there are some multimodal data annotation methods, these methods often fail to effectively extract and utilize the deep features of multimodal data, resulting in inaccurate and incomplete annotation results.

[0003] In addition, existing multimodal data annotation methods also have certain difficulties in dealing with the fusion and alignment of different modal data. Since different modal data have differences in expression form, spatiotemporal resolution, etc., how to fully ensure the quality of multimodal data annotation by real-time monitoring of their fusion and alignment is a difficult problem in current research.

[0004] Therefore, there is an urgent need for a new multimodal data annotation method to solve the current technical problems in multimodal data annotation. Summary of the invention

[0005] The purpose of the present invention is to provide a multimodal data annotation system and method based on deep learning, which is used to solve the technical problem that it is difficult to achieve real-time monitoring of the fusion and alignment of different modal data in the prior art.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions: Multimodal data annotation methods based on deep learning include: Step 1: Input the original video data stream, perform modal decomposition and preprocessing on the original video data to obtain multimodal data; Step 2: Classify and analyze the collected multimodal data, extract the multimodal data features, and set the feature coefficients corresponding to the multimodal data; Step 3: Set a joint feature vector based on the feature coefficients corresponding to the multi-modalities, and implement real-time labeling quality monitoring based on the joint feature vector.

[0007] Furthermore, the original video data stream is input, and the original video data is modally decomposed and preprocessed to obtain multimodal data. The specific method is as follows: Input the original video data stream, and decompose the video into a timestamp-aligned image dataset according to the preset frame rate, denoted as , where m means that the video has a total of m frames after decomposition. The motion intensity is quantified to define the motion intensity threshold. According to the threshold, it is judged whether different video frames belong to the intense motion intensity. The sliding window algorithm is used to dynamically adjust the frame interval. The recording format includes the frame number, timestamp value and image data reference path. The image data composed of video frame data is obtained. The deep learning model is used to detect the potential text area in the image, and the text detected by the image is recorded as image text. Separate an independent audio track from the video data stream, save it in a lossless audio format, maintain the sampling rate f and the 16-bit depth, perform fast Fourier transform on the audio, determine the speech time that lasts more than a1 milliseconds and has an energy lower than a2 as a silent segment, and the non-silent segment as a valid audio segment, record the spectrum data of the valid speech segment and the silent segment as audio data, use an end-to-end speech recognition model to recognize the text content contained in the audio data, output the text content and the corresponding time interval, and record the output text as audio text; Image data, audio data, image text and audio text are collectively referred to as multimodal data, among which image text and audio text are collectively referred to as text data.

[0008] Furthermore, the collected multimodal data is classified and analyzed, and the characteristic coefficients corresponding to the multimodal data are set by extracting the multimodal data features. The specific method is as follows: The brightness parameters, texture parameters and edge density of the image data of each video frame are extracted through a convolutional neural network, and the image feature coefficients are obtained comprehensively; The collected text data is preprocessed, and the preprocessed text data is divided according to the number of video frames and images from which the original text information comes, and all the text information contained in each video frame is recorded as a document content, and the text feature coefficient is obtained comprehensively; The video frames separated from the video data stream are collected, and the audio feature coefficients are determined by performing short-time Fourier transform on the audio data corresponding to each video frame.

[0009] Furthermore, the brightness parameter, texture parameter and edge density of the image data of each video frame are extracted through a convolutional neural network, and the image feature coefficient is obtained comprehensively. The specific method is as follows: Using the formula represents the image feature coefficient, where x represents the xth video frame, represents the image feature coefficient of the xth video frame, d(x) represents the area size of the specific image of the xth video frame, represents the horizontal and vertical coordinates of the pixel in the image data of the xth video frame, Indicates that the constant represents the maximum pixel coordinate of the image data of the xth video frame. Indicates the image data of the xth video frame at coordinates The pixel value, A is a constant value, The image data of the xth video frame. pixels, The image data of the xth video frame. pixels, Indicates that the gray level is equal to Pixel gray level and The joint probability distribution of pixel pairs of gray levels, Indicates the number of edge pixels obtained by applying the Canny edge detection algorithm to the image data of the x-th video frame. represents the pixel value weight coefficient, represents the gray level weight coefficient, Represents the edge weight coefficient.

[0010] Furthermore, the collected text data is preprocessed, and the preprocessed text data is divided according to the number of video frames and images from which the original text information comes. All the text information contained in each video frame is recorded as a document content, and the text feature coefficient is obtained comprehensively. The specific method is as follows: Using the formula represents the text feature coefficient, where x represents the xth video frame, represents the text feature coefficient of the x-th video frame, pa(x) represents the maximum number of word occurrences in the document content corresponding to the x-th video frame, pi(x) represents the average number of word occurrences in the document content corresponding to the x-th video frame, and n(x) represents the number of dimensions of the document content corresponding to the x-th video frame. Represents the dimension factor.

[0011] Furthermore, the video frames separated from the video data stream are collected, and the audio feature coefficients are determined by performing short-time Fourier transform on the audio data corresponding to each video frame. The specific method is as follows: Using the formula represents the audio feature coefficient, where x represents the xth video frame, represents the audio feature coefficient of the xth video frame, represents the short-time Fourier transform coefficient of the audio segment data of the x-th video frame, k represents the frequency point number of the discrete Fourier transform, each k corresponds to a discrete frequency component, and N represents the total number of Fourier transform numbers. This means that only when the actual physical frequency Y is equal to There is energy contribution at the frequency point, and the energy is zero at other frequency points.

[0012] Furthermore, the joint feature vector is set based on the feature coefficients corresponding to the multi-modal states. The specific method is as follows: Using the formula represents the joint feature vector, where x represents the xth video frame, represents the joint feature vector of the x-th video frame, represents the image feature coefficient of the xth video frame, represents the text feature coefficient of the xth video frame, represents the audio feature coefficient of the xth video frame, is the text feature factor, xu(i) represents the image feature factor of the x-th video frame, and xt(j) represents the audio feature factor of the x-th video frame.

[0013] Furthermore, real-time labeling quality monitoring is achieved based on the joint feature vector. The specific method is as follows: Set the image feature coefficient similarity threshold P. If the ratio of the image feature coefficients of adjacent video frames is greater than or equal to the threshold, it proves that the pair of adjacent video frames belong to the same scene image. Set the joint feature vector threshold E, and calculate the absolute value of the difference between the joint feature vectors of all video frames belonging to the same scene image. If the absolute value is greater than or equal to E, it means that there is a difference in the dimensions of the multimodal data modalities corresponding to the pair of adjacent video frames, the spatiotemporal relationship is not aligned, and the annotation quality is abnormal. It is necessary to reset the image features, text features and audio features of the same scene image to the same time window according to the video frame timestamp. If the absolute value is less than E, it means that the annotation quality of the multimodal data of the same scene image is normal.

[0014] The present invention also provides a multimodal data annotation system based on deep learning, comprising: The multimodal data collection module inputs the original video data stream, performs modal decomposition and preprocessing on the original video data to obtain multimodal data; The multimodal feature extraction module classifies and analyzes the collected multimodal data, extracts the multimodal data features, and sets the feature coefficients corresponding to the multimodal data; The real-time annotation quality monitoring module sets a joint feature vector based on the feature coefficients corresponding to the multi-modalities, and implements real-time annotation quality monitoring according to the joint feature vector.

[0015] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. The present invention performs modal decomposition and preprocessing on the original video data, collectively refers to image data, audio data and text data extracted from the image and audio as multimodal data, and establishes an association between image text and audio text. By establishing a tracking association for stable text across multiple frames, repeated recognition of adjacent frames is avoided, which not only helps to improve the efficiency of text recognition, but also ensures the accuracy and consistency of text information; 2. The present invention uses a convolutional neural network to extract features from multimodal data according to data type. For image data, the key features of brightness parameters, texture parameters and edge density are extracted to form image feature coefficients. For text data, the text feature coefficients represented by a vector composed of word frequencies are obtained through preprocessing and the establishment of a bag-of-words model. For audio data, the audio feature coefficients are determined through short-time Fourier transform. This refined feature extraction helps to more accurately measure the feature strength and importance of image, text and audio data in each video frame. 3. The present invention sets a joint feature vector by comprehensively combining the image feature coefficients, text feature coefficients and audio feature coefficients of the same video frame, which helps to more accurately judge the content and quality of the video frame. By setting the image feature coefficient similarity threshold, it can intelligently judge whether adjacent video frames belong to the same scene image, which helps to optimize the data processing flow. By setting the joint feature vector threshold, it can judge in real time whether there are dimensional differences and misalignment of time and space relationships between multimodal data modes, and can reset the time window where the image features, text features and audio features of the same scene image are located in real time according to the video frame timestamp. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A step diagram of a multimodal data annotation method based on deep learning is shown; Figure 2 A method step diagram for implementing real-time labeling quality monitoring based on a joint feature vector is shown; Figure 3 The module diagram of the multimodal data annotation system based on deep learning is shown. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] Embodiment 1: Figure 1 , Figure 2 The multimodal data annotation method based on deep learning shown in the figure specifically includes the following steps: Step 1: Input the original video data stream, perform modal decomposition and preprocessing on the original video data to obtain multimodal data.

[0020] Input the original video data stream, and decompose the video into a continuous image sequence according to the preset frame rate, such as 30fps. If the original frame rate is less than the preset frame rate, use the interpolation algorithm, such as the optical flow method to fill in the frame. If the original frame rate is greater than the preset frame rate, use frame extraction and downsampling to further generate a timestamp-aligned image dataset, which is recorded as , where m means that there are m frames of images after the video is decomposed. The SSIM structural similarity index or the optical flow field energy value of adjacent frames is used to quantify the motion intensity. The motion intensity threshold is further defined. For example, when the optical flow amplitude is >50 pixels per frame, the frame image is judged to be of intense motion intensity. The sliding window algorithm is used to dynamically adjust the frame interval. When intense motion is detected, the sampling rate in the window is increased until it is ensured that no key action frames are missed. A millisecond-level presentation timestamp is generated for each frame of the image. The recording format includes the frame number, timestamp value and image data reference path. Image data composed of video frame data is obtained. A deep learning model is used to detect potential text areas in the image. Non-text interference is eliminated through geometric feature filtering. Multi-language recognition is performed on the detected text area. The recognition result is bound and stored with the timestamp and position coordinates of the video frame. A tracking association is established for stable text across multiple frames to avoid repeated recognition of adjacent frames. The text detected by the image is recorded as image text. Separate an independent audio track from the video data stream, save it in a lossless audio format, maintain the sampling rate f and 16-bit depth to retain voice details, perform fast Fourier transform on the audio, use the VAD algorithm to divide valid audio segments and silent segments, apply a voice endpoint detection model based on deep learning, and determine the voice time that lasts more than a1 milliseconds and has an energy lower than a2 as a silent segment. In this embodiment, f is set to 48kHz, a1=30ms, a2=-30dB, and the non-silent segment is a valid audio segment. The spectrum data of the valid voice segment and the silent segment are recorded as audio data. The end-to-end speech recognition model is used to process the text content contained in the audio data, and the text content and the corresponding time interval are output. Each text segment is marked with the start and end positions on the video timeline, and the output text is recorded as audio text; Image data, audio data, image text and audio text are collectively referred to as multimodal data, among which image text and audio text are collectively referred to as text data.

[0021] Step 2: Classify and analyze the collected multimodal data, extract the multimodal data features, and set the feature coefficients corresponding to the multimodality.

[0022] Classify multimodal data according to data type and time series, and extract features of multimodal data according to data type through a convolutional neural network; Extract the brightness parameter, texture parameter and edge density of the image data of each video frame through a convolutional neural network, and comprehensively obtain the image feature coefficient. The image data of each video frame refers to the data of each frame of video or each image in the image data. The specific formula of the image feature coefficient is as follows: ; Among them, x represents the x-th video frame, represents the image feature coefficient of the x-th video frame, d(x) represents the area size of the specific image of the x-th video frame, represents the abscissa and ordinate of the pixel in the image data of the x-th video frame, represents a constant representing the maximum value of the pixel coordinates of the image data of the x-th video frame, represents the image data of the x-th video frame at the coordinate of the pixel value, A is a constant value, represents the th pixel of the image data of the x-th video frame, represents the th pixel of the image data of the x-th video frame, represents the gray level equal to the joint probability distribution of the pixel pairs of the pixel gray level and the pixel gray level, represents the number of edge pixels obtained by applying the Canny edge detection algorithm to the image data of the x-th video frame, represents the pixel value weight coefficient, represents the gray level weight coefficient, represents the edge weight coefficient; Preprocess the collected text data, remove the stop words, numbers and symbols in it. Stop words refer to words that appear frequently in the text but have no actual meaning. For example: "de", "shi". Divide the preprocessed text data according to the video frames and the number of images in the original text information source. Denote all the text information contained in each video frame as a document content. Establish a bag-of-words model according to different document contents, represent each document as a vector composed of word frequencies, and represent the text data containing n words with an n-dimensional vector, where each dimension represents a word, and its value is the number of times the word appears in the document. The specific formula of the text feature coefficient is as follows: ; Among them, x represents the x-th video frame, represents the text feature coefficient of the x-th video frame, pa(x) represents the maximum number of word occurrences in the document content corresponding to the x-th video frame, pi(x) represents the average number of word occurrences in the document content corresponding to the x-th video frame, and n(x) represents the number of dimensions of the document content corresponding to the x-th video frame. Indicates the dimension factor. In this embodiment, Equal to -0.2.

[0023] The video frames separated from the video data stream are collected, and the audio feature coefficients are determined by performing short-time Fourier transform on the audio data corresponding to each video frame. The specific formula of the audio feature coefficients is as follows: ; Where x represents the xth video frame, represents the audio feature coefficient of the xth video frame, represents the short-time Fourier transform coefficient of the audio segment data of the x-th video frame, k represents the frequency point number of the discrete Fourier transform, each k corresponds to a discrete frequency component, and N represents the total number of Fourier transform numbers. This means that only when the actual physical frequency Y is equal to There is energy contribution at the frequency point, and the energy is zero at other frequency points.

[0024] Step 3: Set a joint feature vector based on the feature coefficients corresponding to the multi-modalities, and implement real-time labeling quality monitoring based on the joint feature vector.

[0025] The image feature coefficient, text feature coefficient and audio feature coefficient of the same video frame are combined to set the joint feature vector. The specific formula is as follows: ; Where x represents the xth video frame, represents the joint feature vector of the x-th video frame, represents the image feature coefficient of the xth video frame, represents the text feature coefficient of the xth video frame, represents the audio feature coefficient of the xth video frame, is the text feature factor, xu(i) represents the image feature factor of the x-th video frame, i is equal to 0 or 1, when i is equal to 1, it means that the frame image belongs to the image of intense motion intensity, when i is equal to 0, it means that the frame image does not belong to the image of intense motion intensity, xt(j) represents the audio feature factor of the x-th video frame, j is equal to 0 or 1, when j is equal to 1, it means that the audio data corresponding to the frame image is in the silent segment, when j is equal to 0, it means that the audio data corresponding to the frame image is in the valid audio segment, in this embodiment, it is set Equal to 0.3.

[0026] Set the image feature coefficient similarity threshold P. If the ratio of the image feature coefficients of adjacent video frames is greater than or equal to the threshold, it proves that the pair of adjacent video frames belong to the same scene image. Set the joint feature vector threshold E, and calculate the absolute value of the difference between the joint feature vectors of all video frames belonging to the same scene image. If the absolute value is greater than or equal to E, it means that there is a difference in the dimensions of the multimodal data modalities corresponding to the pair of adjacent video frames, the spatiotemporal relationship is not aligned, and the annotation quality is abnormal. It is necessary to reset the image features, text features and audio features of the same scene image to the same time window according to the video frame timestamp. If the absolute value is less than E, it means that the annotation quality of the multimodal data of the same scene image is normal.

[0027] Embodiment 2, as Figure 3 The multimodal data annotation system based on deep learning shown in the figure specifically includes: The multimodal data collection module inputs the original video data stream and decomposes the video into a continuous image sequence at a preset frame rate, such as 30fps. If the original frame rate is less than the preset frame rate, an interpolation algorithm is used, such as the optical flow method. If the original frame rate is greater than the preset frame rate, frame sampling is used to further generate a timestamp-aligned image data set, denoted as, where m represents a total of m frames of images after the video is decomposed. The motion intensity is quantified by the SSIM structural similarity index of adjacent frames or the optical flow field energy value, and the motion intensity threshold is further defined. For example, when the optical flow amplitude is greater than 50 pixels per frame, the frame image is judged to be of intense motion intensity, and a sliding window algorithm is used to dynamically adjust the image data set. Integer frame intervals, when violent motion is detected, the sampling rate in the window is increased until it is ensured that no key action frames are missed, and a millisecond-level precision presentation timestamp is generated for each frame of the image. The recording format includes the frame number, timestamp value, and image data reference path, and image data composed of video frame data is obtained. The deep learning model is used to detect potential text areas in the image, and non-text interference is eliminated through geometric feature filtering. Multi-language recognition is performed on the detected text area, and the recognition results are bound and stored with the timestamp and position coordinates of the video frame. Tracking association is established for stable text across multiple frames to avoid repeated recognition of adjacent frames, and the text detected by the image is recorded as image text; Separate an independent audio track from the video data stream, save it in a lossless audio format, maintain the sampling rate f and 16-bit depth to retain voice details, perform fast Fourier transform on the audio, use the VAD algorithm to divide valid audio segments and silent segments, apply a voice endpoint detection model based on deep learning, and determine the voice time that lasts more than a1 milliseconds and has an energy lower than a2 as a silent segment. In this embodiment, f is set to 48kHz, a1=30ms, a2=-30dB, and the non-silent segment is a valid audio segment. The spectrum data of the valid voice segment and the silent segment are recorded as audio data. The end-to-end speech recognition model is used to process the text content contained in the audio data, and the text content and the corresponding time interval are output. Each text segment is marked with the start and end positions on the video timeline, and the output text is recorded as audio text; Image data, audio data, image text and audio text are collectively referred to as multimodal data, among which image text and audio text are collectively referred to as text data.

[0028] The multimodal feature extraction module classifies the multimodal data according to data type and time series, and extracts features from the multimodal data according to data type through a convolutional neural network; The brightness parameters, texture parameters and edge density of the image data of each video frame are extracted through the convolutional neural network, and the image feature coefficient is obtained comprehensively. The image data of each video frame refers to the data of each frame of video or each image in the image data. The specific formula of the image feature coefficient is as follows: ; Where x represents the xth video frame, represents the image feature coefficient of the xth video frame, d(x) represents the area size of the specific image of the xth video frame, represents the horizontal and vertical coordinates of the pixel in the image data of the xth video frame, Indicates that the constant represents the maximum pixel coordinate of the image data of the xth video frame. Indicates the image data of the xth video frame at coordinates The pixel value, A is a constant value, The image data of the xth video frame. pixels, The image data of the xth video frame. pixels, Indicates that the gray level is equal to Pixel gray level and The joint probability distribution of pixel pairs with respect to pixel gray levels, Indicates the number of edge pixels obtained by applying the Canny edge detection algorithm to the image data of the x-th video frame. represents the pixel value weight coefficient, represents the grayscale weight coefficient, represents the edge weight coefficient; Preprocess the collected text data to remove stop words, numbers, and symbols. Stop words refer to words that frequently appear in the text but have no actual meaning, such as "de" (of), "shi" (is). Divide the preprocessed text data according to the video frames and the number of image sheets in the original text information source. Denote all the text information contained in each video frame as a document content. Establish a bag-of-words model based on different document contents, represent each document as a vector composed of word frequencies, and represent the text data containing n words with an n-dimensional vector, where each dimension represents a word and its value is the number of times the word appears in the document. The specific formula for the text feature coefficient is as follows: ; where x represents the xth video frame, represents the text feature coefficient of the xth video frame, pa(x) represents the maximum number of times a word appears in the document content corresponding to the xth video frame, pi(x) represents the average number of times a word appears in the document content corresponding to the xth video frame, and n(x) represents the number of dimensions of the document content corresponding to the xth video frame. represents the dimension factor. In this embodiment, it is set to be equal to -0.2.

[0029] Collect the video frames separated from the video data stream. Determine the audio feature coefficient by performing a short-time Fourier transform on the audio data corresponding to each video frame. The specific formula for the audio feature coefficient is as follows: ; where x represents the xth video frame, represents the audio feature coefficient of the xth video frame, represents the short-time Fourier transform coefficient of the audio segment data of the xth video frame, k represents the frequency point serial number of the discrete Fourier transform, each k corresponds to a discrete frequency component, and N represents the total number of serial numbers of the Fourier transform. represents that there is only energy contribution at the actual physical frequency Y equal to and the energy is zero at other frequency points.

[0030] The real-time annotation quality monitoring module sets a joint feature vector by integrating the image feature coefficient, text feature coefficient, and audio feature coefficient of the same video frame. The specific formula is as follows: ; where x represents the xth video frame, represents the joint feature vector of the xth video frame, represents the image feature coefficient of the xth video frame, represents the text feature coefficient of the xth video frame, represents the audio feature coefficient of the xth video frame, is the text feature factor, xu(i) represents the image feature factor of the x-th video frame, i is equal to 0 or 1, when i is equal to 1, it means that the frame image belongs to the image of intense motion intensity, when i is equal to 0, it means that the frame image does not belong to the image of intense motion intensity, xt(j) represents the audio feature factor of the x-th video frame, j is equal to 0 or 1, when j is equal to 1, it means that the audio data corresponding to the frame image is in the silent segment, when j is equal to 0, it means that the audio data corresponding to the frame image is in the valid audio segment, in this embodiment, it is set Equal to 0.3.

[0031] Set the image feature coefficient similarity threshold P. If the ratio of the image feature coefficients of adjacent video frames is greater than or equal to the threshold, it proves that the pair of adjacent video frames belong to the same scene image. Set the joint feature vector threshold E, and calculate the absolute value of the difference between the joint feature vectors of all video frames belonging to the same scene image. If the absolute value is greater than or equal to E, it means that there is a difference in the dimensions of the multimodal data modalities corresponding to the pair of adjacent video frames, the spatiotemporal relationship is not aligned, and the annotation quality is abnormal. It is necessary to reset the image features, text features and audio features of the same scene image to the same time window according to the video frame timestamp. If the absolute value is less than E, it means that the annotation quality of the multimodal data of the same scene image is normal.

[0032] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

[0033] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A multimodal data annotation method based on deep learning, characterized in that: include: Step 1: Input the original video data stream, perform modal decomposition and preprocessing on the original video data to obtain multimodal data; Step 2: Classify and analyze the collected multimodal data, extract the multimodal data features, and set the feature coefficients corresponding to the multimodal data; Step 3: Set a joint feature vector based on the feature coefficients corresponding to the multi-modalities, and implement real-time labeling quality monitoring based on the joint feature vector.

2. The multimodal data annotation method based on deep learning according to claim 1, characterized in that: Input the original video data stream, perform modal decomposition and preprocessing on the original video data to obtain multimodal data. The specific method is as follows: Input the original video data stream, and decompose the video into a timestamp-aligned image dataset according to the preset frame rate, denoted as , where m means that the video has a total of m frames after decomposition. The motion intensity is quantified to define the motion intensity threshold. According to the threshold, it is judged whether different video frames belong to the intense motion intensity. The sliding window algorithm is used to dynamically adjust the frame interval. The recording format includes the frame number, timestamp value and image data reference path. The image data composed of video frame data is obtained. The deep learning model is used to detect the potential text area in the image, and the text detected by the image is recorded as image text. Separate an independent audio track from the video data stream, save it in a lossless audio format, maintain the sampling rate f and the 16-bit depth, perform fast Fourier transform on the audio, determine the speech time that lasts more than a1 milliseconds and has an energy lower than a2 as a silent segment, and the non-silent segment as a valid audio segment, record the spectrum data of the valid speech segment and the silent segment as audio data, use an end-to-end speech recognition model to recognize the text content contained in the audio data, output the text content and the corresponding time interval, and record the output text as audio text; Image data, audio data, image text and audio text are collectively referred to as multimodal data, among which image text and audio text are collectively referred to as text data.

3. The multimodal data annotation method based on deep learning according to claim 1, characterized in that: Classify and analyze the collected multimodal data, extract the multimodal data features, and set the feature coefficients corresponding to the multimodal data. The specific method is as follows: The brightness parameter, texture parameter and edge density of the image data of each video frame are extracted through a convolutional neural network, and the image feature coefficients are obtained comprehensively; The collected text data is preprocessed, and the preprocessed text data is divided according to the number of video frames and images from which the original text information comes, and all the text information contained in each video frame is recorded as a document content, and the text feature coefficient is obtained comprehensively; The video frames separated from the video data stream are collected, and the audio feature coefficients are determined by performing short-time Fourier transform on the audio data corresponding to each video frame.

4. The multimodal data annotation method based on deep learning according to claim 1, characterized in that: The brightness parameters, texture parameters and edge density of the image data of each video frame are extracted through the convolutional neural network, and the image feature coefficients are obtained comprehensively. The specific method is as follows: Using the formula represents the image feature coefficient, where x represents the xth video frame, represents the image feature coefficient of the xth video frame, d(x) represents the area size of the specific image of the xth video frame, represents the horizontal and vertical coordinates of the pixel in the image data of the xth video frame, Indicates that the constant represents the maximum pixel coordinate of the image data of the xth video frame. Indicates the image data of the xth video frame at coordinates The pixel value, A is a constant value, The image data of the xth video frame. pixels, The image data of the xth video frame. pixels, Indicates that the gray level is equal to Pixel gray level and The joint probability distribution of pixel pairs with respect to pixel gray levels, Indicates the number of edge pixels obtained by applying the Canny edge detection algorithm to the image data of the x-th video frame. represents the pixel value weight coefficient, represents the gray level weight coefficient, Represents the edge weight coefficient.

5. The multimodal data annotation method based on deep learning according to claim 1, characterized in that: The collected text data is preprocessed, and the preprocessed text data is divided according to the number of video frames and images from which the original text information comes. All the text information contained in each video frame is recorded as a document content, and the text feature coefficient is obtained comprehensively. The specific method is as follows: Using the formula represents the text feature coefficient, where x represents the xth video frame, represents the text feature coefficient of the x-th video frame, pa(x) represents the maximum number of word occurrences in the document content corresponding to the x-th video frame, pi(x) represents the average number of word occurrences in the document content corresponding to the x-th video frame, and n(x) represents the number of dimensions of the document content corresponding to the x-th video frame. Represents the dimension factor.

6. The multimodal data annotation method based on deep learning according to claim 1, characterized in that: Collect the video frames separated from the video data stream, and determine the audio feature coefficients by performing short-time Fourier transform on the audio data corresponding to each video frame. The specific method is as follows: Using the formula represents the audio feature coefficient, where x represents the xth video frame, represents the audio feature coefficient of the xth video frame, represents the short-time Fourier transform coefficient of the audio segment data of the x-th video frame, k represents the frequency point number of the discrete Fourier transform, each k corresponds to a discrete frequency component, and N represents the total number of Fourier transform numbers. This means that only when the actual physical frequency Y is equal to There is energy contribution at the frequency point, and the energy is zero at other frequency points.

7. The multimodal data annotation method based on deep learning according to claim 1, characterized in that: The joint feature vector is set based on the feature coefficients corresponding to the multi-modalities. The specific method is: Using the formula represents the joint feature vector, where x represents the xth video frame, represents the joint feature vector of the x-th video frame, Represents the image feature coefficient of the xth video frame, represents the text feature coefficient of the xth video frame, represents the audio feature coefficient of the xth video frame, is the text feature factor, xu(i) represents the image feature factor of the x-th video frame, and xt(j) represents the audio feature factor of the x-th video frame.

8. The multimodal data annotation method based on deep learning according to claim 1, characterized in that: Real-time labeling quality monitoring is achieved based on the joint feature vector. The specific method is as follows: Set the image feature coefficient similarity threshold P. If the ratio of the image feature coefficients of adjacent video frames is greater than or equal to the threshold, it proves that the pair of adjacent video frames belong to the same scene image. Set the joint feature vector threshold E, and calculate the absolute value of the difference between the joint feature vectors of all video frames belonging to the same scene image. If the absolute value is greater than or equal to E, it means that there is a difference in the dimensions of the multimodal data modalities corresponding to the pair of adjacent video frames, the spatiotemporal relationship is not aligned, and the annotation quality is abnormal. It is necessary to reset the image features, text features and audio features of the same scene image to the same time window according to the video frame timestamp. If the absolute value is less than E, it means that the annotation quality of the multimodal data of the same scene image is normal.

9. A multimodal data annotation system based on deep learning, applied to the multimodal data annotation method based on deep learning according to any one of claims 1 to 8, characterized in that: include: The multimodal data collection module inputs the original video data stream, performs modal decomposition and preprocessing on the original video data to obtain multimodal data; The multimodal feature extraction module classifies and analyzes the collected multimodal data, extracts the multimodal data features, and sets the feature coefficients corresponding to the multimodal data; The real-time annotation quality monitoring module sets a joint feature vector based on the feature coefficients corresponding to the multi-modalities, and implements real-time annotation quality monitoring according to the joint feature vector.

Images