Video semantic retrieval method and device based on deep learning, equipment and medium

Abstract

The invention discloses a video semantic retrieval method and device based on deep learning, equipment and a medium, and generally relates to the technical field of computers. The method comprises the following steps: receiving a video file, decomposing the video file through a target model, and determining a target frame and a target audio in the video file; analyzing the target frame and the target audio into multi-dimensional vector information; converting the multi-dimensional vector information in the video file into single-dimensional vector information through a target model; the single-dimension vector information is stored in a target database; and performing comparative analysis on preset vector information in the target database and the single-dimensional vector information, and determining that the video file is a target file under the condition that the preset vector information is matched with the single-dimensional vector information. And the target file detection efficiency can be greatly improved while the accuracy of determining the target file is ensured.

Description

Technical Field

[0001] The present disclosure generally relates to the field of computer technology, and more particularly to a method, apparatus, device and medium for video semantic retrieval based on deep learning. Background Art

[0002] With the continuous development of Internet technology, more and more information appears. It is very important for users to quickly analyze and filter out useful information.

[0003] Among the information types on the Internet, video information occupies a large part. Video information often has a certain length and relatively rich information data. In the related art, when there is video information that does not meet the requirements, the text in the title information of the video or the image information in the video is analyzed to filter the video corresponding to the title information or the image information, and the video that does not meet the content requirements is filtered out. In fact, the content in the image cannot be judged.

[0004] However, since the information in the video is actually very rich, this screening method is actually rough and often cannot accurately identify the information that really needs to be excluded. Summary of the invention

[0005] In view of the above-mentioned defects or deficiencies in the related art, it is desired to provide a video semantic retrieval method, apparatus, device and medium based on deep learning, which can solve the problem that the information in the existing videos is essentially very rich, and the method of screening the video clips that users are concerned about in the related art is too rough and often cannot accurately identify the information that really needs to be excluded.

[0006] In a first aspect, a video semantic retrieval method based on deep learning is provided, the method comprising: Receive a video file and decompose the video file through a target model to determine a target frame and a target audio in the video file, wherein the target frame includes a video frame for displaying key information in the video file, and the target audio includes audio data for displaying key information in the video file; Parsing the target frame and the target audio into multi-dimensional vector information, wherein the multi-dimensional vector information includes information content of different dimensions and the position of the information content in the video file; Converting the multi-dimensional vector information in the video file into single-dimensional vector information through a target model; Storing the single-dimensional vector information in a target database; By comparing and analyzing the preset vector information in the target database with the single-dimensional vector information, and when the preset vector information matches the single-dimensional vector information, determining that the video file is the target file In the present application, after receiving a video file and decomposing the video file through a target model, the target frame and target audio in the video file are determined, and then the target frame and target audio are parsed into multi-dimensional vector information (the multi-dimensional vector information includes information content of different dimensions and the position of these information contents in the video file); then, the multi-dimensional vector information in the video file is converted into single-dimensional vector information through the target model; then, the above-mentioned single-dimensional vector information is stored in the target database; the preset vector information in the target database is compared and analyzed with the single-dimensional vector information, and when the preset vector information matches the single-dimensional vector information, the above-mentioned video file is determined to be a target file. In this way, by screening out some frames in the video file as target frames, the range of data that needs to be compared is narrowed, and at the same time, by detailed analysis of the screened target frames, multi-dimensional vector information is formed, so that the multi-dimensional vector information is analyzed to form a single-dimensional vector of the analysis, and the single-dimensional information is compared and analyzed to finally determine whether it is a target file. Since the single-dimensional information is obtained by mutual analysis and verification of the multi-dimensional vector information, and the target frame also greatly reduces the data volume of the original video file, thereby ensuring the accuracy of determining the target file while greatly improving the efficiency of detecting the target file.

[0007] In a second aspect, a video semantic retrieval device based on deep learning is provided, the device comprising: A receiving module, used for receiving a video file and decomposing the video file through a target model to determine a target frame in the video file and a video segment corresponding to the target frame; An execution module, configured to parse the target frame into multi-dimensional vector information, wherein the multi-dimensional vector information includes information content and a position of the information content in a video file; The execution module is further used to convert the multi-dimensional vector information in the video file into single-dimensional vector information through a target model; A storage module, used for storing the single-dimensional vector information in a target database; The determination module is used to compare and analyze the preset vector information in the target database with the single-dimensional vector information, and determine that the video file is a target file when the preset vector information matches the single-dimensional vector information.

[0008] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the first aspect is implemented.

[0009] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the program is executed by a processor, the method described in the first aspect is implemented.

[0010] In a fifth aspect, a computer program product is provided. The computer program product includes instructions, and the instructions are executed by a processor to implement the method described in the first aspect.

[0011] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings: Figure 1 A flowchart of a video semantic retrieval method based on deep learning provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a video semantic retrieval device based on deep learning provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0013] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the invention are shown in the accompanying drawings.

[0014] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0015] With the development of Internet technology, there is a lot of video information in Internet information. Especially with the development of short video and communication technology, users can watch short videos anytime and anywhere on terminals (such as handheld terminals). Therefore, it is very important to automatically identify and screen videos on the Internet that do not meet the requirements (for example, do not meet the platform management requirements or data governance and supervision requirements).

[0016] In the related art, the technical solution for screening videos that do not meet the requirements on the Internet is generally: by analyzing the titles or text on the images, the videos are screened and the videos that do not meet the content requirements are filtered out.

[0017] However, this method actually cannot judge the content of the image. At the same time, due to the lack of logical description in the image, it is also impossible to filter according to the actual content. If you need to accurately filter out videos that do not meet the requirements, you need to quickly locate the location of specific clips in the video. Obviously, the relevant technology cannot meet this requirement.

[0018] Based on this, the present application proposes a video semantic retrieval method, apparatus, device and medium based on deep learning, which can solve the problem that the method of screening video clips that users are concerned about in related technologies is too rough and often cannot accurately identify the information that really needs to be excluded.

[0019] Figure 3 FIG. 1 is a flow chart of a video semantic retrieval method based on deep learning provided by an embodiment of the present application. Figure 1 As shown, the method includes the following steps 301 to 305: Step 301: Receive a video file and decompose the video file through a target model to determine a target frame and a target audio in the video file.

[0020] In an embodiment of the present application, the above-mentioned video file may be a video file on the Internet.

[0021] It can be understood that in the present application, for the above-mentioned video files, the source of the video files can be all video files of a certain Internet platform, or part of the video files of a certain Internet platform, for example, it can be all video files of some modules in a certain Internet platform.

[0022] It is understandable that, in the case where the video files are part of the video files of the Internet platform, the part of the video files may be video files captured periodically on the Internet platform, or may be video files captured randomly, that is, video files obtained by extracting and checking video files. The above extraction method is not limited in this application, and may be preset or user-defined.

[0023] It should be noted that the above-mentioned video files are videos that have not been analyzed by the target model, that is, they are newly added video files.

[0024] In an embodiment of the present application, the target model may be a deep learning model for analyzing semantics.

[0025] Exemplarily, the target model may be an artificial intelligence semantic analysis model.

[0026] In an embodiment of the present application, the target frame is a video frame that can be used to display key information in a video file.

[0027] In an embodiment of the present application, the target audio includes audio data for displaying key information in the video file.

[0028] It is understandable that a video file is composed of multiple video frames, each of which includes some static image information. The video file is formed by quickly playing the video frames. For example, in a 1-minute video, 24 video frames are played per second to form a 1-minute video. From the perspective of video frames constituting a video, different video frames are used to show different content in the video. The above-mentioned target frame can be a frame used to display key information in the video, for example, the target frame includes complete image information; and other frames are used to display transition information between different target frames, that is, other frames may not contain complete image information.

[0029] Exemplarily, the key information mentioned above includes complete image information, and the content required by the user can be read from the key information.

[0030] In an embodiment of the present application, the above-mentioned target frames include some video frames in a video file. By acquiring the above-mentioned target frames, the content of some video frames used to display key information is determined. In the subsequent process of determining the specific content expressed by the video file, only these target frames need to be analyzed and confirmed, and there is no need to analyze the entire video file, thereby greatly reducing the amount of data that needs to be analyzed.

[0031] In an embodiment of the present application, the target audio may be the audio of a video clip corresponding to a target frame, or may be the audio of a video file, which is not limited in the embodiment of the present application.

[0032] Exemplarily, the target audio is used for subsequent use in extracting data information of the audio dimension.

[0033] Step 302: Parse the target frame and the target audio into multi-dimensional vector information.

[0034] In an embodiment of the present application, the multi-dimensional vector information includes information content of different dimensions and the position of the information content in the video file.

[0035] It can be understood that, for videos, there are many ways to display content. Video files can include image content, audio content and text content. That is, during the video playback process, it displays content to users in three ways at the same time, which belongs to multi-dimensional content. In the related art, if the video file is analyzed directly, the amount of data is large, and it is necessary to analyze the above-mentioned image content, audio content and text content at the same time. Therefore, in an embodiment of the present application, according to the aforementioned target frame, the data information corresponding to the target frame, such as image information and text information, and the data information of the video clip corresponding to the target frame, such as audio information, are parsed to extract data information of different dimensions. Each dimension only includes data information of a single form of expression under the dimension, thereby greatly reducing the complexity of the data that needs to be analyzed in the process of analyzing the video clips within a certain time period in the video file. There is no need to synchronously analyze the comprehensive data information of the video, decompose it into multi-dimensional vector information, and analyze it one by one. While reducing the complexity of the analysis, it can also reduce the amount of calculation, greatly improving the accuracy and efficiency of the analysis.

[0036] In an embodiment of the present application, the above-mentioned multi-dimensional vector information includes the information content under each dimension and the position of the information content in the video file, so as to facilitate the positioning of the video content and facilitate the alignment of the data information of each dimension in the subsequent process of comparing and converting the multi-dimensional vector information.

[0037] Step 303: convert the multi-dimensional vector information in the above video file into single-dimensional vector information through the target model.

[0038] It is understandable that for the process of analyzing video semantics, in order to confirm the exact meaning represented by the multi-dimensional vector information in the video file, it is necessary to finally convert the different dimensions, that is, the multi-dimensional vector information, into vector information of the same dimension, so as to improve the accuracy of determining the video semantics.

[0039] Furthermore, the above-mentioned conversion of the multi-dimensional vector information in the video file into single-dimensional vector information is actually a process of aligning data information of different dimensions in the same time period, verifying the semantic information between each other, and expressing the final semantic information in the form of one of the dimensions. At the same time, if the data information of different dimensions cannot be aligned in the same time period, it is also necessary to verify whether their data information matches the preset vector information in the subsequent process to ensure comprehensive recognition.

[0040] In one example, the single-dimensional vector information may be text-dimensional data information.

[0041] It is understandable that text dimension data information is more intuitive and the data volume is smaller than that of data information in other dimensions, which is convenient for transmission and generation, and is conducive to improving the efficiency of determining video semantics.

[0042] Step 304: Store the above single-dimensional vector information into the target database.

[0043] In the embodiment of the present application, the above-mentioned target database can be preset or can be customized by the user.

[0044] It can be understood that after the target model analyzes the above-mentioned video file, generates multi-dimensional vector information and converts it into single-dimensional vector information, the semantic analysis process of the target model on the video file is completed. In order to minimize the amount of data in the current and subsequent video semantic information analysis process of the target model, the single-dimensional vector information can be stored in the target database, and the subsequent information matching and confirmation process can be performed in the target database, thereby improving the working efficiency of the target model.

[0045] Step 305: Compare and analyze the preset vector information in the target database with the single-dimensional vector information, and determine that the video file is a target file when the preset vector information matches the single-dimensional vector information.

[0046] In the embodiment of the present application, the above-mentioned preset vector information can be preset or custom-set, and the embodiment of the present application does not limit this.

[0047] In the embodiment of the present application, the preset vector information can be used to indicate the information content that is focused on. For example, the preset vector information can be highly sensitive vector information, and the video file corresponding to the single-dimensional vector information matching the preset vector information is the video file that the user needs to focus on, that is, the target file.

[0048] In an embodiment of the present application, the above-mentioned preset vector information can be expressed in the same form as the single-dimensional vector information. For example, if the single-dimensional vector information is data information of text dimension, then the preset vector information is also data information of text dimension. The above-mentioned preset vector information can also be associated information corresponding to the single-dimensional vector information. For example, if the single-dimensional vector information is vector information of text dimension, then the preset vector information is also vector information of image dimension corresponding to the single-dimensional vector, wherein the preset vector information is pre-stored vector information of image dimension for indicating the information content in the single-dimensional vector information.

[0049] In the embodiment of the present application, the above comparison method can be a cosine similarity comparison method.

[0050] The following is a description of the cosine similarity comparison method: The cosine similarity comparison method can measure the similarity between vectors.

[0051] The formula for calculating cosine similarity is: cosine_similarity(A, B) = (A dot B) / (||A|| * ||B||).

[0052] Where: A and B are two vectors to be compared, dot represents the dot product of the vectors, also known as the inner product, which is the sum of the products of the corresponding elements of the two vectors, ||A|| and ||B|| represent the modulus (or length) of the vector, which can be obtained by calculating the sum of the squares of each vector element and then taking the square k root. The value range of cosine similarity is between -1 and 1: when cosine similarity is equal to 1, it means that the two vectors have exactly the same direction in the multidimensional space, that is, they are completely similar, when cosine similarity is equal to 0, it means that there is no linear relationship between the two vectors, they are unrelated, and when cosine similarity is equal to -1, it means that the two vectors have exactly opposite directions in the multidimensional space, that is, they are completely different.

[0053] Characteristics of cosine similarity: 1. Not affected by vector length: Cosine similarity is not affected by the absolute size of the vector, only by the direction, so it is applicable to vector comparisons of any dimension; 2. Ignoring zero elements: Cosine similarity can effectively handle sparse vectors because it ignores zero elements in both vectors; 3. Bounded range: The range of cosine similarity is always between -1 and 1, which is easy to interpret and compare.

[0054] For example, the video type is divided into several vectors according to the video content; comedy: [1, 0, 0, 0, 0], action: [0,1, 0, 0, 0], romance: [0, 0, 1, 0, 0], science fiction: [0, 0, 0, 1, 0], horror: [0, 0, 0, 0, 1]. Suppose a user likes comedy and science fiction. The user's preference can be represented as a vector obtained by weighted average of comedy and science fiction vectors: user preference: [0.5, 0, 0, 0.5, 0]. If you want to recommend a movie for this user, you can calculate the cosine similarity between the user preference vector and the movie type vector. The larger the value, the more worthy of recommendation. Similarly, the monitoring video classification can be changed to public opinion monitoring.

[0055] In the embodiment of the present application, the above-mentioned A and B may be preset vector information and the single-dimensional vector information respectively.

[0056] In the method provided in the embodiment of the present application, after receiving a video file and decomposing the video file through a target model, a target frame and a target audio in the video file are determined, and then the target frame and the target audio are parsed into multi-dimensional vector information (the multi-dimensional vector information includes information content of different dimensions and the position of these information contents in the video file); then, the multi-dimensional vector information in the video file is converted into single-dimensional vector information through the target model; then, the single-dimensional vector information is stored in a target database; the preset vector information in the target database is compared and analyzed with the single-dimensional vector information, and when the preset vector information matches the single-dimensional vector information, the video file is determined to be the target file. In this way, by screening out some frames in the video file as target frames, the scope of data that needs to be compared is narrowed down, and at the same time, by detailed analysis of the screened target frames, multi-dimensional vector information is formed, and then the multi-dimensional vector information is analyzed to form a single-dimensional vector for analysis, and the single-dimensional information is compared and analyzed to finally determine whether it is the target file. Since the single-dimensional information is obtained by mutual analysis and verification of multi-dimensional vector information, the target frame also greatly reduces the data volume of the original video file. Therefore, while ensuring the accuracy of determining the target file, the efficiency of detecting the target file can also be greatly improved.

[0057] In another embodiment of the present application, a specific implementation method for determining the target frame is also provided. Exemplarily, the specific implementation of "determining the target frame in the above video file" mentioned above includes: classifying all video frames in the above video file through the preset video frame classification method of the above target model; confirming the target frame and the target audio in the above video file.

[0058] It is understandable that the preset video frame classification method of the above target model can be preset or customized by the user.

[0059] Exemplarily, the above-mentioned video frame classification method may include: generating a key frame corresponding to the video frame file by using a target model.

[0060] It is understandable that, for video files, they may include a variety of multimedia formats, including MP4, MKV, AVI, MP3, OGG, etc., and the video data of each multimedia format can be parsed and classified through the four steps of input, decoding, extraction, and output, and converted into a unified representation format, and then the process of extracting the video content and extracting the target frame is performed. After reading and inputting the above video file, it is necessary to first parse its format and encoding information, and then extract the target frame.

[0061] For video files, there may be multiplexing formats (Container Format) and encoding formats (Codec). The above process of parsing format and encoding information can be performed by encoding and decoding using different codecs.

[0062] It can be understood that from the above content, a video file is composed of continuous frames. According to the preset video encoding method, frames can be divided into the following types: I frame (also known as target frame, key frame): a full frame, containing complete image information, and other frames can be decoded based on it. The I frame is independent of other frames and is the benchmark of the video sequence. P frame (forward prediction frame): depends on the previous I frame or P frame, and only has difference information relative to the reference frame. P frame can reduce the amount of data, but cannot be decoded independently. B frame (bidirectional prediction frame): depends on the previous and subsequent frames, can be predicted forward and backward, so it usually has the highest compression rate, but the delay is also higher.

[0063] In one example, the above process of decoding the video frames in the video file can be implemented by the filters corresponding to the target model. In general, a powerful filter system can be used to process various effects of video and audio, such as cropping, cutting, rotating, scaling, etc. Filters can also implement stream processing, that is, merging multiple audio / video streams into one file, or separating one file into multiple audio / video.

[0064] In the above process of confirming and extracting key frames, it can be known from the above content that the I frame in the video file can be identified and extracted as an independent image file, that is, it belongs to the picture dimension vector information.

[0065] Furthermore, the above-mentioned I frame can be confirmed by specifying a time interval or a number of interval frames.

[0066] Exemplarily, the target audio may be the audio of a video clip corresponding to the target frame, or may be the audio of a video file, which is not limited in this embodiment of the present application.

[0067] In one example, when the target audio is the audio of a video clip corresponding to a target frame, after the target frame is determined, the video clips before and after the target frame can be obtained based on the target frame, and the audio of the video clip can be determined as the target audio.

[0068] In an example, when the target audio is the audio of a video file, the audio data is extracted from the video file to obtain the target audio.

[0069] Exemplarily, the above-mentioned methods of obtaining the target audio may be a method of extracting audio data from a video file through a decoder and encoding the audio data into audio dimension vector information in a specified format.

[0070] Exemplarily, after obtaining the image dimension vector information and audio temperature vector information, you can output the file and save the extracted key frames and audio data as a file of a specified format, thereby obtaining the image dimension vector information of the target frame and the audio dimension vector information of the target audio.

[0071] In another embodiment of the present application, a specific implementation method for converting the image dimension vector information into single-dimensional data information is also provided. Exemplarily, in the case where the multi-dimensional vector information includes image dimension vector information, the "converting the multi-dimensional vector information in the above video file into single-dimensional vector information through the target model" mentioned above includes: identifying the text area in the image dimension vector information in the above video file through the target model; segmenting the above text area to generate multiple text sub-areas, each text sub-area including a character; identifying the text content of each of the above multiple text sub-areas; combining the above text content according to the corresponding arrangement of the multiple text sub-areas to generate the first text vector information.

[0072] Exemplarily, the first text vector information is single-dimensional vector information including text dimension vector information.

[0073] It is understandable that after analyzing the above video file and obtaining the image dimension vector information, the text information in the image of the target frame can be identified by using the image recognition text method, thereby extracting the text information of the target frame.

[0074] For example, after recognizing a text region, the text needs to be cut into individual text sub-regions, that is, regions corresponding to individual characters, so as to facilitate subsequent recognition of the text sub-regions.

[0075] In one example, the above-mentioned method of dividing the text sub-areas can be segmented by projection method, and the row and column cutting method can be used to cut individual text, and the text rows and columns in the image can be segmented by using the projection principle. In this way, the positions of the text rows and columns can be determined through horizontal and vertical projection analysis, thereby realizing the cutting and division of individual text.

[0076] It should be noted that the above text may also be characters, that is, individual characters in any form.

[0077] For example, after obtaining the text sub-region, text recognition, which can also be called character recognition, needs to be performed. Character recognition is the core part of the technology.

[0078] In one example, the cut single characters can be recognized by a machine learning algorithm and converted into editable text, and the audio can be transcribed and converted into editable text. For example, the above method of recognizing text content can be completed by OCR technology.

[0079] For example, after the text content of a separate text sub-region is identified, the individual characters need to be combined into text content according to the original layout structure, that is, the layout needs to be restored.

[0080] In one example, a layout restoration algorithm can be used, that is, by analyzing the relative positions and relationships between characters, the original layout structure can be restored. In this layout restoration process, the rows and columns, paragraphs and chapters of the text can be determined.

[0081] Optionally, in an embodiment of the present application, before identifying the text area in the picture dimension vector information in the video file through the target model, the deep learning-based video semantic retrieval method provided in the embodiment of the present application also includes: preprocessing the above picture dimension vector information through the above target model.

[0082] Exemplarily, the above preprocessing includes: picture content display direction preprocessing, picture light and shadow preprocessing.

[0083] It is understandable that before performing the above-mentioned image recognition and analyzing the text content therein, the image can be pre-processed first to make the subsequent reading process more accurate and faster.

[0084] In one example, the target frame may contain light and shadow, be blurred, or be tilted, which is not conducive to identifying the content. The image light and shadow preprocessing can be used to adjust the brightness and contrast of the image to eliminate the influence of light and shadow on text recognition.

[0085] In one example, the target frame may have light and shadow, and the image may be tilted, which can be corrected by minimum area rectangle processing. Minimum area rectangle processing includes tilt preprocessing and distortion preprocessing. Tilt preprocessing is to use image processing algorithms to detect and correct the tilt angle of the image so that the text lines remain horizontal; distortion preprocessing is to use image processing algorithms to correct images with spatial distortion.

[0086] This usually involves geometric transformation of the image to restore the normal form of the text. Some unqualified target frames are processed through light and shadow preprocessing and minimum area rectangle processing to obtain qualified images. After preprocessing, the text content in the image has become relatively clear and standardized.

[0087] Optionally, in an embodiment of the present application, after the first text vector information is generated, the deep learning-based video semantic retrieval method provided in the embodiment of the present application also includes: performing semantic recognition on the first text vector information through a target model to generate a semantic recognition result; and updating the first text vector information according to the semantic recognition result.

[0088] Exemplarily, the readability of the updated first text vector information is enhanced.

[0089] Exemplarily, the process of performing the semantic recognition to generate the semantic recognition result may be a process of post-processing text.

[0090] Exemplarily, the above post-processing text mainly corrects and optimizes the recognized text to improve the accuracy and readability of the text.

[0091] In one example, the post-processed text can be corrected using the context information and language model in the target model. Specifically, the context information and language model can correct the recognized text to generate the first text vector information. This can correct some common recognition errors, such as typos, punctuation errors, etc.

[0092] In another embodiment of the present application, a specific implementation method for converting audio dimension vector information into single-dimensional data information is also provided. Exemplarily, in the case where the above-mentioned multi-dimensional vector information includes audio dimension vector information, the above-mentioned "converting the multi-dimensional vector information in the above-mentioned video file into single-dimensional vector information through the target model" includes: decoding the above-mentioned audio dimension vector information according to the preset audio information in the above-mentioned target model, and outputting the second text vector information corresponding to the audio dimension vector information, and the above-mentioned target model includes the correspondence between the audio encoding information corresponding to the preset audio information and the audio decoding information.

[0093] Exemplarily, the second text vector information is single-dimensional vector information including text dimension vector information.

[0094] Exemplarily, the decoding of the audio dimension vector information may be accomplished through an encoder-decoder architecture in a target model.

[0095] The main components of the encoder-decoder architecture include: Encoder: Converts the input audio signal into a series of feature representations. Decoder: Converts the feature representation into text output.

[0096] Exemplarily, the preset audio processing model may be an autoregressive decoding model.

[0097] Furthermore, after receiving the audio dimension vector information, the decoding strategy of the autoregressive decoding model is adopted to gradually generate the text output. The specific process is as follows: after receiving the audio dimension vector information, the autoregressive decoding model is used to generate the first character or word from the audio dimension vector information representation, and then the next character or word is predicted by recursion based on the generated text and audio features, and the next character or word is generated and predicted. This process continues until the end symbol is generated, thereby generating the second text vector information.

[0098] For the generated second text vector information, post-processing steps can be performed to enhance its readability. For example: Denoising the second text vector information: removing extra spaces, punctuation, etc.; Spell Correction the second text vector information: spell checking and correcting the generated text The following describes the training process of the above autoregressive decoding model: The training process includes: Supervised Learning: The model is trained with labeled audio and corresponding text. The goal is to minimize the gap between audio features and real text. Multi-Task Learning: By training multiple languages ​​and tasks at the same time, the generalization ability of the model can be enhanced, making its recognition accuracy in different languages ​​and dialects higher. Optionally, in an embodiment of the present application, before decoding the audio dimensional vector information according to the preset audio processing model in the target model and outputting the second text vector information corresponding to the audio dimensional vector information, the deep learning-based video semantic retrieval method provided in the embodiment of the present application also includes: performing sampling processing on the audio dimensional vector information to generate discrete feature information corresponding to the audio dimensional vector information; and extracting feature information corresponding to the target audio in the discrete feature information.

[0099] It is understandable that after receiving the audio input, the audio dimension vector information of the target audio is first preprocessed, which usually includes the following steps: Sampling: Convert the continuous audio signal into a discrete signal. Feature Extraction: Use technologies such as Mel Frequency Cepstral Coefficient (MFCC) and Short Time Fourier Transform (STFT) to extract features from the audio dimension vector information of the target audio. These features can effectively capture the voice information in the audio.

[0100] The present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the training rule determination method described in the present application is implemented. For example, the following can be executed: Figure 1 The individual steps of the method are shown.

[0101] The present invention provides a computer program product, which includes instructions that are executed by a processor when the processor executes the instructions. Figure 1 The individual steps of the method are shown.

[0102] It should be noted that although the operations of the method of the present invention are described in a particular order in the drawings, this does not require or imply that the operations must be performed in this particular order or that all illustrated operations must be performed to achieve desired results.

[0103] Figure 2 This is a block diagram of a video semantic retrieval request device based on deep learning according to an embodiment of the present application. Figure 2 The device includes a receiving module 601, an execution module 602, a storage module 603 and a determination module 604.

[0104] A receiving module 601 is used to receive a video file and decompose the video file through a target model to determine a target frame and a target audio in the video file, wherein the target frame includes a video frame for displaying key information in the video file, and the target audio includes audio data for displaying key information in the video file; An execution module 602 is used to parse the target frame and the video segment corresponding to the target frame into multi-dimensional vector information, where the multi-dimensional vector information includes information content and the position of the information content in the video file; The execution module 602 is further used to convert the multi-dimensional vector information in the video file into single-dimensional vector information through the target model; A storage module 603 is used to store the single-dimensional vector information in a target database; The determination module 604 is used to compare and analyze the preset vector information in the target database with the single-dimensional vector information, and determine that the video file is a target file when the preset vector information matches the single-dimensional vector information.

[0105] In one embodiment, the receiving module 601 is specifically used to: classify all video frames in the video file by a preset video frame classification method of the target model; and confirm the target frame and the target audio in the video file.

[0106] In one embodiment, when the multi-dimensional vector information includes picture dimension vector information, the execution module 602 is specifically used to: identify the text area in the picture dimension vector information in the video file through the target model; segment the text area to generate multiple text sub-areas, each text sub-area includes a character; identify the text content of each text sub-area in the multiple text sub-areas; combine the text contents according to the corresponding arrangement of the multiple text sub-areas to generate first text vector information, which is single-dimensional vector information including text dimension vector information.

[0107] In one embodiment, the execution module 602 is further used to preprocess the image dimension vector information through the target model, and the preprocessing includes: minimum area rectangle processing and image light and shadow preprocessing.

[0108] In one embodiment, the above-mentioned execution module 602 is also used to: perform semantic recognition on the first text vector information through a target model to generate a semantic recognition result; update the first text vector information according to the semantic recognition result, and enhance the readability of the updated first text vector information.

[0109] In one embodiment, when the multi-dimensional vector information includes audio dimension vector information, the above-mentioned execution module 602 is specifically used to: decode the audio dimension vector information according to the preset audio processing model in the target model, and output the second text vector information corresponding to the audio dimension vector information, wherein the target model includes the correspondence between the audio encoding information and the audio decoding information corresponding to the preset audio information, and the second text vector information is a single dimension vector information including text dimension vector information.

[0110] In one embodiment, the execution module 602 is further used to: perform sampling processing on the audio dimension vector information to generate discrete feature information corresponding to the audio dimension vector information; Extract feature information corresponding to the target audio from the discrete feature information.

[0111] It should be understood that the units recorded in the video semantic retrieval device based on deep learning correspond to the various steps in the method described in the accompanying drawings. Therefore, the operations and features described above for the method are also applicable to the video semantic retrieval device based on deep learning and the units contained therein, and will not be repeated here. The video semantic retrieval device based on deep learning can be pre-implemented in a browser or other security application of a computer device, or can be loaded into a browser or its security application of a computer device by downloading or the like. The corresponding units in the video semantic retrieval device based on deep learning can cooperate with the units in the computer device to implement the solution of the embodiment of the present application.

[0112] For the several modules or units mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0113] It should be noted that for details not disclosed in the deep learning-based video semantic retrieval device in the embodiment of the present application, please refer to the details disclosed in the above embodiments of the present application, which will not be repeated here.

[0114] Reference below Figure 3 , Figure 3 FIG. 1 is a schematic diagram showing a structure of a computer device suitable for implementing an embodiment of the present application. Figure 3 As shown, the computer system 1700 includes a central processing unit (CPU) 1701, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1702 or the program loaded from the storage part 1708 to the random access memory (RAM) 1703. In the RAM 1703, various programs and data required for the operation instructions of the system are also stored. The CPU 1701, the ROM 1702, and the RAM 1703 are connected to each other through a bus 1704. An input / output (I / O) interface 1705 is also connected to the bus 1704.

[0115] The following components are connected to the I / O interface 1705: an input section 1706 including a keyboard, a mouse, etc.; an output section 1707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1708 including a hard disk, etc.; and a communication section 1709 including a network interface card such as a LAN card, a modem, etc. The communication section 1709 performs communication processing via a network such as the Internet. A drive 1710 is also connected to the I / O interface 1705 as needed. A removable medium 1711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1710 as needed, so that a computer program read therefrom is installed into the storage section 1708 as needed.

[0116] In particular, according to an embodiment of the present application, the above reference flow chart Figure 1 The described process can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 1709, and / or installed from a removable medium 1711. When the computer program is executed by the central processing unit (CPU) 1701, the above-mentioned functions defined in the system of the present application are executed.

[0117] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium such as a computer-readable storage medium that can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0118] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operating instructions of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the aforementioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, the boxes represented by two connections can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operating instruction, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] The units or modules involved in the embodiments described in the present application may be implemented by software or hardware. The units or modules described may also be set in a processor. For example, it may be described as: a processor includes a first receiving module, a second receiving module, and a sending module. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves.

[0120] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device. The above computer-readable storage medium stores one or more programs, and when the above programs are used by one or more processors to execute the deep learning-based video semantic retrieval method described in the present application.

[0121] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the aforementioned disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present application (but not limited to) by each other to form a technical solution.

Claims

1. A video semantic retrieval method based on deep learning, characterized in that: include: Receive a video file and decompose the video file through a target model to determine a target frame and a target audio in the video file, wherein the target frame includes a video frame for displaying key information in the video file, and the target audio includes audio data for displaying key information in the video file; Parsing the target frame and the target audio into multi-dimensional vector information, wherein the multi-dimensional vector information includes information content of different dimensions and the position of the information content in the video file; Converting the multi-dimensional vector information in the video file into single-dimensional vector information through a target model; Storing the single-dimensional vector information in a target database; The preset vector information in the target database is compared and analyzed with the single-dimensional vector information, and when the preset vector information matches the single-dimensional vector information, the video file is determined to be the target file.

2. The method according to claim 1, characterized in that Receiving a video file and decomposing it through a target model to determine a target frame and a target audio in the video file, including: Classifying all video frames in the video file by using a preset video frame classification method of the target model; A target frame and the target audio in the video file are confirmed.

3. The method according to claim 1, characterized in that In a case where the multi-dimensional vector information includes picture dimension vector information, converting the multi-dimensional vector information in the video file into single-dimensional vector information through the target model includes: Using the target model, identifying a text area in the picture dimension vector information in the video file; Segmenting the text area to generate a plurality of text sub-areas, each of which includes one character; Identify the text content of each of the multiple text sub-regions; The text contents are combined according to the arrangement methods corresponding to the multiple text sub-regions to generate first text vector information, where the first text vector information is single-dimensional vector information including text dimension vector information.

4. The method according to claim 3, characterized in that Before identifying the text area in the picture dimension vector information in the video file through the target model, the method further includes: The image dimension vector information is preprocessed through the target model, and the preprocessing includes: minimum area rectangle processing and image light and shadow preprocessing.

5. The method according to claim 3, characterized in that: After generating the first text vector information, the method further includes: Performing semantic recognition on the first text vector information through a target model to generate a semantic recognition result; The first text vector information is updated according to the semantic recognition result, and the readability of the updated first text vector information is enhanced.

6. The method according to claim 1, characterized in that In a case where the multi-dimensional vector information includes audio dimension vector information, converting the multi-dimensional vector information in the video file into single-dimensional vector information through the target model includes: According to the preset audio processing model in the target model, the audio dimension vector information is decoded and second text vector information corresponding to the audio dimension vector information is output. The target model includes the correspondence between the audio encoding information and the audio decoding information corresponding to the preset audio information. The second text vector information is single-dimensional vector information including text dimension vector information.

7. The method according to claim 6, characterized in that Before decoding the audio dimension vector information according to the preset audio processing model in the target model and outputting the second text vector information corresponding to the audio dimension vector information, the method further includes: Performing sampling processing on the audio dimension vector information to generate discrete feature information corresponding to the audio dimension vector information; Extract feature information corresponding to the target audio from the discrete feature information.

8. A video semantic retrieval device based on deep learning, characterized in that: include: A receiving module, configured to receive a video file and decompose the video file through a target model to determine a target frame and a target audio in the video file, wherein the target frame includes a video frame for displaying key information in the video file, and the target audio includes audio data for displaying key information in the video file; An execution module, configured to parse the target frame and the video clip corresponding to the target frame into multi-dimensional vector information, wherein the multi-dimensional vector information includes information content and a position of the information content in the video file; The execution module is further used to convert the multi-dimensional vector information in the video file into single-dimensional vector information through a target model; A storage module, used for storing the single-dimensional vector information in a target database; The determination module is used to compare and analyze the preset vector information in the target database with the single-dimensional vector information, and determine that the video file is a target file when the preset vector information matches the single-dimensional vector information.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising instructions, characterized in that: When the instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images