A deep learning-based information positioning method and related equipment

Abstract

The present invention provides an information positioning method based on deep learning and related equipment, which respectively converts the audio feature vector and the image feature vector corresponding to the audio information of the video file to be processed into first text information by obtaining the audio feature vector corresponding to the audio feature vector and the image feature vector corresponding to the audio feature vector; extracts the target information and the positioning information of the target information in the first text information. When the method of the present invention performs information positioning, while recognizing the video image, it also adds the recognition of the audio information corresponding to the video file to be processed, so that the entire content of the video is fully considered, the accuracy of video content recognition is improved, and the accuracy of information positioning is improved.

Description

Technical Field

[0001] The present invention relates to the field of display control technology, and in particular to an information positioning method based on deep learning and related equipment. Background Art

[0002] With the development of network technology, the Internet has become more and more popular, among which sharing videos shot through the Internet has become increasingly popular among users.

[0003] Faced with a large amount of video information on the Internet, if users want to locate the video clips they want to watch, such as their favorite crosstalk or skit clips, or if they want to filter out some disliked clips, they usually locate the video clips by identifying video tags. However, the method of locating videos using video tags may consume a lot of human resources, as each video may have dozens or even hundreds of tags. In addition, since many videos do not have corresponding tags, the positioning method using tags cannot meet the requirements for accurate information positioning.

[0004] Therefore, the prior art needs to be further improved. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides an information positioning method and related equipment based on deep learning, which overcomes the defects of the prior art in which the located or filtered target information is low in accuracy and consumes a lot of human resources when only using video tags to search, locate or filter target information in videos.

[0006] In a first aspect, an embodiment of the present invention provides an information positioning method based on deep learning, wherein the method comprises the steps of:

[0007] Acquire audio information and video image information of a video file to be processed, wherein a time sequence of the video image information in the video file to be processed is the same as a time sequence of the audio information in the video file to be processed;

[0008] Determining first text information according to the audio information and the video image information;

[0009] The first text information is input into a trained text integration model to obtain extracted target information and location information of the target information in the first text information; the text integration model is trained based on the correspondence between sample text information of marked target information and sample text information of unmarked target information.

[0010] Optionally, determining the first text information according to the audio information and the video image information includes:

[0011] Determine an audio feature vector corresponding to the audio information according to the audio information, and determine an image feature vector corresponding to the video image information according to the video image information;

[0012] The audio feature vector and the image feature vector are converted into first text information.

[0013] Optionally, the step of converting the audio feature vector and the image feature vector into first text information includes:

[0014] splicing the audio feature vector and the image feature vector into a video vector matrix;

[0015] The video vectors contained in the video vector matrix are translated into first text information.

[0016] Optionally, the step of translating the video vector contained in the video vector matrix into first text information includes:

[0017] The video vector matrix is ​​input into a trained content recognition model, and the content recognition model outputs the first text information. The content recognition model is trained based on the correspondence between the video vectors of the marked text information and the video vectors of the unmarked text information.

[0018] Optionally, the step of determining, according to the audio information, an audio feature vector corresponding to the audio information includes:

[0019] According to preset sampling points, the frequency spectrum of the audio information is sampled to obtain a sampled frequency spectrum;

[0020] The sampled spectrum is encoded into the audio feature vector.

[0021] Optionally, the step of determining, according to the video image information, an image feature vector corresponding to the video image information includes:

[0022] intercepting an image frame of the video image information;

[0023] The feature maps corresponding to the respective image frames in the video image information are extracted, and the image feature vectors corresponding to the respective image frames are obtained according to the feature maps corresponding to the respective image frames.

[0024] Optionally, the step of intercepting an image frame of the video image information includes:

[0025] Cutting the video image information into a plurality of video image frame segments according to a preset video frame length;

[0026] intercepting image frames of each video image frame segment to obtain image frame sets corresponding to each video image frame segment, wherein the image frame set corresponding to each video image frame segment includes each image frame in the video image frame segment;

[0027] The step of extracting the feature graphs corresponding to the respective image frames in the video image information comprises:

[0028] Extract feature maps corresponding to each image frame in each image frame set.

[0029] Optionally, the step of obtaining the image feature vectors corresponding to the image frames according to the feature maps corresponding to the image frames includes:

[0030] The image features corresponding to each image frame are input into a trained convolutional neural network to obtain image feature vectors corresponding to each image frame. The convolutional neural network is trained based on the correspondence between the labeled image feature vectors and the input feature map.

[0031] Optionally, the step of inputting the feature map into a trained convolutional neural network to obtain an image feature vector of each image frame includes:

[0032] Through convolution and pooling operations, the two-dimensional image feature vectors corresponding to each feature map are converted into one-dimensional image feature vectors corresponding to each feature map.

[0033] Optionally, the step of splicing the audio feature vector and the image feature vector into a video vector matrix includes:

[0034] The one-dimensional image feature matrices corresponding to the feature maps are merged and added to the two-dimensional matrix corresponding to the audio feature vector in the form of column vectors to obtain a two-dimensional video vector matrix.

[0035] Optional,

[0036] The text integration model is an encoding-decoding model, which includes: an encoding layer, an attention layer and a decoding layer;

[0037] The step of inputting the first text information into the trained text integration model to obtain the extracted target information and the location information of the target information in the first text information includes:

[0038] Convert the first text information into a text sequence, input the text sequence into the encoding layer, and output a hidden layer sequence after hidden encoding the text sequence;

[0039] Input the hidden layer sequence into the attention layer, and output the key information contained in the hidden layer sequence;

[0040] Inputting the identified key information and the hidden layer sequence into the decoding layer, extracting the theme of the hidden layer sequence and the target information analyzed based on the theme;

[0041] The positioning information of the target information is obtained according to the position information of the target information in the first text information.

[0042] Optionally, after the step of extracting the theme of the hidden layer sequence and analyzing the target information based on the theme, the step further includes:

[0043] The extracted target information is displayed, and analysis information of the target information is output.

[0044] Optionally, the information positioning method further includes:

[0045] The target information extracted by the information positioning method and the positioning information of the target information in the first text information;

[0046] The audio information and the video image information are filtered according to the target information and the positioning information, and a filtered video file is generated according to the filtered audio information and the video image information.

[0047] Optionally, the step of filtering the audio information and the video image information according to the target information and the positioning information, and generating a filtered video file according to the filtered audio information and the video image information includes:

[0048] Respectively splicing a plurality of audio feature vectors contained in the audio information and a plurality of image feature vectors corresponding to the plurality of audio feature vectors into a plurality of video vectors;

[0049] Filtering the plurality of video vectors in sequence according to the extracted target information and the positioning information;

[0050] According to the time series corresponding to each video vector obtained after filtering, the video vectors are sorted, and the sorted video vectors are integrated into a filtered video file.

[0051] In a second aspect, this embodiment further provides an information positioning device based on deep learning, which includes:

[0052] A video information extraction module, used to obtain audio information and video image information of a video file to be processed, wherein the time sequence of the video image information in the video file to be processed is the same as the time sequence of the audio information in the video file to be processed;

[0053] A description conversion module, used to determine first text information according to the audio information and the video image information;

[0054] A text integration module is used to input the first text information into a trained text integration model to obtain the extracted target information and the positioning information of the target information in the first text information; the text integration model is trained based on the correspondence between the text information marked as target information and the sample text information that is not marked as target information.

[0055] According to a third aspect, a computer device comprises a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the steps of the method when executing the computer program.

[0056] In a fourth aspect, a computer-readable storage medium stores a computer program thereon, wherein the computer program implements the steps of the described method when executed by a processor.

[0057] Compared with the prior art, the embodiments of the present invention have the following advantages:

[0058] According to the method provided by the embodiment of the present invention, the audio information of the video file to be processed and the video image information having the same time sequence as the audio information are obtained, and the first text information is determined based on the audio information and the video image information; the first text information is input into the trained text integration model to obtain the extracted target information and the positioning information of the target information in the first text information. It can be seen that when the method of the present invention performs information positioning on the video image, it also adds the information positioning of the audio information corresponding to the video file, thus fully considering the entire content of the video, improving the accuracy of video content recognition and improving the positioning and searching effect of the target information in the video file. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 is a flowchart of the steps of an information method based on deep learning in an embodiment of the present invention;

[0061] Figure 2 is a schematic diagram of the implementation principle of audio information recognition in an embodiment of the present invention;

[0062] Figure 3 is a schematic diagram of the implementation principle of image feature vector recognition in an embodiment of the present invention;

[0063] Figure 4 It is a schematic diagram of the realization principle of the key information and target information identification in the method of the present invention;

[0064] Figure 5a It is a schematic diagram of the filtering effect without considering audio filtering in the prior art;

[0065] Figure 5b is a schematic diagram of the filtering effect considering audio and image features in an embodiment of the present invention;

[0066] Figure 6 It is a schematic diagram of the structure of a device in an embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0068] After research, the inventor found that the information positioning methods existing in the prior art only locate the video image information of the video file, but do not consider the audio information of the video file in combination with the video image information. A complete video file contains two parts: audio information and video image information. Since the information positioning methods used in the prior art lack consideration of audio information, it is impossible to achieve accurate positioning of the information in the video file.

[0069] In order to solve the above problems, in an embodiment of the present invention, comprehensive consideration is given to filtering both audio information and video image information in a video file, so as to obtain a precisely filtered video file. In an embodiment of the present invention, when a user wants to filter a video file, the audio information of the video file is first obtained, and then the video image information corresponding to the audio information and in the same time series is obtained, and the audio feature vector of the audio information and the image feature vector of the video image information are obtained respectively, and the audio feature vector and the image feature vector are converted into text information expressed in text, and the target information contained in the converted text information is located, and then the located target information is deleted from the video file, so as to achieve accurate positioning of the target information in the video file.

[0070] For example, this embodiment can be applied to the following scenario. The scenario includes: users and devices. The terminal device can be any device with a video file input function such as a mobile phone, IPAD, desktop computer, etc. The user can input the video file to be processed into the terminal device, and the terminal device can respond to the user's operation of information positioning on the video file to be processed to start information positioning. When the terminal device starts to position the information of the video file to be processed, the audio information of the video file is first obtained, and an audio feature vector is obtained based on the audio information. After obtaining the video image information of the video file, the image feature vector of the video file is obtained. The time sequence in the video file where the image feature vector is located is the same as the time sequence in the video file where the audio feature vector is located, so that the information positioning is performed for the same video frame segment, and the accuracy of the information positioning is guaranteed. Secondly, the audio feature vector and the image feature vector are combined and converted into a text description, and the text is recognized as a whole to identify the target information contained therein, thereby realizing the positioning of the target information. Among them, the target information can be the character (such as words, phrases, sentences) information that needs to be located in the process of information positioning. For example, the target information can include at least one type of information, such as illegal content, pornography, violence, and any one or more information that may have adverse effects. Of course, the target information can also include pre-set keywords, such as music, pork, etc.

[0071] Exemplary Methods

[0072] See also Figure 1 , showing that this embodiment is an information positioning method based on deep learning, and the method embodiment may include the steps of:

[0073] Step S1, obtaining audio information and video image information of a video file to be processed, wherein a time sequence of the video image information in the video file to be processed is the same as a time sequence of the audio information in the video file to be processed.

[0074] The time series is the time period in which the segment corresponding to the audio information or video image information is in the playback timeline of the video file to be processed. For example, the time period corresponding to the audio information in the playback timeline of the video file to be processed is from the 10th second to the 200th second, then its time series is from the 10th second to the 200th second, and the time series corresponding to the corresponding video image information of the same time series is also from the 10th second to the 200th second in the video file to be processed.

[0075] Since the embodiment of the present invention realizes accurate positioning of information in a video file by comprehensive filtering of audio information and video image information, when it is necessary to locate information in a video file, it is necessary to first extract the audio information and video image information of the video file to be processed respectively in this step.

[0076] The first step is to extract the audio information of the video file to be processed.

[0077] There are many methods in the prior art for extracting audio information from video files in this step. For example, audio information can be extracted using an audio extraction tool, such as a function for extracting audio information from video files in an audio and video application, or by converting the format of the video file into MP3 or MP4 format, or by re-recording the audio information in the video file, etc.

[0078] The second step is to obtain video image information in the same time sequence as the audio information.

[0079] There are many ways in the prior art to obtain video image information that has the same time sequence as the audio information in the video file to be processed. For example, it can be obtained using video editing tools, such as video editing APP, quick editing APP or Pai Da Shi APP and other application software. It is also possible to re-record the corresponding video information in the video file to be processed through a tool with a video recording function, etc.

[0080] The first step and the second step may also be interchanged, that is, the video image information may be acquired first, and then the audio information having the same time sequence as the video image information may be acquired.

[0081] Step S2: Determine first text information according to the audio information and the video image information.

[0082] In this step, the audio information and video image information of the video file to be processed obtained in the above step S1 are converted into text descriptions, and the text descriptions converted from the audio information and video image information are analyzed to thereby identify whether the audio information and video image information contain target information. Specifically, step S2 includes the following two steps:

[0083] Step S21: determining an audio feature vector corresponding to the audio information according to the audio information, and determining an image feature vector corresponding to the video image information according to the video image information.

[0084] Step S22: Convert the audio feature vector and the image feature vector into first text information.

[0085] Since the audio information needs to be converted into a text description, it is necessary to extract the audio feature vector of the audio information in this step, and obtain the text description of the audio information based on the audio feature vector. Also, since the video image information needs to be converted into a text description, it is necessary to extract the image feature vector of the video image information in this step, and obtain the text description of the video image information based on the image feature vector.

[0086] Specifically, the method for determining the audio feature vector corresponding to the audio information according to the audio information in step S21 includes the following two methods:

[0087] The first one is implemented by sampling coding. Specifically, Figure 2 As shown, the following steps are included:

[0088] According to preset sampling points, the frequency spectrum of the audio information is sampled to obtain a sampled frequency spectrum;

[0089] The sampled spectrum is encoded into the audio feature vector.

[0090] The second method is implemented through a deep learning model, which includes the following steps:

[0091] After the language information contained in the audio information is converted into text information, the text information is converted into an expression of a feature vector using a preset autoencoder model, that is, into an audio feature vector.

[0092] The self-encoding model can choose to use an automatic encoder based on deep learning, and convert the text information input therein and the weights of each word, phrase or sentence in the text information into an expression of a feature vector, thereby obtaining the audio feature vector output in the self-encoding model.

[0093] In addition, the step of determining the image feature vector corresponding to the video image information according to the video image information in step S21 includes:

[0094] intercepting an image frame of the video image information;

[0095] The feature maps corresponding to the respective image frames in the video image information are extracted, and the image feature vectors corresponding to the respective image frames are obtained according to the feature maps corresponding to the respective image frames.

[0096] The step of intercepting the image frame of the video image information comprises:

[0097] Cutting the video image information into a plurality of video image frame segments according to a preset video frame length;

[0098] intercepting image frames of each video image frame segment to obtain image frame sets corresponding to each video image frame segment, wherein the image frame set corresponding to each video image frame segment includes each image frame in the video image frame segment;

[0099] The step of extracting the feature graphs corresponding to the respective image frames in the video image information comprises:

[0100] Extract feature maps corresponding to each image frame in each image frame set.

[0101] The feature map is mainly an image containing color features, texture features, shape features and spatial relationship features of the image. It is a global feature that describes the surface properties of the scene corresponding to the image or image area. In this embodiment, the shape features of the image are mainly considered. The main idea of ​​obtaining the image feature vector based on the feature map is to project the original sample into a low-dimensional feature space to obtain the low-dimensional image features that best reflect the nature of the feature map or distinguish the feature map.

[0102] In the above steps, the image frames of each frame in the video image information are obtained or sampled to obtain the image frames in the video image information, and the feature map corresponding to the image frame is extracted, and the image feature vector that best reflects the essence of the feature map is extracted based on the feature map.

[0103] Specifically, the step of obtaining the image feature vectors corresponding to each image frame according to the feature graphs corresponding to each image frame includes:

[0104] The image features corresponding to each image frame are input into a trained convolutional neural network to obtain image feature vectors corresponding to each image frame. The convolutional neural network is trained based on the correspondence between the labeled image feature vectors and the input feature map.

[0105] In the training of the convolutional neural network, the labeled image feature vector is used as the convolution kernel, and the labeled image feature vector is used to convolve the input feature map to obtain the image feature vector of each image frame. The convolutional neural network modifies and marks the image feature vector contained in the unknown feature map through the image feature vector in the known feature map, and finally outputs all the image frames with the image feature vectors marked, and obtains the image feature vector of the feature map based on the feature map with the image feature vector marked in the output of the convolutional neural network.

[0106] Optionally, the step of inputting the feature map into a trained convolutional neural network to obtain an image feature vector of each image frame includes:

[0107] Through convolution and pooling operations, the two-dimensional image feature vectors corresponding to each feature map are converted into one-dimensional image feature vectors corresponding to each feature map.

[0108] Specifically, the text information is converted into a word vector matrix with M rows and N columns after being converted into a feature vector by the autoencoder model, wherein M rows represent word vectors and N rows represent the weights of the word vector matrix in the text information. The image feature vectors are converted from the video image frames segmented from the video file, and the method for converting the video image frames into image feature vectors is to use a convolutional neural network model, that is, by performing convolution and pooling operations on the features extracted from the image, the two-dimensional matrix corresponding to each frame of the image is converted into a low-dimensional image feature vector corresponding to it, that is, a one-dimensional image feature vector with a lower dimension than two dimensions.

[0109] The convolutional neural network is composed of an input layer, a convolutional layer, an activation function, a pooling layer, and a fully connected layer. The convolutional layer is used to extract the features of the input image, and the pooling layer compresses the input feature map. On the one hand, it makes the feature map smaller and simplifies the network calculation complexity. On the other hand, it compresses the features and extracts the main features. The two-dimensional matrix of the video image frame is converted into a one-dimensional image feature vector through the convolutional neural network model.

[0110] The above content describes in detail the step of determining the audio feature vector according to the audio information in step S21, and determining the image feature vector corresponding to the video image information according to the video image information. Next, the step of converting the audio feature vector and the image feature vector into the first text information in step S22 is described in detail.

[0111] Since the audio feature vector and the image feature vector are integrated into text information, it is possible to more accurately identify whether the video file contains target information. Therefore, the audio feature vector and the image feature vector need to be converted into text information. In this embodiment, the text information is called first text information.

[0112] Step S22, converting the audio feature vector and the image feature vector into the first text information specifically includes:

[0113] Step S221: concatenate the audio feature vector and the image feature vector into a video vector matrix.

[0114] In order to integrate the audio feature vector and the image feature vector, this embodiment adopts the method of splicing the two vectors, adding the image feature vector to the audio feature vector to form a video vector matrix, and converting the video vector spliced ​​by the image feature vector and the audio feature vector contained in the spliced ​​video vector matrix to obtain the first text information.

[0115] Since the audio feature vector is a two-dimensional vector and the image feature vector is a one-dimensional vector, the audio feature vector and the image feature vector are spliced ​​together to form a two-dimensional matrix. If the dimensions of the matrices to which the audio feature vector and the image feature vector belong are different, the two feature vectors are made into a two-dimensional matrix with the same rows and columns by adding the default values ​​of the default settings, and the splicing between the audio feature vector and the image feature vector is realized by adding the row data above or below the row data of the other feature vector, and the splicing obtained by the splicing is the video vector matrix containing the information of the entire video file.

[0116] In this embodiment, if the audio feature vector and the image feature vector are converted separately and then the text translation results of the two vectors are obtained separately, since the correlation between the audio information and the video image information is not considered, misjudgment may be caused.

[0117] In order to avoid misjudgment caused by not considering the correlation between audio information and video image information, in this embodiment, the video vector that is a combination of audio feature vector and image feature vector is converted to obtain the first text information. In this embodiment, the image feature vector and the audio feature vector corresponding to the same time series are combined as a whole for processing, and the correlation between audio information and video image information is fully considered, so that the effect of more accurately identifying whether the target information is contained can be achieved.

[0118] For example, when the target information is to locate the content related to "violence" in the video, if the video image corresponding to a certain video file is military news, and the audio playback information is: audio related to terrorist attacks related to violence (such as the bombing of Iraq), if only the image of the military news is analyzed, it is found that the video image belongs to the news image, not the content related to "violence", so an erroneous judgment will be made that the video file does not contain content related to "violence". However, if the method provided in this embodiment is adopted, both the audio information and the image video information of the video file are analyzed. When the audio information of the video file is analyzed, it is found that the audio information of the video file is audio related to the terrorist attack. Since the audio of the terrorist attack contains content related to "violence", it can be determined that the video file contains content related to "violence". The audio clip of the terrorist attack in the video file and the video clip corresponding to the audio clip both belong to the target information to be located. Therefore, the analysis result of the method provided in this embodiment is more accurate.

[0119] Step S222: convert the video vector contained in the video vector matrix into first text information.

[0120] The video vector matrix is ​​input into a trained content recognition model, and the content recognition model outputs the first text information. The content recognition model is trained based on the correspondence between the video vectors of the marked text information and the video vectors of the unmarked text information.

[0121] In the training of the content recognition model, the video vector of the marked text information is used as the convolution kernel, and the video vector of the marked text information is used to convolve the video vector of the input unmarked text information to obtain the output text information of the input video vector. For example: the text information marked by a certain video vector is: scenery, mountain, river and tree, then the content recognition model recognizes the video vector of the unmarked text information in the input content recognition model according to the text information marked by the video vector, and marks the video vectors that also contain scenery, mountain, river and tree, and based on the same marking method, achieves the goal of marking all input video vectors.

[0122] The text information of the video vector matrix is ​​recognized by using a trained content recognition model, so as to obtain the text information corresponding to each video vector in the video vector matrix.

[0123] Step S3, input the first text information into the trained text integration model to obtain the extracted target information and the positioning information of the target information in the first text information; the text integration model is trained based on the correspondence between the sample text information marked as target information and the sample text information not marked as target information.

[0124] The positioning information is the position information of the target information in the first text information. It can be understood that the positioning information can reflect the position of the target information in the video image information and the audio information respectively. For example: if the target information is in the second to third lines of the second paragraph and the first line of the third paragraph of the first text information, and the content of the second to third lines of the second paragraph of the first text information corresponds to the position of the 30th to 50th frame of the image video frame, the content of the second to third lines of the second paragraph of the first text information corresponds to the position of the 3rd to 5th second of the audio information, the content of the first line of the third paragraph of the first text information corresponds to the position of the 100th to 120th frame of the image video frame, and the content of the first line of the third paragraph of the first text information corresponds to the 10th to 12th second of the audio information; then the positioning information of the target information is the second to third lines of the second paragraph and the first line of the third paragraph of the first text information, and the positioning information of the target information reflects that the target information is located at the 30th to 50th frame and the 100th to 120th frame of the image video information, and the target information is located at the 3rd to 5th second and the 10th to 12th second of the audio information.

[0125] In a possible implementation, the text integration model used in this embodiment is an encoding-decoding model, and the encoding-decoding model includes: an encoding layer, an attention layer, and a decoding layer;

[0126] Specifically, the step of inputting the first text information into the trained text integration model to obtain the extracted target information and the location information of the target information in the first text information includes:

[0127] Convert the first text information into a text sequence, input the text sequence into the encoding layer, and output a hidden layer sequence after hidden encoding the text sequence;

[0128] Input the hidden layer sequence into the attention layer, and output the key information contained in the hidden layer sequence;

[0129] Inputting the identified key information and the hidden layer sequence into the decoding layer, extracting the theme of the hidden layer sequence and the target information located based on the theme analysis;

[0130] The positioning information of the target information in the first text information is obtained based on the target information.

[0131] In order to enable the user to more conveniently obtain the position of the identified target information in the video file and the reason for determining it as the target information, after the steps of extracting the theme of the hidden layer sequence and locating the target information based on the theme analysis, the method further includes:

[0132] The target information located by the extracted subject is displayed, and analysis information of the target information is output.

[0133] For example, when analyzing the text information translated from the video vector, if it is found that the theme is "children's songs related to birds" and the scene of "fighting" and words related to "quarrel" appear, the violent scene and its corresponding audio information are located in the corresponding time series, and the violent scene and its corresponding audio information are output and displayed, and the filtering reason is given as: violence related. For example: when the theme is "fighting", the time series of the violent scene is first determined. If the time series is: 10 seconds to 200 seconds, the audio information in the same time series is located according to the time series, that is, the audio information with a time series of: 10 seconds to 200 seconds is obtained, and the violent scene and the obtained audio information of the same time series are synchronously output and displayed, and the reason for locating the video corresponding to the time series is given as: violence related.

[0134] In one implementation, when determining the image feature vector corresponding to the video information according to the video image information, in order to facilitate quick identification and deletion according to the positioning information of the image frame, step S21 adopts the method of cutting the video image information into multiple video image frame segments according to the preset video frame length; intercepting the image frames of each video image frame segment to obtain the image frame sets corresponding to each video image frame segment, wherein the image frame sets corresponding to each video image frame segment include each image frame in the video image frame segment; the step of extracting the feature graphs corresponding to each image frame in the video image information includes: extracting the feature graphs corresponding to each image frame in each image frame set, and obtaining the feature vectors corresponding to each image frame according to the feature graphs. For example, a ten-minute video is first divided into one-minute segments, and then the frame information of the one-minute segment is collected. Since the image frames in this step are image frames in different video image frame segments, the image frames corresponding to each image frame segment can be searched synchronously and respectively, so as to quickly locate the position of the image frame.

[0135] Specifically, taking the specific application embodiment of this method as an example, combined with Figure 2 FIG5 is a further explanation of the method of the present invention.

[0136] First, if Figure 2 As shown, firstly, the audio information of the video file needs to be extracted from the video file, and the audio information is encoded through sampling to obtain an audio feature vector corresponding to the audio information.

[0137] Specifically, Figure 3 As shown, according to the time span of the video image frame set screening in the video file, the text information corresponding to the video image frame set is converted into a vector representation through the autoencoder model, and the audio feature vector is spliced ​​with the next frame set to generate a new feature vector matrix to complete the training of the neural network.

[0138] Since the amount of information contained in the video file is too large, the video file needs to be preprocessed. Therefore, it is necessary to cut it into video segments of a certain length. Take a frame every K frames in the segment and cut it into a frame set of length J frames. The frame set contains the information of K*J frames in the video segment. Each frame in the frame set is configured with corresponding weights in the neural network and mapped into a two-dimensional matrix containing frame image information. The main framework of the model is a CNN network (convolutional neural network). Its main function is to convert the two-dimensional matrix corresponding to each frame into an image feature vector corresponding to it through convolution and pooling operations, and add the audio feature vector to the image feature vector. Since the audio feature vector is added to the image feature vector in this method, and a large part of the information to be expressed by the real video is provided by the audio information, this method, compared with the traditional method of only analyzing the image content and ignoring the audio content, realizes a more comprehensive recognition of the information contained in the video file and improves the accuracy of positioning the target information in the video file.

[0139] Secondly, it is necessary to transform the image feature vectors contained in the spliced ​​feature vector matrix to obtain the corresponding text description. In the application, the neural network model used in this step is trained based on the transformer neural network. After obtaining the feature vector representation of the complete video information, the image feature vector contained therein is translated into a text description. However, the representation of the image feature vector is much longer than the separate text representation. This model is more suitable for our task than the RNN-based translation model in terms of computational complexity and long sequence translation. Therefore, this model is used to obtain faster and more accurate video content recognition.

[0140] Thirdly, it is necessary to integrate the text information translated from the above steps and identify the keywords and target information contained therein.

[0141] Specifically, Figure 4 As shown, the text information integration module used in this step is an encoding-decoding model commonly used in the NLP field, which consists of an encoding layer, an attention layer, and a decoding layer.

[0142] The step of inputting the first text information into the trained text integration model to obtain the extracted target information and the location information of the target information in the first text information includes:

[0143] K1, converting the first text information into a text sequence, and inputting the text sequence into the encoding layer, wherein the encoding layer calculates and outputs a hidden layer sequence after hidden encoding of the text sequence;

[0144] K2. Input the hidden layer sequence into the attention layer, and output the key information contained in the hidden layer sequence; the attention layer is a key part of understanding the entire first text information content, because the attention mechanism in this layer can well preserve the information of the entire sequence and will not ignore the information at the beginning of the sentence.

[0145] K3, inputting the identified key information and the hidden layer sequence into the decoding layer, extracting the theme of the hidden layer sequence and the target information analyzed based on the theme;

[0146] K4. Obtain positioning information of the target information according to the position information of the target information in the first text information.

[0147] In order to facilitate the user to obtain the specific content of the target information located in the video file, after the step of extracting the theme of the hidden layer sequence and analyzing the located target information based on the theme in the above step K3, the following steps are also included:

[0148] K31. Display the extracted target information and output analysis information of the target information.

[0149] The user can easily know the reason why the content is located through the displayed target information and analysis information, so as to have a more comprehensive understanding of the located information.

[0150] The information locating method disclosed in this embodiment, because when locating information, adds the analysis of the audio information corresponding to the video file on the basis of the analysis of the image and video information, and combines the image and video information with the audio information, realizes a comprehensive analysis of the entire video file, improves the accuracy of video content recognition, and provides a technical guarantee for accurately identifying a certain type of information in the video file, or one or more character information.

[0151] With the development of network technology, the Internet is becoming more and more popular, and sharing videos through the Internet is becoming more and more popular among users. However, since the dissemination of video information through the Internet is not subject to the necessary restrictions, a large number of videos with bad problems are also spread on the Internet. The spread of such videos will cause trouble to users, especially for teenagers who have no ability to distinguish, because the guidance of bad information in the video may cause more serious impact.

[0152] In order to filter out bad information in the video information on the network, after step S3 in the above embodiment, the following steps are further included:

[0153] The audio information and the video image information are filtered according to the extracted target information and the positioning information, and a filtered video file is generated according to the filtered audio information and the video image information.

[0154] The target information in the video file is located using the method for locating target information in the above embodiment to obtain the position information of the target information in the video file, and then the target information is filtered according to the location information of the located target information, thereby obtaining the video file after filtering the target information.

[0155] For example: when the target information is negative information, the positioning method of this embodiment first locates the location information of the negative information in the video file, and then filters the negative information contained in the video file according to the located location information to obtain a filtered video file, thereby providing a clean audio-visual environment for network videos.

[0156] In order to filter all the target information in the video file and achieve filtering accuracy, the positioning method in the above embodiment is used to locate the target information contained in the audio information of the video file and the target information contained in the video image information, and according to the location of the located target information, the target information contained in the audio information and the video image information is filtered.

[0157] Specifically, the step of extracting the target information and the positioning information, filtering the audio information and the video image information, and generating a filtered video file according to the filtered audio information and the video image information includes:

[0158] splicing a plurality of audio feature vectors and a corresponding plurality of image feature vectors contained in the audio information and the video image information into a plurality of video vectors;

[0159] Filtering the plurality of video vectors in sequence according to the extracted target information and the positioning information;

[0160] According to the time series corresponding to each video vector obtained after filtering, the video vectors are sorted, and the sorted video vectors are integrated into a filtered video file.

[0161] The audio feature vectors and image feature vectors contained in the audio information and video image information are spliced ​​respectively. When splicing, the audio feature vectors can be directly spliced ​​with the corresponding image feature vectors of the same time series, or the one-dimensional image feature vectors can be inserted in the form of columns into the matrix where the audio feature vectors are located to obtain a video vector matrix. Each column vector in the video vector matrix is ​​called a video vector.

[0162] According to the target information and its location information analyzed in the above steps, the target information corresponding to the corresponding position is deleted from the video vector in turn to obtain multiple filtered video vectors. For example, when the target information is a violent scene, and its location is in the scene between the 20th frame and the 100th frame, the target information is filtered and deleted according to the location of the scene.

[0163] The video vectors after deleting the target information are integrated into a complete video file according to their time sequence.

[0164] In order to facilitate the user to know the filtered target information, the step of filtering the audio information and the video image information according to the extracted target information and the positioning information includes:

[0165] The extracted subject and the filtered target information are displayed, and the filtering analysis information of the target information is output.

[0166] According to the displayed subject of the filtered information and the content of the filtered target information, the user can know the subject and detailed content corresponding to the target information contained in this video file, so that the user can have a more detailed understanding of the video file.

[0167] The neural network model used in this step is a model based on LSTM neural network, whose input is the video vector matrix after the image feature vector corresponding to the frame set of the video file and the audio feature vector corresponding to the audio file are combined, and the output is the located video file. The neural network model is used because the network has good time continuity, because the result corresponding to its training is in the form of video, and the video itself has a strong temporal nature, and can well learn the rules of video screening frames. The accuracy will be much higher than other neural networks, and it can also reduce a lot of computing resources. Through this module, you can get a video that saves the complete content of the video as much as possible and filters out a small amount of bad information, instead of completely deleting the entire video.

[0168] The video filtering method adopted in this embodiment filters the target information contained in the video file, so that the filtered video file does not contain illegal content, pornography, violence and other information that may have negative effects, providing video users with a healthy viewing environment and providing technical support for the normal dissemination of video files.

[0169] Compared with the prior art, the filtering method adopted in this embodiment adds filtering of target information in the audio information in the video file, and combines the audio information with the video image information for synchronous identification and filtering, thereby achieving a more accurate filtering effect. Figure 5a and Figure 5bAs shown, when the picture in the input video file is two people, if only the video image is identified, it can only be identified that the two people are talking, but the specific content of their discussion cannot be identified, and the picture does not contain relevant target information. Then, after analyzing it using the filtering method in the prior art, no filtering is adopted. However, if the filtering method used in this embodiment is adopted, not only what the two people are talking about can be analyzed based on the picture, but also the target information of the two people talking can be analyzed, for example: the language content contains dirty words, then after analyzing it, the conclusion obtained is to filter the video frames in which the two people are talking and the content of the discussion contains dirty words, and the filtering reason given is: the content of the discussion contains target information. Therefore, compared with the prior art, the filtering method disclosed in the present invention has a more accurate filtering effect.

[0170] The present invention also provides another embodiment of an information positioning device based on deep learning, such as Figure 6 As shown, including:

[0171] The video information extraction module 610 is used to obtain the audio information and video image information of the video file to be processed, wherein the time sequence of the video image information in the video file to be processed is the same as the time sequence of the audio information in the video file to be processed; its function is as described in step S1.

[0172] The description conversion module 620 is used to describe the conversion module, which is used to determine the first text information according to the audio information and the video image information; its function is as described in step S2.

[0173] The text integration module 630 is used to input the first text information into a trained text integration model to obtain the extracted target information and the positioning information of the target information in the first text information; the text integration model is trained based on the correspondence between the sample text information marked as target information and the sample text information not marked as target information; its function is as described in step S3.

[0174] The present invention also provides another embodiment of a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein the processor implements the steps of the method when executing the computer program.

[0175] The present invention also provides another embodiment of a computer-readable storage medium having a computer program stored thereon, wherein the steps of the method described are implemented when the computer program is executed by a processor.

[0176] The present invention provides an information positioning method and related equipment based on deep learning, which obtains the audio information of the video file to be processed and the video image information with the same time sequence as the audio information, and determines the first text information according to the audio information and the video image information; the first text information is input into the trained text integration model to obtain the extracted target information and the positioning information of the target information in the first text information. It can be seen that when the method of the present invention performs information positioning, while positioning the information contained in the video image, it also adds the positioning of the information contained in the audio information corresponding to the video file. Since the information contained in the video image and the audio information in the video file are fully considered, the information positioning method of this embodiment considers the entire content of the video file, and improves the accuracy of information positioning in the video content. In addition, since the method of this embodiment can realize the accurate positioning of the target information, it can realize the accurate filtering of the corresponding bad information in the video file, which provides technical support for creating a good network video environment.

[0177] It is understandable that those skilled in the art can make equivalent substitutions or changes based on the technical solution and inventive concept of the present invention, and all these changes or substitutions should fall within the protection scope of the claims attached to the present invention.

Claims

1. A method for information positioning based on deep learning, characterized in that: The method comprises the steps of: Acquire audio information and video image information of a video file to be processed, wherein a time sequence of the video image information in the video file to be processed is the same as a time sequence of the audio information in the video file to be processed; Determining first text information according to the audio information and the video image information; Inputting the first text information into a trained text integration model to obtain extracted target information and location information of the target information in the first text information; the text integration model is trained based on the correspondence between sample text information of the marked target information and sample text information of the unmarked target information; The positioning information is used to reflect the position of the target information in the video image information and the audio information respectively; The determining the first text information according to the audio information and the video image information includes: Determine an audio feature vector corresponding to the audio information according to the audio information, and determine an image feature vector corresponding to the video image information according to the video image information; Converting the audio feature vector and the image feature vector into first text information; The step of converting the audio feature vector and the image feature vector into first text information comprises: splicing the audio feature vector and the image feature vector into a video vector matrix; Translating the video vectors contained in the video vector matrix into first text information; The text integration model is an encoding-decoding model, which includes: an encoding layer, an attention layer and a decoding layer; The step of inputting the first text information into the trained text integration model to obtain the extracted target information and the location information of the target information in the first text information includes: Convert the first text information into a text sequence, input the text sequence into the encoding layer, and output a hidden layer sequence after hidden encoding the text sequence; Input the hidden layer sequence into the attention layer, and output the key information contained in the hidden layer sequence; Inputting the identified key information and the hidden layer sequence into the decoding layer, extracting the theme of the hidden layer sequence and the target information analyzed based on the theme; The positioning information of the target information is obtained according to the position information of the target information in the first text information.

2. The information positioning method based on deep learning according to claim 1, characterized in that: The step of translating the video vector contained in the video vector matrix into first text information comprises: Inputting the video vector matrix into a trained content recognition model, the content recognition model outputting the first text information, the content recognition model being trained based on the correspondence between the video vectors of the labeled text information and the video vectors of the unlabeled text information; In the training of the content recognition model, the video vector of the labeled text information is used as a convolution kernel, and the video vector of the input unlabeled text information is convolved with the video vector of the labeled text information to obtain the text information of the input video vector.

3. The information positioning method based on deep learning according to claim 1, characterized in that: The step of determining the audio feature vector corresponding to the audio information according to the audio information comprises: According to preset sampling points, the frequency spectrum of the audio information is sampled to obtain a sampled frequency spectrum; The sampled spectrum is encoded into the audio feature vector.

4. The information positioning method based on deep learning according to claim 1, characterized in that: The step of determining the image feature vector corresponding to the video image information according to the video image information comprises: intercepting an image frame of the video image information; The feature maps corresponding to the respective image frames in the video image information are extracted, and the image feature vectors corresponding to the respective image frames are obtained according to the feature maps corresponding to the respective image frames.

5. The information positioning method based on deep learning according to claim 4, characterized in that: The step of intercepting the image frame of the video image information comprises: Cutting the video image information into a plurality of video image frame segments according to a preset video frame length; intercepting image frames of each video image frame segment to obtain image frame sets corresponding to each video image frame segment, wherein the image frame set corresponding to each video image frame segment includes each image frame in the video image frame segment; The step of extracting the feature graphs corresponding to the respective image frames in the video image information comprises: Extract feature maps corresponding to each image frame in each image frame set.

6. The information positioning method based on deep learning according to claim 4, characterized in that: The step of obtaining the image feature vectors corresponding to each image frame according to the feature graphs corresponding to each image frame comprises: The image features corresponding to each image frame are input into a trained convolutional neural network to obtain image feature vectors corresponding to each image frame. The convolutional neural network is trained based on the correspondence between the labeled image feature vectors and the input feature map.

7. The information positioning method based on deep learning according to claim 6, characterized in that: The step of inputting the image features corresponding to each image frame into the trained convolutional neural network to obtain the image feature vectors corresponding to each image frame comprises: Through convolution and pooling operations, the two-dimensional image feature vectors corresponding to each feature map are converted into one-dimensional image feature vectors corresponding to each feature map.

8. The information positioning method based on deep learning according to claim 7, characterized in that: The step of splicing the audio feature vector and the image feature vector into a video vector matrix comprises: The one-dimensional image feature matrices corresponding to the feature maps are merged and added to the two-dimensional matrix corresponding to the audio feature vector in the form of column vectors to obtain a two-dimensional video vector matrix.

9. The information positioning method based on deep learning according to claim 1, characterized in that: After the steps of extracting the theme of the hidden layer sequence and locating the target information based on the theme analysis, the method further includes: The extracted target information is displayed, and analysis information of the target information is output.

10. The information positioning method based on deep learning according to any one of claims 1 to 8, characterized in that: Also includes: The audio information and the video image information are filtered according to the target information and the positioning information, and a filtered video file is generated according to the filtered audio information and the video image information.

11. The information positioning method based on deep learning according to claim 10, characterized in that: The step of filtering the audio information and the video image information according to the target information and the positioning information, and generating a filtered video file according to the filtered audio information and the video image information comprises: Respectively splicing a plurality of audio feature vectors contained in the audio information and a plurality of image feature vectors corresponding to the plurality of audio feature vectors into a plurality of video vectors; Filtering the plurality of video vectors in sequence according to the extracted target information and the positioning information; According to the time series corresponding to each video vector obtained after filtering, the video vectors are sorted, and the sorted video vectors are integrated into a filtered video file.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images