Video analysis method, device and storage medium

Abstract

The present application relates to a video analysis method, device and storage medium. The video analysis method includes: obtaining a video to be analyzed and a question to be answered related to the video to be analyzed; determining at least one video feature information corresponding to the video to be analyzed; determining question feature information corresponding to the question to be answered; inputting at least one video feature information and question feature information into a trained video memory model for processing, so as to determine first target feature information related to the question to be answered from the video feature information; and determining answer information corresponding to the question to be answered based on the first target feature information and the question feature information, so that when performing semantic understanding analysis on the video, the video can be memorized in a targeted manner guided by the question, thereby improving the memory effect of long-time videos.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video analysis method, device and storage medium. Background Art

[0002] Video Question Answering (VideoQA) aims to perform high-level reasoning about the spatiotemporal content of videos and infer the correct answer to a given video-related question expressed in natural language.

[0003] At present, the technical solution used for video question-answering tasks is to use a trained deep learning model to extract the representation vector of the video, and then use an attention mechanism or a memory model to fuse and memorize the features of the two modalities of video and question, and finally generate the answer through a classifier.

[0004] However, the existing memory module will memorize a large amount of video information that is irrelevant to the problem, which leads to poor memory effect for long-term video information. Summary of the invention

[0005] The purpose of this application is to provide a video analysis method, device and storage medium to improve the memory effect of long-term videos.

[0006] The present application provides a video analysis method, including:

[0007] Obtain the video to be analyzed and the questions to be answered related to the video to be analyzed;

[0008] Determine at least one video feature information corresponding to the video to be analyzed;

[0009] Determine the problem feature information corresponding to the problem to be answered;

[0010] Inputting at least one video feature information and question feature information into the trained video memory model for processing, so as to determine first target feature information related to the question to be answered from the video feature information;

[0011] Answer information corresponding to the question to be answered is determined based on the first target feature information and the question feature information.

[0012] The present application also provides a video analysis device, including:

[0013] An acquisition module, used to acquire the video to be analyzed and the questions to be answered related to the video to be analyzed;

[0014] A first determination module, used to determine at least one video feature information corresponding to the video to be analyzed;

[0015] The second determination module is used to determine the problem feature information corresponding to the problem to be answered;

[0016] A third determination module is used to input at least one video feature information and question feature information into the trained video memory model for processing, so as to determine first target feature information related to the question to be answered from the video feature information;

[0017] The fourth determination module is used to determine answer information corresponding to the question to be answered based on the first target feature information and the question feature information.

[0018] The trained video memory model includes a first sub-model and a second sub-model, and the first determination module specifically includes:

[0019] An extraction unit, used for extracting a plurality of video frames from the video to be analyzed;

[0020] A first determining unit, configured to determine at least one type of video feature information corresponding to each video frame;

[0021] The third determination module specifically includes:

[0022] A second determining unit is used to input the at least one video feature information corresponding to the plurality of video frames into the first sub-model in chronological order for processing, so as to obtain a first memory content corresponding to each video frame;

[0023] The third determination unit is used to determine the first target feature information related to the question to be answered from the video feature information corresponding to each video frame based on at least one video feature information corresponding to multiple video frames, the question feature information, the first memory content and the second sub-model.

[0024] The third determining unit is specifically used for:

[0025] Determine a current video frame from the multiple video frames in chronological order, and obtain a first memory content and a first target feature information corresponding to a previous video frame as a first historical memory content and a first historical feature information, respectively;

[0026] Inputting at least one video feature information corresponding to the current video frame, the question feature information, the first historical memory content, and the first historical feature information into the second sub-model for processing, so that the second sub-model determines the first target feature information related to the question to be answered from the at least one video feature information corresponding to the current video frame;

[0027] The first memory content and the first target feature information corresponding to the current video frame are updated to the first historical memory content and the first historical feature information respectively, and the current video frame is updated using the remaining video frames, and then the step of returning to execute the step of inputting at least one video feature information, question feature information, first historical memory content and first historical feature information corresponding to the current video frame into the second sub-model for processing.

[0028] Among them, at least one video feature information includes dynamic feature information and static feature information, the first target feature information includes target dynamic feature information, target static feature information and target global feature information, and the first target feature information related to the question to be answered is determined from the at least one video feature information corresponding to the current video frame, specifically including:

[0029] According to the dynamic feature information corresponding to the current video frame, the first historical memory content, the first historical feature information and the question feature information, determining target dynamic feature information related to the question to be answered from the dynamic feature information corresponding to the current video frame;

[0030] According to the static feature information corresponding to the current video frame, the first historical memory content, the first historical feature information and the question feature information, determining target static feature information related to the question to be answered from the static feature information corresponding to the current video frame;

[0031] According to the first historical memory content, the first historical feature information and the question feature information, target global feature information related to the question to be answered corresponding to the current video frame is determined.

[0032] Among them, the fourth determination module specifically includes:

[0033] a fourth determination unit, configured to input at least one video feature information and question feature information into a trained question memory model for processing, so as to determine second target feature information related to the video to be analyzed from the question feature information;

[0034] The fifth determining unit is used to determine answer information corresponding to the question to be answered based on the first target feature information and the second target feature information.

[0035] The question feature information includes multiple word feature information, the trained question memory model includes a third sub-model and a fourth sub-model, and the fourth determination unit specifically includes:

[0036] The first determining subunit is used to input multiple word feature information into the third submodel in sequence according to the word order of the question to be answered for processing, so as to obtain the second memory content corresponding to each word feature information;

[0037] The second determination subunit is used to determine second target feature information related to the video to be analyzed from each word feature information based on multiple word feature information, at least one video feature information, the second memory content and the fourth sub-model.

[0038] The second determining subunit is specifically used for:

[0039] Determine the current word feature information from the plurality of word feature information in word order, and obtain the second memory content and the second target feature information corresponding to the previous word feature information as the second historical memory content and the second historical feature information respectively;

[0040] Inputting the current word feature information, at least one video feature information, the second historical memory content and the second historical feature information into the fourth sub-model for processing, so that the fourth sub-model determines the second target feature information related to the video to be analyzed from the current word feature information;

[0041] The second memory content and the second target feature information corresponding to the current word feature information are updated to the second historical memory content and the second historical feature information respectively, and the current word feature information is updated using the remaining word feature information, and then the step of inputting the current word feature information, the question feature information, the second historical memory content and the second historical feature information into the fourth sub-model for processing is returned to be executed.

[0042] The fifth determining unit specifically includes:

[0043] A third determination subunit is used to obtain a first target feature matrix according to the first target feature information corresponding to the plurality of video frames, and to obtain a second target feature matrix according to the second target feature information corresponding to the plurality of word feature information;

[0044] A fourth determination subunit is used to input the first target feature matrix into the trained first self-attention model for processing to obtain first semantic long-range dependency information of the first target feature information, and input the second target feature matrix into the trained second self-attention model for processing to obtain second semantic long-range dependency information of the second target feature information;

[0045] The fifth determining subunit is used to determine answer information corresponding to the question to be answered based on the first semantic remote dependency information and the second semantic remote dependency information.

[0046] An embodiment of the present application also provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are suitable for being loaded by a processor to execute any of the above-mentioned video analysis methods.

[0047] An embodiment of the present application also provides a server, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in any one of the above-mentioned video analysis methods when executing the computer program.

[0048] The video analysis method, device and storage medium provided by the present application obtain a video to be analyzed and a question to be answered related to the video to be analyzed, then determine at least one video feature information corresponding to the video to be analyzed, and determine the question feature information corresponding to the question to be answered, and then input the at least one video feature information and the question feature information into a trained video memory model for processing, so as to determine the first target feature information related to the question to be answered from the video feature information, and then determine the answer information corresponding to the question to be answered based on the first target feature information and the question feature information, so that when performing semantic understanding analysis on the video, the video can be targetedly memorized with the question as a guide, thereby improving the memory effect of long-term videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The technical solution and other beneficial effects of the present application will be made apparent by describing in detail the specific implementation methods of the present application in conjunction with the accompanying drawings.

[0050] Figure 1 is a schematic diagram of a scenario of a video analysis system provided in an embodiment of the present application;

[0051] Figure 2 It is a flow chart of the video analysis method provided in the embodiment of the present application;

[0052] Figure 3 is a screenshot of the video to be analyzed provided in the embodiment of the present application;

[0053] Figure 4 is another flow chart of the video analysis method provided in an embodiment of the present application;

[0054] Figure 5 It is a schematic diagram of the execution flow of the video analysis method provided in the embodiment of the present application.

[0055] Figure 6 is a schematic diagram of the structure of the video memory model provided in the embodiment of the present application;

[0056] Figure 7 It is a structural diagram of a question memory model provided in an embodiment of the present application;

[0057] Figure 8 It is a schematic diagram of the effect of targeted memory of the video to be analyzed and the question to be answered provided by the embodiment of the present application;

[0058] Fig. 9 is a structural schematic diagram of a video analysis device provided in an embodiment of the present application;

[0059] Fig.10 It is a schematic diagram of the structure of the server provided in the embodiment of the present application. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0061] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0062] Artificial intelligence is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0063] The video analysis method provided in the embodiment of the present application can perform semantic understanding and analysis on the video content through computer vision technology. Among them, computer vision is a science that studies how to make machines "see". To put it more concretely, computer vision refers to machine vision such as using cameras and computers to replace human eyes to identify, follow up and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, synchronous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0064] The solution provided in the embodiments of the present application relates to artificial intelligence computer vision technology, specifically to a video analysis method, device and storage medium.

[0065] See also Figure 1 , Figure 1A scene diagram of a video analysis system provided in an embodiment of the present application. The video analysis system may include any one of the video analysis devices provided in the embodiments of the present application. The video analysis device may be specifically integrated in a server, such as a video server, wherein the server may be a single server or a server cluster composed of multiple servers.

[0066] The server can obtain a video to be analyzed and a question to be answered related to the video to be analyzed; determine at least one video feature information corresponding to the video to be analyzed; determine question feature information corresponding to the question to be answered; input at least one video feature information and question feature information into a trained video memory model for processing, so as to determine first target feature information related to the question to be answered from the video feature information; and determine answer information corresponding to the question to be answered based on the first target feature information and the question feature information.

[0067] In addition, the video analysis system may further include a terminal connected to the server via a network, and the terminal may be a smart phone, a tablet computer, a smart Bluetooth device, a laptop computer, or a personal computer (PC) or other device with a video playback function.

[0068] Specifically, Figure 1 As shown, the terminal can play the video V1 and receive questions input by the user regarding the played video V1 during the video playback process (for example, "What does the person do after lifting the opponent and before punching the opponent in the face?") to trigger the server to analyze the played video V1. Thereafter, the terminal can receive the answer sent by the server (for example, "throw") and provide the answer to the user.

[0069] like Figure 2 As shown, Figure 2 : is a flow chart of a video analysis method provided in an embodiment of the present application. The specific flow of the video analysis method may be as follows:

[0070] S101. Obtain a video to be analyzed and questions to be answered related to the video to be analyzed.

[0071] The question to be answered may be a question for the video to be analyzed input by the user on the terminal when the user uses the terminal to watch the video to be analyzed, and may specifically be a question described in natural language, for example, Figure 3As shown in FIG. 1 , for the video V2 to be analyzed, the question to be answered related to the video V2 to be analyzed may be “What is the woman holding?” The video to be analyzed may be a complete existing video stored locally on the terminal or on a server. It is understandable that the video to be analyzed contains content related to the answer to the question to be answered, that is, by using computer vision technology to understand the semantics of the video to be analyzed, the answer to the question to be answered can be obtained. For example, Figure 3 (That is, a frame of image in the video V2 to be analyzed) It can be seen that the answer to the above question "What is the woman holding?" is "cat".

[0072] S102. Determine at least one type of video feature information corresponding to the video to be analyzed.

[0073] Among them, Figure 4 As shown, the above S102 may specifically include:

[0074] S1021. Extract multiple video frames from the video to be analyzed.

[0075] Specifically, since a video has a large number of frames, processing each frame of the image may easily cause unnecessary redundancy, and the processing volume may also be too large. Therefore, the above-mentioned video analysis device can extract multiple video frames from the video to be analyzed according to a preset time interval (for example, 1 second) or a preset frame interval (for example, 60 frames), wherein each video frame can be a static image. For example, taking the total length of the video to be analyzed as 3 minutes, the above-mentioned video analysis device can start from the first frame of the video to be analyzed, and extract one frame from the video to be analyzed every 1 second to obtain 181 video frames. It can be understood that the sampling time interval or sampling frame interval of the above-mentioned video frames should be of appropriate size to improve the efficiency of video analysis without affecting the accuracy of semantic understanding of the video to be analyzed.

[0076] S1022. Determine at least one type of video feature information corresponding to each video frame.

[0077] Specifically, the at least one video feature information may include dynamic feature information and static feature information, wherein the static feature information is used to characterize the features of a video frame itself, and the dynamic feature information is used to characterize the dynamic features of the process from other video frames in the video to be analyzed to the video frame.

[0078] In a specific embodiment, Figure 4 As shown, the above S1022 may include:

[0079] S1-1. Use the dynamic feature extraction network to extract original dynamic feature information from each video frame, and encode the original dynamic feature information to obtain corresponding dynamic feature information.

[0080] Specifically, Figure 5 As shown, the video analysis device can input multiple video frames F extracted from the video to be analyzed into a dynamic feature extraction network (for example, a trained C3D network) for processing to obtain the original dynamic feature information f corresponding to each video frame F. m t , and the original dynamic feature information f corresponding to the above multiple video frames F is recorded in time sequence m t Sort them to get the original dynamic feature information sequence F of the video to be analyzed m , F m =(f m 1, f m 2, ..., f m N ). Then, the video analysis device can convert the original dynamic feature information sequence F m The input is encoded in the first encoder B1 composed of a long short-term memory network (LSTM) to obtain a dynamic feature information sequence I corresponding to the video to be analyzed. m , I m =(i m 1,i m 2, ..., i m N ), the tth video frame corresponds to the dynamic feature information sequence I m The tth dynamic feature information i in m t Where N is equal to the number of sampled frames, the superscript m represents the extracted dynamic features of the video, t is a natural number in the interval [1, N], and the C3D network is a 3D convolutional network.

[0081] S1-2. Use the static feature extraction network to extract original static feature information from each video frame, and encode the original static feature information to obtain corresponding static feature information.

[0082] Specifically, Figure 5 As shown, the video analysis device can input multiple video frames F extracted from the video to be analyzed into a static feature extraction network (for example, a trained ResNet network or a VGG network) for processing to obtain the original static feature information f corresponding to each video frame F. a t , and the original static feature information f corresponding to the above multiple video frames F is stored in chronological order a t Sort them to get the original static feature information sequence F of the video to be analyzed a , F a =(f a1, f a 2, ..., f a N ). Then, the video analysis device may convert the original static feature information sequence F a The input is encoded in the second encoder B2 composed of a long short-term memory network (LSTM) to obtain a static feature information sequence I corresponding to the video to be analyzed. a , I a =(i a 1,i a 2, ..., i a N ), the tth video frame corresponds to the static feature information sequence I a The tth static feature information i in a t Where N is equal to the number of sampled frames, the superscript a represents the extracted static features of the video, t is a natural number in the interval [1, N], the ResNet network is a residual network, and the VGG network is a deep convolutional neural network.

[0083] S103. Determine the question feature information corresponding to the question to be answered.

[0084] Among them, Figure 4 As shown, the above S103 may specifically include:

[0085] S1031. Convert each word in the question to be answered into a corresponding word vector to obtain a question embedding representation corresponding to the question to be answered.

[0086] Specifically, Figure 5 As shown in the figure, the video analysis device can convert the question to be answered into a word sequence C according to the word sequence of the question to be answered, where C = (c1, c2, ..., c T ), the tth word in the question to be answered corresponds to the tth word c in the word sequence C t Next, we can use the word mapping method to map each word ct in the word sequence C to its semantic expression using the embedding layer, and use the trained GloVe model to initialize the semantic expression to obtain a 300-D (300 dimensions) word vector q t , and then get the problem embedding representation Q corresponding to the problem to be answered, Q = (q1, q2, ..., q T ). Where T is the number of words in the question to be answered, and t is a natural number in the interval [1, N].

[0087] S1032. Encode the problem embedding representation to obtain problem feature information corresponding to the problem to be processed.

[0088] Specifically, the above problem can be embedded in the third encoder B3, which represents the Q input and is composed of a long short-term memory network, to obtain the problem feature information I corresponding to the problem to be processed. q =(i q 1,i q 2, ..., i q N ). It can be understood that the above problem feature information I q Contains multiple word feature information i q t , word feature information i q t Corresponds to the tth word in the question to be answered. Where T is the number of words in the question to be answered, and t is a natural number in the interval [1, N].

[0089] S104. Input at least one video feature information and question feature information into the trained video memory model for processing, so as to determine first target feature information related to the question to be answered from the video feature information.

[0090] The at least one video feature information may include dynamic feature information i m t and static feature information i a t , and each video feature information is obtained by extracting the original video feature information from the corresponding video frame using the corresponding feature extraction network and encoding the original video feature information. The above-mentioned question feature information may include multiple word feature information i q t , and each word feature information i q t The trained video memory model may include a first sub-model and a second sub-model connected to the first sub-model. S104 may specifically include:

[0091] S1041. Input at least one video feature information corresponding to multiple video frames into the first sub-model in chronological order for processing, so as to obtain the first memory content corresponding to each video frame.

[0092] Specifically, the above S1041 may specifically include:

[0093] S2-1. Determine a current video frame from a plurality of video frames in chronological order, and obtain the first target feature information and the first memory content corresponding to the previous video frame as the first historical feature information and the first historical memory content, respectively.

[0094] Specifically, Figure 6 As shown, the above video memory model may also include a video memory layer M v and at least one hidden layer h m / h a / h v , where the memory layer M v =(m1, m2, ..., m S ), also known as the video memory layer M v In this embodiment, when the current video frame is the first video frame in the time sequence among the multiple video frames extracted from the video to be analyzed, that is, when there is no video frame in the time sequence before the current video frame among the multiple video frames, the video analysis device may store at least one hidden layer h in the video memory model. m ,h a and h v The initial value of the state parameter, that is, and As the first target feature information corresponding to the previous video frame of the current video frame, the memory layer M in the video memory model v The initial value of the state parameter, that is, M0, is used as the first memory content corresponding to the previous video frame of the current video frame. v and at least one hidden layer h m ,h a and h v The initial values ​​of the state parameters can be obtained by pre-training the above-mentioned video memory model.

[0095] S2-2. Input at least one video feature information, first historical feature information and first historical memory content corresponding to the current video frame into the first sub-model for processing, so that the first sub-model determines the first memory content corresponding to the current video frame.

[0096] Specifically, after obtaining the first memory content corresponding to the current video frame, the first memory content corresponding to the current video frame can also be used to store the memory layer M in the above video memory model. v The multiple state parameter values ​​are updated to store the first memory content in the memory layer M v middle.

[0097] S2-3. Update the first target feature information and the first memory content corresponding to the current video frame to the first historical feature information and the first historical memory content respectively, and use the remaining video frames to update the current video frame, and then return to execute the above S2-2.

[0098] In this way, the above S2-2 and S2-3 can form a loop, and each time the loop is repeated, the updated first memory content corresponding to the current video frame can be obtained until the first memory content corresponding to all video frames is obtained.

[0099] S1042. According to at least one video feature information corresponding to a plurality of video frames, question feature information, the first memory content and the second sub-model, determine the first target feature information related to the question to be answered from the video feature information corresponding to each video frame.

[0100] Specifically, the above S1042 may specifically include:

[0101] S3-1. Determine the current video frame from the multiple video frames in chronological order, and obtain the first memory content and the first target feature information corresponding to the previous video frame as the first historical memory content and the first historical feature information respectively.

[0102] Among them, the specific implementation method of the above S3-1 can refer to the specific implementation method of the above S2-1, so it will not be repeated here.

[0103] S3-2. Input at least one video feature information, question feature information, first historical memory content and first historical feature information corresponding to the current video frame into the second sub-model for processing, so that the second sub-model can determine the first target feature information related to the question to be answered from at least one video feature information corresponding to the current video frame.

[0104] Specifically, after obtaining the first target feature information corresponding to the current video frame, the first target feature information corresponding to the current video frame can also be used to identify the hidden layers h in the above video memory model. m / h a / h v The state parameter value of the first target feature information is updated to store the first target feature information in the at least one hidden layer h m / h a / h v middle.

[0105] S3-3. Update the first memory content and the first target feature information corresponding to the current video frame to the first historical memory content and the first historical feature information respectively, and use the remaining video frames to update the current video frame, and then return to execute the above S3-2.

[0106] In this way, the above S3-2 and S3-3 can form a loop, and each time the loop is completed, the first target feature information related to the question to be answered will be determined from the video feature information corresponding to the updated current video frame, until the first target feature information corresponding to all video frames is obtained.

[0107] The above-mentioned updating of the current video frame by using the remaining video frames can be understood as updating the current video frame by using a video frame that is one position after the current video frame in time sequence among the remaining video frames.

[0108] In a specific embodiment, the at least one video feature information may include dynamic feature information and static feature information. Accordingly, the first target feature information includes target dynamic feature information, target static feature information and target global feature information. The second sub-model determines the first target feature information related to the question to be answered from the at least one video feature information corresponding to the current video frame, which may specifically include:

[0109] According to the dynamic feature information corresponding to the current video frame, the first historical memory content, the question feature information and the first historical feature information, determining target dynamic feature information related to the question to be answered from the dynamic feature information corresponding to the current video frame;

[0110] Determine target static feature information related to the question to be answered from the static feature information corresponding to the current video frame according to the static feature information corresponding to the current video frame, the first historical memory content, the question feature information, and the first historical feature information;

[0111] According to the first historical memory content, the question feature information and the first historical feature information, target global feature information related to the question to be answered corresponding to the current video frame is determined.

[0112] Specifically, after obtaining the target dynamic feature information, target static feature information and target global feature information corresponding to the current video frame, the target dynamic feature information, target static feature information and target global feature information corresponding to the current video frame can be used to respectively perform the hidden layer h in the above video memory model. m , the state parameter values ​​of the hidden layer ha and the hidden layer hv are updated to store the target dynamic feature information in the hidden layer hm, store the target static feature information in the hidden layer ha, and store the target global feature information in the hidden layer hv. The target global feature information can be used to represent the fusion semantic information of the dynamic feature information and the static feature information of the video to be analyzed.

[0113] Accordingly, the first sub-model determines the first memory content corresponding to the current video frame, which may specifically include:

[0114] The dynamic feature memory content corresponding to the current video frame is determined according to the target dynamic feature information contained in the first historical feature information and the dynamic feature information corresponding to the current video frame.

[0115] The static feature memory content corresponding to the current video frame is determined according to the target static feature information included in the first historical feature information and the static feature information corresponding to the current video frame.

[0116] The first memory content corresponding to the current video frame is determined according to the dynamic feature memory content and the static feature memory content corresponding to the current video frame, the target global feature information contained in the first historical feature information, and the first historical memory content.

[0117] For example, in the second sub-model, the target dynamic feature information, target static feature information and target global feature information corresponding to the current video frame can be calculated by using formulas (1) to (4), where the calculation formula is as follows:

[0118] r t =β t ·M t-1 (1)

[0119]

[0120] where · represents the inner product, is a learnable weight. t It is the target dynamic feature information corresponding to the previous video frame of the current video frame Target static feature information and target global feature information The determined reading weight. In one embodiment, d can be 512. FC represents a fully connected layer, using tanh (hyperbolic tangent function) as a nonlinear activation function. t Represents the memory layer M from the above video memory model v The content read in is specifically the memory layer M v The weighted sum of multiple memory states in . Then, based on the current read content r t , and the dynamic feature information currently input Static feature information and problem feature information i q , use calculation formulas (3) and (4) to calculate the target dynamic feature information corresponding to the current video frame (That is, the hidden layer h at the current time t m state parameter value), target static characteristic information (That is, the hidden layer h at the current time t a state parameter values) and target global feature information (That is, the hidden layer h at the current time t v state parameter value).

[0121]

[0122]

[0123] in, express or σ represents a sigmoid function. In this embodiment, by including the question guidance in the state parameter value update operation of the hidden layer at each time step, the video memory module enhanced by the question (text) can store the video content most relevant to the question, thereby improving the storage efficiency of the video information.

[0124] Accordingly, in the first sub-model, the first memory content M corresponding to the current video frame can be calculated by calculating formulas (5) to (10): t (That is, the memory layer M at the current time t v The state parameter value of is calculated as follows:

[0125]

[0126]

[0127] Among them, c t The target dynamic feature information corresponding to the previous video frame and target static feature information The content vector determined by W is a learnable parameter and b is a bias. t Will be used to calculate the write weight corresponding to the current video frame

[0128]

[0129] In the memory layer M v When updating the state parameter value of , it is necessary to consider the proportion of the dynamic feature information and static feature information of the above-mentioned video to be analyzed, that is, to calculate As shown in calculation formula (7). It is a weight between 0 and 1 obtained by c through a softmax function.

[0130]

[0131]

[0132] In the memory layer M v When updating the state parameter value of the previous video frame, it is also necessary to consider the first memory content corresponding to the previous video frame (that is, the memory layer M at the previous moment (t-1) vHow much of the state parameter value of g needs to be retained at the current time t? This ratio is μ. μ is a weight between 0 and 1 obtained by passing g through a softmax function. g is the target global feature information corresponding to the previous video frame. And calculate c in formulas (5) and (6) t Determined.

[0133]

[0134] Finally, the first memory content M corresponding to the current video frame t (That is, the memory layer M at the current time t v The state parameter value of ) can be calculated by formula (10).

[0135] S105. Determine answer information corresponding to the question to be answered based on the first target feature information and the question feature information.

[0136] Among them, Figure 4 As shown, the above S105 may specifically include:

[0137] S1051. Input at least one video feature information and question feature information into the trained question memory model for processing, so as to determine second target feature information related to the video to be analyzed from the question feature information.

[0138] The question feature information may include multiple word feature information, the trained question memory model may include a third sub-model and a fourth sub-model connected to the third sub-model, and S1051 may specifically include:

[0139] S4-1. Input multiple word feature information into the third sub-model in sequence according to the word order of the question to be answered for processing to obtain the second memory content corresponding to each word feature information.

[0140] Specifically, the above S4-1 may specifically include:

[0141] S4-1-1. Determine the current word feature information from the plurality of word feature information in word order, and obtain the second target feature information and the second memory content corresponding to the previous word feature information as the second history feature information and the second history memory content respectively.

[0142] Specifically, Figure 7 As shown, the above problem memory model can also include a problem memory layer M q and the hidden layer h q , where the memory layer M q =(m1, m2, ..., m S ), which is the problem memory layer Mq In this embodiment, when the current word feature information is the word feature information that ranks first in the word order among the multiple word feature information, that is, when there is no word feature information whose word order is before the current word feature information among the multiple word feature information, the video analysis device may store the hidden layer h in the question memory model. q The initial value of the state parameter, that is, As the second target feature information corresponding to the previous word feature information of the current word feature information, the memory layer M in the question memory model q The initial value of the state parameter, that is, M q 0, as the second memory content corresponding to the previous word feature information of the current word feature information. In addition, the initial value of the state parameter can be obtained by pre-training the video memory model.

[0143] S4-1-2. Input the current word feature information, the second historical feature information and the second historical memory content into the third sub-model for processing, so that the third sub-model determines the second memory content corresponding to the current word feature information.

[0144] Specifically, after obtaining the second memory content corresponding to the current word feature information, the second memory content corresponding to the current word feature information can also be used to store the memory layer M in the question memory model. q The multiple state parameter values ​​are updated to store the second memory content in the memory layer M q middle.

[0145] S4-1-3. Update the second target feature information and the second memory content corresponding to the current word feature information to the second historical feature information and the second historical memory content respectively, and use the remaining word feature information to update the current word feature information, and then return to execute the above S4-1-2.

[0146] In this way, the above S4-1-2 and S4-1-3 can form a loop, and each time the loop is completed, the second memory content corresponding to the updated current word feature information can be obtained until the second memory content corresponding to all word feature information is obtained.

[0147] S4-2. According to a plurality of word feature information, at least one video feature information, the second memory content and the fourth sub-model, second target feature information related to the video to be analyzed is determined from each word feature information.

[0148] Specifically, the above S4-2 may include:

[0149] S4-2-1. Determine the current word feature information from the plurality of word feature information in word order, and obtain the second memory content and the second target feature information corresponding to the previous word feature information as the second historical memory content and the second historical feature information respectively.

[0150] Among them, the specific implementation method of the above S4-2-1 can refer to the specific implementation method of the above S4-1-1, so it will not be repeated here.

[0151] S4-2-2. Input the current word feature information, at least one video feature information, the second historical memory content and the second historical feature information into the fourth sub-model for processing, so that the fourth sub-model determines the second target feature information related to the video to be analyzed from the current word feature information.

[0152] Specifically, after obtaining the second target feature information corresponding to the current word feature information, the second target feature information corresponding to the current word feature information can also be used to perform a search on the hidden layer h in the question memory model. q The state parameter value of is updated to store the second target feature information in the hidden layer h q middle.

[0153] S4-2-3. Update the second memory content and the second target feature information corresponding to the current word feature information to the second historical memory content and the second historical feature information respectively, and use the remaining word feature information to update the current word feature information, and then return to execute the above S4-2-2.

[0154] In this way, the above S4-2-2 and S4-2-3 can form a loop, and each time the loop is completed, the second target feature information related to the video to be analyzed will be determined from the updated current word feature information until the second target feature information corresponding to all the word feature information is obtained.

[0155] Among them, the above-mentioned use of the remaining word feature information to update the current word feature information can be understood as using the word feature information in the remaining word feature information whose word order is one place after the current word feature information to update the current word feature information, wherein the word order can be understood as the order in which each word appears in the question to be answered. For example, the question to be answered is "what is this person doing", and the question to be answered includes the seven words "this", "individual", "person", "in", "do", "what" and "what". According to the order of appearance, they can be "this", "individual", "person", "in", "do", "what" and "what" in order.

[0156] For example, in the fourth sub-model, the second target feature information corresponding to the current word feature information can be calculated by calculating formulas (11) to (13): The calculation formula is as follows:

[0157]

[0158]

[0159] where · represents the inner product, are learnable weights. The second target feature information corresponding to the previous word feature information of the current word feature information and the current word feature information input at the current time t The determined read weight. t Represents the memory layer M of the memory model from the above problem q The content read in is specifically the memory layer M q Then, based on the second target feature information corresponding to the previous word feature information, The current content is read t , and the current word feature information input at the current time t Dynamic feature information m and static feature information i a , use formula (13) to calculate the current word feature information Corresponding second target feature information (That is, the hidden layer h at the current time t q state parameter value).

[0160]

[0161] Accordingly, in the third sub-model, formulas (14) to (16) can be used to calculate the current word feature information: Corresponding second memory content (That is, the memory layer M at the current time t q The state parameter value of is calculated as follows:

[0162]

[0163] in, is the feature information of the current word input at the current time t And the second target feature information corresponding to the previous word feature information The content vector determined. Content vector The write weight α will be used to calculate the feature information of the current word t，i , as shown in calculation formula (15).

[0164]

[0165] The memory layer M of the above problem memory model q The write weight α of all memory states in t，i Depends on the content vector at the current time t The second target feature information corresponding to the previous word feature information.

[0166]

[0167] Finally, the current word feature information Corresponding second memory content (That is, the memory layer M at the current time t q The state parameter value of can be calculated by formula (16), where: S is the memory layer M q The number of memory states contained.

[0168] It is understandable that the video memory model and question memory model in the above embodiments are guided by cross-modal information and can respectively realize the function of targeted memory of long videos and long question sentences, thereby playing a positive role in the comprehension of long videos and long question sentences. Figure 8 As shown, the video to be analyzed is a wrestling match video, and in the video to be analyzed, the man first lifts up his opponent, then throws him to the ground and punches him. Figure 8 The memory weights of the video shown show that the above video memory model can solve the problem "What is a man doing after lifting his opponent before punches his face" e ?” under the guidance of the “memory weight bar”, the information related to the question to be answered in the video to be analyzed (for example, “lifting his opponent” and “punches his face”) is memorized in a targeted manner. The darker the color of the video frame of the video to be analyzed on the video’s memory weight bar, the more memory content the video frame has. Figure 8 The memory weights of the questions shown indicate that the above-mentioned question memory model can, under the guidance of several video frames of the video to be analyzed, specifically memorize information related to the video to be analyzed in the question to be answered, wherein the darker the color corresponding to each word in the question to be answered on the memory weight bar of the question, the more memory content of the word.

[0169] Based on the above analysis, it can be seen that the video memory model and question memory model provided in this embodiment are not limited to the field of video question answering. In some embodiments, the above video memory model and question memory model can also be applied to many fields involving video understanding and cross-modal information analysis (for example, video retrieval, video understanding, video text matching and other fields). In other embodiments, the above video memory model and question memory model can also be applied to fields such as search recommendation to realize the search of video content based on text, or the search of corresponding text based on video.

[0170] S1052. Determine answer information corresponding to the question to be answered based on the first target feature information and the second target feature information.

[0171] The above S1052 may specifically include:

[0172] S5-1. Obtain a first target feature matrix according to first target feature information corresponding to multiple video frames, and obtain a second target feature matrix according to second target feature information corresponding to multiple word feature information.

[0173] Specifically, when the first target feature information includes target dynamic feature information, target static feature information and target global feature information, the hidden layer h in the above video memory model can be v Connect to the first target feature matrix V id , to obtain the target global feature information corresponding to all video frames (that is, the hidden layer h at each moment v The first target feature matrix V of the state parameter value id Correspondingly, the hidden layer h in the above question memory model can also be q Connected to the second target feature matrix T ex , to obtain the second target feature information corresponding to all word feature information (that is, the hidden layer h at each moment q The second target feature matrix T of the state parameter value ex .

[0174] S5-2. Input the first target feature matrix into the trained first self-attention model for processing to obtain the first semantic long-range dependency information of the first target feature information, and input the second target feature matrix into the trained second self-attention model for processing to obtain the second semantic long-range dependency information of the second target feature information.

[0175] Specifically, the first self-attention model and the second self-attention model may be self-attention models based on a scaled dot product attention mechanism as shown in equation (17).

[0176]

[0177] Among them, Q, K and V represent query, key and value respectively.

[0178] In one embodiment, the above-mentioned first self-attention model can be shown as equation (18).

[0179] V0=Attention(V id W q , V id W k , V id W υ ) (18)

[0180] The above second self-attention model can be shown as Equation (19).

[0181]

[0182] Wherein W is a learnable parameter. Specifically, the purpose of the first self-attention model is to extract the semantic long-range dependency relationship of the video feature information of interest from the video memory model to obtain the corresponding first semantic long-range sequential information, and the purpose of the second self-attention model is to extract the semantic long-range dependency relationship of the question feature information of interest from the question memory model to obtain the corresponding second semantic long-range sequential information. In this way, by using the self-attention model based on the scaled dot product attention mechanism as a non-local network model, the global dependency relationship of the feature information output by the memory model can be better extracted. For example, Figure 8 As shown, the first self-attention model and the second self-attention model will respectively focus on the global dependency relationship of the feature information output by the video memory model and the question memory model to obtain the corresponding self-attention weight of the video and the self-attention weight of the question, and then determine the correct answer "throw", wherein the darker the color of the video frame of the video to be analyzed on the self-attention weight bar of the video, the more important the video frame is, and the darker the color of each word in the question to be answered on the self-attention weight bar of the question, the more important the word is.

[0183] S5-3. Determine answer information corresponding to the question to be answered based on the first semantic remote dependency information and the second semantic remote dependency information.

[0184] Among them, the above S5-3 may specifically include:

[0185] S5-3-1. Use the trained mutual attention model to fuse the first semantic long-range dependency information and the second semantic long-range dependency information to obtain the final feature information.

[0186] Specifically, the attention A from the video to be analyzed to the question to be answered can be calculated by the following calculation formula (20):

[0187]

[0188] Similarly, the attention B from the question to be answered to the video to be analyzed can be calculated by the following calculation formula (21):

[0189]

[0190] Among them, S is the weight matrix, in

[0191] Next, the trained mutual attention model as shown in Equation (22) can be used to generate the final feature information O for answer prediction.

[0192] O=Concat(V0,A,V0⊙A,V0⊙B) (22)

[0193] Among them, V is a learnable parameter, ⊙ represents the product between elements, and Concat is the concatenation function.

[0194] S5-3-2. Input the final feature information into the trained classification model to obtain the answer corresponding to the question to be answered.

[0195] The classification model may be a softmax classifier. In a specific implementation, the classification model may be optimized by using a hinge loss function (for multiple-choice questions) or a cross entropy loss function (for open-ended questions).

[0196] It should be noted that this embodiment illustrates an implementation method of memorizing videos in a targeted manner guided by questions, but memorizing videos in a targeted manner is not limited to this implementation method of memorizing videos in a targeted manner guided by questions. In some embodiments, videos can also be memorized in a targeted manner guided by other information related to the video question-answering task. In addition, the video memory model and question memory model provided in this embodiment can both effectively learn global context-aware information from the video to be analyzed and the question to be answered, and both have a larger storage threshold, that is, a stronger storage capacity, compared to existing memory models.

[0197] From the above, it can be seen that the video analysis method provided in this embodiment obtains a video to be analyzed and a question to be answered related to the video to be analyzed, then determines at least one video feature information corresponding to the video to be analyzed, and determines the question feature information corresponding to the question to be answered, and then inputs the at least one video feature information and the question feature information into a trained video memory model for processing, so as to determine the first target feature information related to the question to be answered from the video feature information, and then determines the answer information corresponding to the question to be answered based on the first target feature information and the question feature information, so that when performing semantic understanding analysis on the video, the video can be targetedly memorized with the question as a guide, thereby improving the memory effect of long-time videos.

[0198] Based on the method described in the above embodiment, this embodiment will be further described from the perspective of the video analysis device. Fig. 9 , Fig. 9 The video analysis device provided in the embodiment of the present application is specifically described, which may include: an acquisition module 610, a first determination module 620, a second determination module 630, a third determination module 640 and a fourth determination module 650, wherein:

[0199] (1) Acquisition module 610

[0200] The acquisition module 610 is used to acquire the video to be analyzed and the questions to be answered related to the video to be analyzed.

[0201] (2) First determination module 620

[0202] The first determination module 620 is used to determine at least one type of video feature information corresponding to the video to be analyzed.

[0203] The first determining module 620 specifically includes:

[0204] An extraction unit, used for extracting a plurality of video frames from the video to be analyzed;

[0205] The first determining unit is used to determine at least one type of video feature information corresponding to each video frame.

[0206] (3) Second determination module 630

[0207] The second determination module 630 is used to determine the question feature information corresponding to the question to be answered.

[0208] (4) Third determination module 640

[0209] The third determination module 640 is used to input at least one video feature information and question feature information into the trained video memory model for processing, so as to determine the first target feature information related to the question to be answered from the video feature information.

[0210] The trained video memory model may include a first sub-model and a second sub-model, and the third determination module 640 may specifically include:

[0211] A second determining unit is used to input the at least one video feature information corresponding to the plurality of video frames into the first sub-model in chronological order for processing, so as to obtain a first memory content corresponding to each video frame;

[0212] The third determination unit is used to determine the first target feature information related to the question to be answered from the video feature information corresponding to each video frame based on at least one video feature information corresponding to multiple video frames, the question feature information, the first memory content and the second sub-model.

[0213] Specifically, the third determining unit may be used to execute:

[0214] Determine a current video frame from the multiple video frames in chronological order, and obtain a first memory content and a first target feature information corresponding to a previous video frame as a first historical memory content and a first historical feature information, respectively;

[0215] Inputting at least one video feature information corresponding to the current video frame, the question feature information, the first historical memory content, and the first historical feature information into the second sub-model for processing, so that the second sub-model determines the first target feature information related to the question to be answered from the at least one video feature information corresponding to the current video frame;

[0216] The first memory content and the first target feature information corresponding to the current video frame are respectively updated to the first historical memory content and the first historical feature information, and the current video frame is updated using the remaining video frames. After that, the third determination unit returns to re-execute the input of at least one video feature information, question feature information, first historical memory content and first historical feature information corresponding to the current video frame into the second sub-model for processing.

[0217] In a specific embodiment, the at least one video feature information may include dynamic feature information and static feature information, the first target feature information may include target dynamic feature information, target static feature information and target global feature information, and the second sub-model may specifically perform the following when determining the first target feature information related to the question to be answered from the at least one video feature information corresponding to the current video frame:

[0218] According to the dynamic feature information corresponding to the current video frame, the first historical memory content, the first historical feature information and the question feature information, determining target dynamic feature information related to the question to be answered from the dynamic feature information corresponding to the current video frame;

[0219] According to the static feature information corresponding to the current video frame, the first historical memory content, the first historical feature information and the question feature information, determining target static feature information related to the question to be answered from the static feature information corresponding to the current video frame;

[0220] According to the first historical memory content, the first historical feature information and the question feature information, target global feature information related to the question to be answered corresponding to the current video frame is determined.

[0221] (5) Fourth determination module 650

[0222] The fourth determination module 650 is used to determine answer information corresponding to the question to be answered based on the first target feature information and the question feature information.

[0223] The fourth determining module 650 may specifically include:

[0224] a fourth determination unit, configured to input at least one video feature information and question feature information into a trained question memory model for processing, so as to determine second target feature information related to the video to be analyzed from the question feature information;

[0225] The fifth determining unit is used to determine answer information corresponding to the question to be answered based on the first target feature information and the second target feature information.

[0226] In one embodiment, the question feature information shell includes multiple word feature information, the trained question memory model may include a third sub-model and a fourth sub-model, and the fourth determination unit may specifically include:

[0227] The first determining subunit is used to input multiple word feature information into the third submodel in sequence according to the word order of the question to be answered for processing, so as to obtain the second memory content corresponding to each word feature information;

[0228] The second determination subunit is used to determine second target feature information related to the video to be analyzed from each word feature information based on multiple word feature information, at least one video feature information, the second memory content and the fourth sub-model.

[0229] Specifically, the second determining subunit may be used to execute:

[0230] Determine the current word feature information from the plurality of word feature information in word order, and obtain the second memory content and the second target feature information corresponding to the previous word feature information as the second historical memory content and the second historical feature information respectively;

[0231] Inputting the current word feature information, at least one video feature information, the second historical memory content and the second historical feature information into the fourth sub-model for processing, so that the fourth sub-model determines the second target feature information related to the video to be analyzed from the current word feature information;

[0232] The second memory content and the second target feature information corresponding to the current word feature information are updated to the second historical memory content and the second historical feature information respectively, and the current word feature information is updated using the remaining word feature information. After that, the above-mentioned second determination subunit returns to re-execute the current word feature information, the question feature information, the second historical memory content and the second historical feature information into the fourth sub-model for processing.

[0233] In another embodiment, the fifth determining unit may specifically include:

[0234] A third determination subunit is used to obtain a first target feature matrix according to the first target feature information corresponding to the plurality of video frames, and to obtain a second target feature matrix according to the second target feature information corresponding to the plurality of word feature information;

[0235] A fourth determination subunit is used to input the first target feature matrix into the trained first self-attention model for processing to obtain first semantic long-range dependency information of the first target feature information, and input the second target feature matrix into the trained second self-attention model for processing to obtain second semantic long-range dependency information of the second target feature information;

[0236] The fifth determining subunit is used to determine answer information corresponding to the question to be answered based on the first semantic remote dependency information and the second semantic remote dependency information.

[0237] In specific implementation, the above sub-units, units and modules can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above sub-units, units and modules can refer to the previous method embodiments and will not be repeated here.

[0238] As can be seen from the above, the video analysis device provided in this embodiment includes an acquisition module for acquiring a video to be analyzed and a question to be answered related to the video to be analyzed; a first determination module for determining at least one video feature information corresponding to the video to be analyzed; a second determination module for determining question feature information corresponding to the question to be answered; a third determination module for inputting at least one video feature information and question feature information into a trained video memory model for processing, so as to determine first target feature information related to the question to be answered from the video feature information; a fourth determination module for determining answer information corresponding to the question to be answered based on the first target feature information and the question feature information, so that when the video is semantically understood, the video can be memorized in a targeted manner guided by the question, thereby improving the memory effect of long-time videos.

[0239] Accordingly, the embodiment of the present application also provides a server, which can be a single server or a server cluster composed of multiple servers, such as Fig.10 As shown, it shows a schematic diagram of the structure of the server involved in the embodiment of the present application, specifically:

[0240] The server may include one or more processing core processors 401, one or more computer-readable storage media memories 402, radio frequency (RF) circuits 403, power supplies 404, input units 405, and display units 406. Those skilled in the art will appreciate that Fig.10 The server structure shown in the figure does not constitute a limitation on the server, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0241] The processor 401 is the control center of the server, and uses various interfaces and lines to connect various parts of the entire server. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 performs various functions of the server and processes data, thereby controlling the server as a whole. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0242] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the server, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0243] The RF circuit 403 can be used for receiving and sending signals during the process of sending and receiving information. In particular, after receiving the downlink information of the base station, it is handed over to one or more processors 401 for processing; in addition, the data related to the uplink is sent to the base station. Generally, the RF circuit 403 includes but is not limited to an antenna, at least one amplifier, a tuner, one or more oscillators, a user identity module (SIM) card, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 403 can also communicate with the network and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to the Global System of Mobilecommunication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0244] The server also includes a power supply 404 (such as a battery) for supplying power to each component. Preferably, the power supply 404 can be logically connected to the processor 401 through a power management system, so that the power management system can manage charging, discharging, power consumption, etc. The power supply 404 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0245] The server may also include an input unit 405, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, in a specific embodiment, the input unit 405 may include a touch-sensitive surface and other input devices. The touch-sensitive surface, also known as a touch display screen or a touchpad, can collect user touch operations on or near it (such as operations performed by users using fingers, styluses, or any other suitable objects or accessories on or near the touch-sensitive surface), and drive corresponding connection devices according to a pre-set program. Optionally, the touch-sensitive surface may include a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 401, and can receive and execute commands sent by the processor 401. In addition, the touch-sensitive surface can be implemented using multiple types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch-sensitive surface, the input unit 405 may also include other input devices. Specifically, other input devices may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, etc.

[0246] The server may also include a display unit 406, which may be used to display information input by the user or information provided to the user and various graphical user interfaces of the server, which may be composed of graphics, text, icons, videos and any combination thereof. The display unit 406 may include a display panel, and optionally, the display panel may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like. Furthermore, the touch-sensitive surface may cover the display panel, and when the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor 401 to determine the type of touch event, and then the processor 401 provides a corresponding visual output on the display panel according to the type of touch event. Although in Fig.10 In the embodiment, the touch-sensitive surface and the display panel are used as two independent components to realize input and output functions, but in some embodiments, the touch-sensitive surface and the display panel can be integrated to realize input and output functions.

[0247] Although not shown, the server may also include a camera, a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the processor 401 in the server will load the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 will run the application stored in the memory 402 to achieve various functions, as follows:

[0248] Obtain the video to be analyzed and the questions to be answered related to the video to be analyzed;

[0249] Determine at least one video feature information corresponding to the video to be analyzed;

[0250] Determine the problem feature information corresponding to the problem to be answered;

[0251] Inputting at least one video feature information and question feature information into the trained video memory model for processing, so as to determine first target feature information related to the question to be answered from the video feature information;

[0252] Answer information corresponding to the question to be answered is determined based on the first target feature information and the question feature information.

[0253] The server can achieve the effective effects that can be achieved by any video analysis device provided in the embodiments of the present application. Please refer to the previous embodiments for details and will not be repeated here.

[0254] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0255] The above is a detailed introduction to a video analysis method, device and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A video analysis method, characterized in that: include: Obtaining a video to be analyzed and questions to be answered related to the video to be analyzed; Determining at least one type of video feature information corresponding to the video to be analyzed; Determine the problem feature information corresponding to the problem to be answered; Inputting the at least one video feature information and the question feature information into a trained video memory model for processing, so as to determine first target feature information related to the question to be answered from the video feature information; Determining answer information corresponding to the question to be answered according to the first target feature information and the question feature information; Wherein, the trained video memory model includes a first sub-model and a second sub-model; The determining of at least one video feature information corresponding to the video to be analyzed specifically includes: Extracting multiple video frames from the video to be analyzed; Determining at least one video feature information corresponding to each of the video frames; The step of inputting the at least one video feature information and the question feature information into a trained video memory model for processing to determine first target feature information related to the question to be answered from the video feature information specifically includes: Inputting the at least one video feature information corresponding to the plurality of video frames into the first sub-model in chronological order for processing, so as to obtain the first memory content corresponding to each of the video frames; Based on the at least one video feature information corresponding to the multiple video frames, the question feature information, the first memory content and the second sub-model, first target feature information related to the question to be answered is determined from the video feature information corresponding to each of the video frames.

2. The video analysis method according to claim 1, characterized in that: The determining, based on the at least one video feature information corresponding to the plurality of video frames, the question feature information, the first memory content, and the second sub-model, first target feature information related to the question to be answered from the video feature information corresponding to each of the video frames specifically includes: Determine a current video frame from the multiple video frames in the time sequence, and obtain a first memory content and a first target feature information corresponding to a previous video frame as a first historical memory content and a first historical feature information, respectively; Inputting the at least one video feature information corresponding to the current video frame, the question feature information, the first historical memory content, and the first historical feature information into the second sub-model for processing, so that the second sub-model determines the first target feature information related to the question to be answered from the at least one video feature information corresponding to the current video frame; The first memory content and the first target feature information corresponding to the current video frame are respectively updated to the first historical memory content and the first historical feature information, and the current video frame is updated using the remaining video frames, and then the step of inputting the at least one video feature information corresponding to the current video frame, the problem feature information, the first historical memory content and the first historical feature information into the second sub-model for processing is returned to be executed.

3. The video analysis method according to claim 2, characterized in that: The at least one video feature information includes dynamic feature information and static feature information, the first target feature information includes target dynamic feature information, target static feature information and target global feature information, and determining the first target feature information related to the question to be answered from the at least one video feature information corresponding to the current video frame specifically includes: Determining target dynamic feature information related to the question to be answered from the dynamic feature information corresponding to the current video frame according to the dynamic feature information corresponding to the current video frame, the first historical memory content, the first historical feature information, and the question feature information; Determining target static feature information related to the question to be answered from the static feature information corresponding to the current video frame according to the static feature information corresponding to the current video frame, the first historical memory content, the first historical feature information, and the question feature information; According to the first historical memory content, the first historical feature information and the question feature information, target global feature information related to the question to be answered corresponding to the current video frame is determined.

4. The video analysis method according to claim 1, characterized in that: The determining, based on the first target feature information and the question feature information, answer information corresponding to the question to be answered specifically includes: Inputting the at least one video feature information and the question feature information into a trained question memory model for processing, so as to determine second target feature information related to the video to be analyzed from the question feature information; Answer information corresponding to the question to be answered is determined based on the first target feature information and the second target feature information.

5. The video analysis method according to claim 4, characterized in that: The question feature information includes a plurality of word feature information, the trained question memory model includes a third sub-model and a fourth sub-model, and the inputting the at least one video feature information and the question feature information into the trained question memory model for processing to determine the second target feature information related to the video to be analyzed from the question feature information specifically includes: Inputting the plurality of word feature information into the third sub-model in sequence according to the word order of the question to be answered for processing, so as to obtain the second memory content corresponding to each word feature information; According to the plurality of word feature information, the at least one video feature information, the second memory content and the fourth sub-model, second target feature information related to the video to be analyzed is determined from each piece of word feature information.

6. The video analysis method according to claim 5, characterized in that: Determining second target feature information related to the video to be analyzed from each of the word feature information according to the plurality of word feature information, the at least one video feature information, the second memory content and the fourth sub-model specifically includes: Determine the current word feature information from the plurality of word feature information according to the word order, and obtain the second memory content and the second target feature information corresponding to the previous word feature information as the second historical memory content and the second historical feature information, respectively; Inputting the current word feature information, the at least one video feature information, the second historical memory content and the second historical feature information into the fourth sub-model for processing, so that the fourth sub-model determines the second target feature information related to the video to be analyzed from the current word feature information; The second memory content and the second target feature information corresponding to the current word feature information are updated to the second historical memory content and the second historical feature information respectively, and the current word feature information is updated using the remaining word feature information, and then the step of inputting the current word feature information, the question feature information, the second historical memory content and the second historical feature information into the fourth sub-model for processing is returned to be executed.

7. The video analysis method according to claim 5, characterized in that: The determining, according to the first target feature information and the second target feature information, answer information corresponding to the question to be answered specifically includes: Obtaining a first target feature matrix according to the first target feature information corresponding to the plurality of video frames, and obtaining a second target feature matrix according to the second target feature information corresponding to the plurality of word feature information; Inputting the first target feature matrix into a trained first self-attention model for processing to obtain first semantic long-range dependency information of the first target feature information, and inputting the second target feature matrix into a trained second self-attention model for processing to obtain second semantic long-range dependency information of the second target feature information; Answer information corresponding to the question to be answered is determined according to the first semantic remote dependency information and the second semantic remote dependency information.

8. A video analysis device, characterized in that: include: An acquisition module, used to acquire a video to be analyzed and questions to be answered related to the video to be analyzed; A first determination module, used to determine at least one video feature information corresponding to the video to be analyzed; A second determination module is used to determine the problem feature information corresponding to the problem to be answered; A third determination module, configured to input the at least one video feature information and the question feature information into a trained video memory model for processing, so as to determine first target feature information related to the question to be answered from the video feature information; A fourth determination module, configured to determine answer information corresponding to the question to be answered based on the first target feature information and the question feature information; Wherein, the trained video memory model includes a first sub-model and a second sub-model; The first determination module specifically includes: An extraction unit, used to extract multiple video frames from the video to be analyzed; A first determining unit, configured to determine at least one type of video feature information corresponding to each of the video frames; The third determination module specifically includes: A second determining unit is used to input the at least one video feature information corresponding to the multiple video frames into the first sub-model in chronological order for processing, so as to obtain a first memory content corresponding to each of the video frames; A third determination unit is used to determine first target feature information related to the question to be answered from the video feature information corresponding to each of the video frames based on the at least one video feature information corresponding to the multiple video frames, the question feature information, the first memory content and the second sub-model.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the steps in the video analysis method according to any one of claims 1 to 7.

Images